A Real-World Story of a Bottom-Up Secure-SDLC Evolution
This story begins in 2010, when a major security breach called Operation Aurora took place at Google. The event is unique to the security industry because it was the first major security breach that was announced publicly.
It marked the end of dark times for ethical hackers and security engineers and a turning point to a secure and transparent internet industry of today. During those times, companies rarely thought about security before products were released to the masses. Security testing was rarely incorporated into engineers’ daily routines.
One client, two projects
At the time, Lohika had two engineering teams working on two separate products from the same client. One product was ALM (application lifecycle management) software and the other was a security scanner for detecting vulnerabilities in software via a web interface.
The ALM software was a top-rated product used by thousands of users. Our engineers were “eating their own dog food” and used the ALM product in their daily job for bug tracking. Also, it won’t be a surprise that the test engineers had a security mindset as a result of their involvement in security scanner development and testing.
Completely on their own initiative, the test engineering team decided to run a set of security scanner tests against the ALM product. When they started exploiting it during their regular test runs, the ALM product turned out to be extremely vulnerable and had many security issues.
The Lohika test engineering team proactively created a comprehensive report about the state of security of the ALM product and a proposal on how to resolve the situation. The test engineering team even traveled to the client’s office to present the security report and the proposal.
The creation of a dedicated security team
This successful presentation to the client was the impetus for the creation of a dedicated security team at Lohika, and a turning point in how we think about software security in the entire organization. By shifting security testing to the left, it was no longer a detached process, but became an integral part of the software development process.
The security team began the task of fixing the ALM product’s security vulnerabilities. Naturally, it was a massive backlog of work from the get-go. Many components contained high-severity security issues related to web UI, backend services, and integrations with 7-8 other products.
Also, there were a few security reports received each month from end users. The security team was using automated security tools and implementing hundreds of security test scripts to cover various scenarios. Many cases required running 3-4 different security scanners simultaneously to verify a single fix.
Sometimes the number of security issues was so significant that the engineering team rejected taking them into development since the effort required for a proper fix was way beyond expectations. The reasoning behind this frustrating situation was straightforward: since security was not a concern at the design stage of the product (from engineering leadership and product owners on the client’s side), then almost every security-related update required a significant effort.
Security becomes an important business objective
Eventually, security testing became an integral process and the engineering team got used to fixing security issues during their regular development activities. All told, however, teams spent a couple of years on manual security testing, static code analysis, automated security scanning, and fixing security issues before the product became secure.
In the end, all critical issues were fixed, and our client stopped receiving security complaints from users.
At some point, it was the time for a new version of the product, with new features, based on modern technologies. Also, security became an important business objective for the new version of the product. From the start, the Lohika security team was involved in the following:
- Security engineers did design and architecture review and advised on security best practices
- Security engineers trained and supported the engineering team, and provided an extensive security analysis of the technologies used
- Algorithms and implementation received extensive code reviews from the security team
- Automation security scans became a mandatory and integral part of CI
- No feature or module was considered ready for release until the security team thoroughly tested it
The results of this approach were astonishing: the client received zero security tickets from users in 2+ years after the product release. Automated security scanners barely found any issues during the entire development period.
Finding a security issue, even with access to the source code, was rare. There was minimal effort required from the security engineers after the version was released to the public.
Compared to the previous version of the product, a secure implementation of the new version was much cheaper and quicker, since all the security-related work was accomplished at the earlier stages of development. The success was a result of the early involvement of security experts.
It is the engagement that the Lohika security team is incredibly proud of, as the new product got a market reputation for being safe.
The evolution to Secure-SDLC
This is the story of a natural evolution from ad-hoc security testing to a mature security-aware development process driven from the bottom up by the internal security champions. Nowadays, the industry calls this process S-SDLC.
When reading through the story it feels obvious that the lessons learned apply to every engineering organization: no software can be secure by itself, and neither adding a story “make the product secure” on top of a backlog can save a situation.
Once the key decisions on design, architecture and implementation are in place, making your product secure means nothing more than re-design, re-architecture, re-implement, re-test, and re-deploy. Making your product secure requires security to be a fundamental business objective permeating product and engineering culture.
Secure-SDLC is an extension of the SDLC process. In a nutshell, it augments every SDLC stage with a security-driven aspect:
- Identifying and defining all the security requirements to meet security goals is a necessary part of the requirements phase
- Ensuring the security of the product architecture by creation of a threat model as a part of the design phase
- Learning and applying secure coding practices, focusing on security during coding, code reviews, and automated secure code analysis as a mandatory part of CI during the development phase
- Automated and manual security testing as a go/no-go decision criteria for any feature before rolling out to production
- A focus on security auditing before deployment, monitoring security of the production environment, and fastest security incident response
Often people sense that S-SDLC has a waterfall-ish nature, and it is hard to implement it within the Agile environment. Here is a good article about Agile S-SDLC: “Why existing SDLC methodologies are failing.”
As you can see, the essential difference between “plain old” SDLC and Agile and Secure SDLC is that to deliver truly secure software, your organization commits to security from day one, shifts left, and drives security from the bottom up.
You can always ask the Lohika security experts how we do Agile and secure SDLC today.
Finally, to strengthen our point with a real-life example, here is how Zoom handles the security issues uncovered at the face of a booming number of users driven by COVID-19.