Trying to secure code is no easy task. It takes real effort to build a product and keep it secure. As we try to shift left and build security into our applications, we are pushed to learn new ways to meet these goals. One of those ways is to use tools. Unfortunately, the complexities of security make it very easy for companies to offer quick fixes and simple solutions. They offer security theater. If you’re not familiar with the term, security theater is when you create measures that give the illusion of security while doing little or nothing to actually secure anything.
For example, some security products provide checkboxes indicating compliance with one or more standards. They claim to show you when your code has achieved compliance. In most cases, they create an illusion of compliance that creates a legal hazard for the unwary. They rely on the fact that time-constrained business are looking for certain keywords – and enough fine print in the documentation to protect their interests. They may offer a tool that is incredible towards meeting that compliance, but they err in making users feel that they may have achieved compliance.
Checkbox marketing
Because it’s rare for companies to have security teams that contain people that specialize in security and development, it creates an environment ripe for exploitation. I’ve seen this in both software shops and managed service providers (hint: beware hosting firms and MSPs that don’t have development expertise on the security team!). Unless teams “shift left” and become one organization that works together, they can miss key details. Let’s explore how this works.
Checkbox security applications will often indicate your code compliance with “PCI DSS”, “OWASP Top 10 2021”, and “NIS SP 800-53” (among others). These tools often show a checkmark when you are “compliant”. The truth about their checkmark is normally hidden in the documentation: they support a small part of complying with those standards, but are not actually capable of providing or validating the majority of the requirements.
The hard truth: there is no single tool that can give you compliance, despite what they may claim. Why do I say that? Security standards consist of numerous requirements, starting with the processes and practices used to create and deploy the software. Building on that, there are additional requirements covering the application security practices, infrastructure security, code quality, and more. While tools can automate some of this, the rest requires written documentation, practice, and internal standardization.
Back in the real world
Let’s look at a real example. Take the NIST SP 800-53 R5. It’s nearly 500 pages long. If you’re using a code analysis tool, you’re meeting portions of the requirements from 4 of those pages. But what about IR-4 (Incident Handling)? Its controls require you to be able to coordinate incident handling activities with contingency panning activities (among other things). While tools can help build those plans, they can’t guarantee that you have a plan or that the team can execute against it.
Most of the other standards are similar. For example, PCI DSS v4.0 is over 300 pages. Part of this standard includes validation of third-party service providers. Another part concerns how specific types of data are stored and accessed. Again, there are tools that can help – but no single tool does everything.
Code-specific compliance
Now let’s look at the next common choice – the OWASP Top 10. This is a bit more straight-forward – but it’s not a compliance standard. It’s actually a collection of numerous security vulnerabilities which have been grouped into categories for understanding. If we examine the 2021 list, A03 Injection is mapped to a set of 33 Common Weakness Enumerations ( CWEs). Each CWE represents a specific hardware or software vulnerability type.
It’s a better candidate for a checklist, but has some caveats. First, you need to know which CWEs it’s able to identify. This is especially true with the lists before 2021. Those included a limited subset of CWEs, but not all related CWEs. It was an exercise to the implementor to identify other related CWEs that could be included. A more important detail comes from OWASP’s How to use the OWASP Top 10 as a Standard:
If you want to use the OWASP Top 10 as a coding or testing standard, know that it is the bare minimum and just a starting point. One of the difficulties of using the OWASP Top 10 as a standard is that we document appsec risks, and not necessarily easily testable issues…
We would encourage anyone wanting to adopt an application security standard to use the OWASP Application Security Verification Standard (ASVS), as it’s designed to be verifiable and tested, and can be used in all parts of a secure development lifecycle.
The ASVS is the only acceptable choice for tool vendors. Tools cannot comprehensively detect, test, or protect against the OWASP Top 10 due to the nature of several of the OWASP Top 10 risks … OWASP discourages any claims of full coverage of the OWASP Top 10, because it’s simply untrue.
In short, even OWASP themselves documents that it is impossible for a vendor to provide full coverage of the OWASP Top 10!
The broader risk
On any given day, there are over 2,200 cyber attacks and an average of 55 new vulnerabilities discovered each day. Lists like the OWASP Top 10 are a constantly expanding definition – not a comprehensive source of compliance. As for the various standards, they are designed to help teams be better prepared and protected from the known and unknown risks.
To this end, tools should always to provide a clear understanding of the kinds of exploits it discovers and their standardized classification. It should transparently help you to understand your security risks, including how to address those risks. At the same time, it should not pretend to offer complete compliance. For example:
- Microsoft Azure provides built-in policies, such as PCI DSS 3.2.1. It clearly documents its mapping to the controls in that standards. It also provides links to the shared responsibility model. There’s even a workbook that helps with understanding how to achieve a compliant state.
- Amazon AWS provides similar guidance. For example, to cover PCI DSS, they offer a technical workbook, an updated whitepaper, documentation PCI requirements addressed by the Quick Start, and other resources.
- GitHub’s CodeQL (part of GitHub Advanced Security) provides with a list of the CWEs it covers. The built-in queries each map to CWEs and provide details to help you understand the nature of the problem, the path taken to exploit the code, and suggested fixes.
Notice that none of these claim to provide full compliance with a standard?
Security is about trust. If a vendor isn’t honest about their capabilities, how much can you trust their solution? Are they providing you a solution, or are they just offering security theater?