Test Driven Compliance

Test Driven Compliance

This is part 2 of a short series on balancing the complexities of regulation and security. The first post provided an overview of some of the challenges that security teams face when they operate in regulated environments, these challenges are then exaggerated by a company's overall size and/or maturity. This post will be the first of several that will focus on possible solutions to these challenges.

Background On Test-Driven Development

Test-driven development (TDD) for those that are not familiar with the concept is the idea of writing code by starting with the test, particularly one that fails. The engineer working on that code will then iterate through a write, test, fail, improve loop until everything is working as expected. This strategy works quite nicely alongside a continuous integration (CI) system so that you can also see how your code functions while you're hacking on it. For those that want to do some additional reading on the topic, check these resources out:

Test-Driven Compliance

With that bit of context out of the way, we can move into how this can be applied in the world of security and compliance. Many controls within a compliance framework can be thought of as a specific feature of a system, consider something like data being encrypted at rest or password policies being of a certain complexity and length. Compliance frameworks today contain a broad and varied set of controls for teams to worry about, many of them though when scoped down to a specific system can be described as a testable feature. Writing tests for each of these features and mapping them back to their respective controls, especially if they run as part of a CI/CD pipeline, will yield several key benefits:

  • Continuously running tests will let teams know when a change that they've made unintentionally "breaks" compliance and puts a system at risk. This allows for swifter remediation of such issues as well as traceability of the issue.
  • Codified policy/controls in the form of CI tests make it easier for a security or compliance team to produce reports internally and for auditors.
  • Faster feedback loops for the individuals that are actually making changes to infrastructure that falls within the compliance boundary.
  • Greater assurance for those implementers and maintainers working on compliance-related systems when they are making changes, increasing overall confidence.

In Practice

At a high level, I'll outline some practical test driven compliance scenarios. This list of examples, like before, is not exhaustive.

  • Testing/linting for specific controls such as password complexity, account within codified endpoint or server configuration.
  • Programmatically testing or monitoring AWS (or other cloud) infrastructure for specific features on cloud resources that align with compliance controls.
  • Specifically calling out controls within alert titles/names from monitoring and alerting processes that run on a continuous basis.

Pragmatic Considerations

Managing compliance for a large system or a large enterprise as you might imagine is no small task. There are several key considerations that can make an objective like this harder to achieve, those will be covered here, keeping in mind that this is by no means close to exhaustive.

Understanding what your compliance boundary looks like can be tricky. For example, if an organization has compliance relevant assets that are stored within servers, databases, traversing network traffic in product infrastructure, or residing on laptops, all of those moving pieces need to be accounted for as part of the compliance boundary. What this translates to is that the relevant controls in a standard likely need to be applied to each and every one of the moving pieces. This leads into the next consideration.

Ensuring that your testing or monitoring infrastructure covers (or is even supported in) all of the places that have compliance considerations is tough, especially when creep occurs. Creep could be defined as any infrastructure currently within a compliance boundary was not part of the original design, it just kind of happens over time as part of system atrophy through maintenance, user interaction, etc. Managing that creep will go a long way towards being able to sustain a test driven compliance strategy that is actually representative of your compliance boundary.

Wrapping Up

Codifying a compliance program is hard work, however leveraging automation and controlling (to the greatest degree possible) creep that might inhibit that 100% coverage goal can pay huge dividends. As you're working through these efforts it's important to keep in mind the end goal, streamlining a business function that is historically a burden, one which historically carries with it fines, time, and a mountain of paperwork.

About Robert Wood