Fear of losing control or being non-compliant has stopped too many DevOps transformations. In most organizations, including all regulated ones, there is a degree of governance required to manage production releases. Policies around releases could include necessary security checks, quality bars, deployment windows or approvals. Compliance with architectural policies and cost impacts may also be relevant for pipelines governing infrastructure changes.
Ensuring these policies are followed can be challenging. As recently as a few years ago, when monthly or quarterly release cycles were the norm, policies could be enforced manually. A release manager would gather evidence of compliance, and an approval group would review the evidence. Only with sufficient compliance would the application be cleared for release.
As we have developed more mature continuous delivery pipelines, our delivery cadence has made manual review of each release untenable. A time-consuming and expensive process may work once a quarter, but when released many times a week, it is not.
Bringing governance to rapidly moving delivery pipelines is familiar. However, it is not something the industry has mastered. We have good guidance from groups such as Capital One around what organizations should do, but scaling and maintaining governance at speed still needs to be improved.
Bowing to the business need for speed, organizations have used a mesh of overlapping techniques to mitigate the risks the old processes attempted to address. These include:
• Security Training for Developers: If fewer security bugs are written, fewer will escape.
• Automating Checks in Pipelines: Application teams are asked to implement automated security scans and enforce promotion policies in their pipelines based on central standards.
• Code Review: After training developers in security and governance policies, teams can require code review, often through the pull request process, of changes to increase compliance.
• Regular Compliance Reviews: The CAB may be replaced by a governance architecture review that periodically (often quarterly) ensures that the appropriate checks appear in place.
Sadly, this patchwork often fails. For several reasons, but most often pressure from the business to release more, causing developers to start shortcutting checks. After the inevitable incident or failed review, organizations often take regressive steps such as handing control of pipelines to central teams, reducing the autonomy of the application team.
Organizations with this patchwork approach will also need help with challenges arising from gaps between central teams that manage policies and the distributed application teams that need to implement them. Policies are misunderstood, changes are slow to be rolled out and compliance may be uneven.
We need a way for policymakers to apply it across an organization while minimizing the day-to-day impact on innovation and developer productivity. A solution here would have to meet a range of needs:
1. Automated Checks: Policy compliance could be validated in moments, not significantly slowing CD pipelines.
2. Can Centralize Definition: GRC (governance, risk and compliance) teams should be able to define policies.
3. Rapid Rollout: Policy changes should roll across the organization quickly and efficiently.
4. Understandability: Policy authors and application teams governed by policies should be able to understand a policy and its impacts.
5. Maintain App-Team Autonomy: Developers on application teams should be able to make policy-compliant changes to their pipelines without needing help from another team.
These needs, in turn, suggest the characteristics of a solution:
● Operates “On” Not “In” a Pipeline: If policies are defined inside pipelines, it is difficult to give developers autonomy. Further, making policy changes means dozens, hundreds or thousands of pipelines must be updated quickly and faithfully — no easy task. As a result, policy definition and enforcement must exist outside pipelines.
● Automation for GRC Teams: Policy automation must be authored by GRC teams while remaining widely understandable.
With a policy-as-code (PaC) approach, such as using projects like the CNCF’s Open Policy Agent (OPA), GRC and platform teams can establish policies using an accessible DSL (domain-specific language). While a code will be machine-understandable, the DSL elements ensure it will remain human-understandable.
By wrapping a PaC engine around your pipelines, you can ensure policies such as requiring security scans. If the pipeline doesn’t have the scan step, it can be blocked. If a step that compares the results to your policies isn’t present, it can be blocked. The application team remains free to control how their pipelines build and deploy, but if they lack the required checks, they won’t be able to reach production. The application team owns application things, and the GRC team owns GRC things.
With PaC projects such as Open Policy Agent (OPA) graduating from the CNCF and an alternative, Kyverno, incubating PaC is moving into the mainstream and beginning to grow in its adoption within DevOps tooling.
Today, we see elements of PaC being adopted by major DevOps platforms:
● Azure DevOps: Azure DevOps Policies have as-code capabilities with a proprietary JSON-based template. Policies can be executed through Powershell so that they can be run through pipelines.
● GitLab: PaC has been introduced for security scans. Policies are defined in proprietary JSON documentation and are executed on merge request events or a schedule.
● Harness: The CNCF project, OPA, powers Harness’s PaC approach. Policies are available across the DevOps platform, from ensuring that security scans are run to requiring greater approvals for infrastructure-as-code executions that increase costs significantly.
As this approach starts taking hold in the industry, now is the time to learn more about technologies such as the CNCF’s OPA. If you are using tools with PaC, start building some policies, and if not, consider experimenting with those tools. You can find Harness’s free tier if you like OPA. Azure DevOps and GitLab have free offerings if JSON suits you.