At Harness, we're often asked, "What is the best practice for scaling Continuous Delivery across an organization?" Naturally, our response is: "Fire anyone who still does waterfall and immediately hire 10 DevOps engineers." I'm joking. Hiring 10 DevOps engineers will take you 6-12 months at best and will probably cost you $1M+ to implement.

Continuous Delivery is a progressive journey that doesn't happen overnight.

For example, not everyone needs to deploy hourly to achieve business outcomes. We've worked with several customers who went from monthly to weekly deployments and this had a massive impact on their business.

This leads to an important first question: What is a success for your organization or team?

What Are Your CI/CD Goals?

Don't try and boil the ocean on day one or set unrealistic goals, like "We want to be like Facebook and do one zillion deployments a day."

Get some quick wins under your belt, gain momentum, and set bigger goals throughout the journey.

If you're starting out, have goals like this for scaling Continuous Delivery:

• Automate the promotion of code across development, QA, staging, and production environments.
• Increase deployment velocity from monthly to bi-weekly cycles.
• Reduce deployment effort from several engineers to a single engineer.

If you're more advanced, have goals like this for scaling Continuous Delivery:

• Migrate from blue-green deployments to canary deployments.
• Reduce rollback time from 30 minutes to 5 minutes.
• Templatize deployment pipelines so all teams have consistent deployment pipelines.

If you're an expert, then you may have goals like this for scaling Continuous Delivery:

• Empower every developer to deploy their own code in production.
• Report and quantify the business outcome of every deployment.
• Perform continuous verification and rollbacks based on business anomalies or regressions.

Where Is Your Organization Today?

Like anything, understand where you are today and where you want to get to.

Below is a simple three-stage maturity model that reflects where most customers are in their Continuous Delivery journey:

Simply put, how often are you shipping code? And who is responsible for deployments today in your org? Is it IT Operations, DevOps, or Developers?

60% of customers we speak to are in phase 1, meaning IT operations (specifically release and deployment teams) are responsible for deployments -- most of which do monthly release cycles. "Release" is a keyword here, representing a major event that requires planning, meetings, and change control processes. This is primarily why deployment cycles are longer. Deployment pipelines do not typically exist and deployments are either managed with ad-hoc scripts or via infrastructure configuration management tools.

~35% of customers are in phase 2 where a dedicated DevOps team exists and is primarily responsible for platform engineering, CI/CD, and automation. In some orgs, the DevOps team may also own deployments. DevOps teams here typically try to build CD capabilities on top of their CI platform like Jenkins or Bamboo using shell scripts.

Less than 5% of customers today are practicing Continuous Delivery as-a-Service. For many, stage 3 is where the majority of customers want to get to; it's very much the "Nirvana" or "Holy Grail" of software delivery. This stage is where DevOps team provide a turnkey "CD-as-a-Service," and developers have high levels of autonomy to build their own deployment pipelines so they can deploy to production on their own.

Organizational Characteristics & Requirements for CI/CD

As with the above maturity model, the vast majority of customers we talk to currently have a centralized deployment model which is either owned by IT Operations or DevOps. This model tends to work best with monolith and SOA applications because small dedicated teams can easily manage monthly release cycles. Security and compliance requirements are lower because less people are deploying, as is the level of visibility and reporting because you have less change and activity.

Unfortunately, this centralized deployment model breaks with microservices, serverless, and cloud-native applications. More components with higher levels of parallel development means a 100X increase in change and deployments.

To overcome this bottleneck, IT Ops and DevOps teams need to empower developers with a self-service platform. This unfortunately does come at a price. Giving freedom to developers has a dramatic impact on security, compliance, visibility, and reporting requirements -- because the number of people now deploying could easily be 100X. With great freedom comes responsibility, and organizations still need to retain a level of control.

For example, here are some critical capabilities Harness built to support Continuous Delivery as-a-Service for our customers:

• Single-Sign-on (SSO) support for authentication
• Role-Based-Access-Control (RBAC) for authorization
• Usage Restrictions for Cloud Accounts & CI/CD tooling
• IP whitelisting to control device access
• Audit logs for detailed reporting of who, what, and when
• Secrets management for the management of credentials, keys, and certificates
• Service-level dashboards that show what version/config is running in what environment for every service/application
• Template library so developers can reuse deployment pipelines, workflows, commands, and scripts across teams
• Configuration-as-code so developers can script and version control their own deployment pipelines

Empowering developers to deploy means your CD platform needs to support major security, compliance, and reporting requirements. If you're building your own CD platform, these are requirements that are not trivial to bolt-on. It makes more sense to buy a CD-as-as-service vs. build it yourself.

What might take you 12+ months to build can be delivered in a few hours with a solution like Harness.

Who Owns Building Deployment Pipelines For Your Services?

A deployment pipeline is a logical group of stages that code must progress through for it to be ready for customers in production. Stages typically map to environments like Dev, QA, Staging, and so on. Within a stage, you typically deploy code to infrastructure (aka environment) and run a series of tests to validate that everything is functionally and non-functionally sound with the latest build artifact.

Think of a deployment pipeline as a car production line. The car goes through a build process and is promoted through different test areas of the plant where it is inspected and verified using tools before it is finally delivered to the customer. If at any stage a defect is found, it is rejected and sent back to be fixed before it can proceed to the next stage.

In most organizations, a centralized DevOps team is typically responsible for building deployment pipelines. Think of DevOps as the designers and architects for the car production line. They're ultimately responsible for the structure and governance of everything within a deployment pipeline (environments, tools, tests, standards, security).

Once a deployment pipeline template has been created, it can then be made available to release/deployment engineers or developers to instantiate and use with their own service artifacts.

For example, Harness has one customer, Jobvite, who uses a single deployment pipeline template for 40 different microservices (dev teams) across 4 environments (dev, QA, staging, and prod). That's a single consistent repeatable way to test, deploy, and verify deployments.

The ultimate goal is to let developers themselves build their own deployment pipelines, making it a true self-service experience. Harness can enable either use case whether you want a centralized or decentralized management of deployment pipelines. Centralized management gives you control; decentralized management will give you scale. Either way, you need an owner for the building and management of pipelines.

Who Owns Deployments For Your Services?

Once deployment pipelines exist, you need to decide who is responsible for pushing that big shiny deployment button.

Again, IT operations and centralized release/deployment teams have traditionally been the owner of deployment buttons. DevOps teams today are slowly transitioning this responsibility to the application development teams. DevOps culture and Continuous Delivery is a huge part of enabling this transition.
You can go two directions with Continuous Delivery:

• Human-driven deployments (semi-automated).
• Event-driven deployments (fully-automated).

The first is where a human manually kicks off a deployment pipeline and it progresses through the various stages. Most of our customers start with this process because they want complete control of their deployment process until confidence grows and they become familiar with how their pipeline executes. The use of manual approval stages within a pipeline also helps customers retain control at every step. When the time is right, you can remove manual approvals and blocks from your pipelines to drive more automation.

Bottom line: start with a deployment pipeline that is triggered by humans and has manual approvals between different pipeline stages. After a few deployments, you can gradually reduce the number of manual approvals and handoffs.

Event-driven deployment is where pipelines are triggered based on events like:

• Creation of new builds or version in your artifact repository.
• Completion of other deployment pipelines.
• Time of day, week or month.
• HTTP Webhook.

The end goal for many customers is to kick off a deployment pipeline automatically based on a developer checking in code and creating a new build: each new artifact should then automatically progress through dev, qa, staging, and into production with little or no human supervision. You rely on your test automation scripts, tools, and verification steps to tell you whether artifacts are fit for the environment they are deployed in (including production). If verifications fail, then artifacts should be automatically rolled back. For example, here is an example from one of our customers, Build.com, who does this.

Who Manages Your CD Configuration?

Finally, where does your Continuous Delivery configuration sit, and who manages it?

Harness supports both a centralized model and a decentralized model for CD configuration.

IT Operations or DevOps teams can manage all CD configuration through a single-pane-of-glass user interface or through a configuration as code API. This can be managed at the account level.

Developers can also manage CD configuration at a service-level using GitOps in Harness. Harness basically allows developers to manage CD config in YAML and manage this config in the same repo as their source code. Developers basically manage their deployment pipelines like their apps, as source code in the repo. Harness has bi-directional sync and version control to ensure everything stays up-to-date. This option gives developers full control of their pipelines, but also gives DevOps teams reporting visibility into who, what, and when surrounding deployments.

Scaling Continuous Delivery With Harness

If you want to give Continuous Delivery a shot, sign up for our free trial. The Harness platform supports all 3 CD maturity levels and it doesn't matter if IT Ops, DevOps, or developers want to own pipelines and deploy.

We can give you the flexibility to support your current deployment process or help you transition to CD-as-a-service where developers self-serve their needs. We run SaaS or on-premises so we also cater to all types of organizations. Scaling Continuous Delivery is a breeze with Harness.

Cheers!
Steve.
@BurtonSays