Verifying CI/CD Pipelines with Prometheus

The most requested monitoring integrations from customers over the past year were: AppDynamics, New Relic, Dynatrace, and Datadog. The fifth most popular was Prometheus, the open-source software (OSS) project.

Unlike Application Performance Monitoring (APM) software, which has its own native run-time agents for different application stacks, Prometheus is very much a "brew your own" approach to monitoring.

With Prometheus, you write your own instrumentation, create your own metrics, and manage it all in your own custom time-series database. Anything is possible.

Why Automate Canary Deployments and Deployment Health Checks?

If you're serious about Continuous Delivery you'll want your deployment pipelines to execute and complete in less than 30 minutes.

Why? A few reasons actually:

• Deployment effort/time seriously limits deployment frequency
• You can't deploy hourly if your deployments take several hours to complete
• Failing fast is better than failing slow, remember deployments are experiments.
• The more time you spend deploying the less you spend developing

As an example, one of our early customers, Build.com, used to deploy 3 times a week to production. After each deployment, 5-6 team leads would log in into New Relic and Sumo Logic and manually verify for 60 minutes that everything was OK. That's 5-6 engineering hours per deployment just spent on verification. With Harness, Build.com automated verification down to one engineer and 15 minutes per deployment.

Applying Machine Learning to Prometheus Monitoring Data

Harness helps customers build deployment pipelines in minutes. It also helps customers auto-verify deployments by combining machine learning with their existing monitoring tools, in this instance, Prometheus.

Once Harness deploys a new application or service to a target environment it will immediately connect to the Prometheus Server and build a model of what it is observing. It will then compare this model with previous deployment models so it can identify new anomalies or regressions, and if need be, automatically rollback to the previous working version. At Harness, we call this Continuous Verification.

Setting up Prometheus for Continuous Verification

The first thing you need to do is Setup > Connectors > Add New Verification Provider:

Next, select Prometheus from the dropdown, enter your Prometheus Server URL, and a display name:

Adding Prometheus Verification To Your Deployment Workflows

Create or open an existing deployment workflow in Harness, click Add Verification, and select Prometheus:

Verifications can be applied to any type of deployment workflow, e.g. Canary, Blue-Green, Rolling, and so on.

Define Your Metrics

Due to the nature of Prometheus being a custom time-series metrics database you need to spend a little more time declaring which metrics you want to monitor as part of your deployment verification. This is somewhat different from APM monitoring tools where metrics are pre-defined and can be automatically configured within Harness.

A metric requires 4 parameters:

• Transaction Name - the service or request context which the metric relates to. E.g. Login
• Metric Name - the actual name of the metric defined in Prometheus
• Metric Type - the type of metric (response time, error, throughput, and value)
• URL - the API query required to retrieve the metric value (this must include placeholders for start time, end time, and hostname).

Once all your metrics are defined you can choose whether you require previous analysis or canary analysis to establish a baseline. You can also specify the duration of the verification (default is 15 minutes).

Let Prometheus Verify Your Deployments

With Prometheus integrated and configured you'll now be able to automatically verify all your deployments.

Take the below example that shows a typical deployment that has been successfully verified using Jenkins and Prometheus.

You can see on the right that all Prometheus metrics have been validated by the Harness machine learning algorithms. Green means no anomalies or regressions have been identified and the deployment is operating within its normal range.

You can get started with Harness and Prometheus today by signing up for your free trial.

Cheers,
Steve.
@BurtonSays