Product
|
Cloud costs
|
released
May 8, 2018
|
3
min read
|

Verifying CI/CD Pipelines with Prometheus

Updated

The most requested monitoring integrations from customers over the past year were: AppDynamics, New Relic, Dynatrace, and Datadog. The fifth most popular was Prometheus, the open-source software (OSS) project.

Unlike Application Performance Monitoring (APM) software, which has its own native run-time agents for different application stacks, Prometheus is very much a "brew your own" approach to monitoring.

With Prometheus, you write your own instrumentation, create your own metrics, and manage it all in your own custom time-series database. Anything is possible.

Why Automate Canary Deployments and Deployment Health Checks?

If you're serious about Continuous Delivery you'll want your deployment pipelines to execute and complete in less than 30 minutes.

Why? A few reasons actually:

  • Deployment effort/time seriously limits deployment frequency
  • You can't deploy hourly if your deployments take several hours to complete
  • Failing fast is better than failing slow, remember deployments are experiments.
  • The more time you spend deploying the less you spend developing

As an example, one of our early customers, Build.com, used to deploy 3 times a week to production. After each deployment, 5-6 team leads would log in into New Relic and Sumo Logic and manually verify for 60 minutes that everything was OK. That's 5-6 engineering hours per deployment just spent on verification. With Harness, Build.com automated verification down to one engineer and 15 minutes per deployment.

Applying Machine Learning to Prometheus Monitoring Data

Harness helps customers build deployment pipelines in minutes. It also helps customers auto-verify deployments by combining machine learning with their existing monitoring tools, in this instance, Prometheus.

Once Harness deploys a new application or service to a target environment it will immediately connect to the Prometheus Server and build a model of what it is observing. It will then compare this model with previous deployment models so it can identify new anomalies or regressions, and if need be, automatically rollback to the previous working version. At Harness, we call this Continuous Verification.

Setting up Prometheus for Continuous Verification

The first thing you need to do is Setup > Connectors > Add New Verification Provider:

Add New Verification Provider



Next, select Prometheus from the dropdown, enter your Prometheus Server URL, and a display name:

Add Prometheus

Adding Prometheus Verification To Your Deployment Workflows

Create or open an existing deployment workflow in Harness, click Add Verification, and select Prometheus:

Add Prometheus Deployment


Verifications can be applied to any type of deployment workflow, e.g. Canary, Blue-Green, Rolling, and so on.

Define Your Metrics

Due to the nature of Prometheus being a custom time-series metrics database you need to spend a little more time declaring which metrics you want to monitor as part of your deployment verification. This is somewhat different from APM monitoring tools where metrics are pre-defined and can be automatically configured within Harness.

A metric requires 4 parameters:

  • Transaction Name - the service or request context which the metric relates to. E.g. Login
  • Metric Name - the actual name of the metric defined in Prometheus
  • Metric Type - the type of metric (response time, error, throughput, and value)
  • URL - the API query required to retrieve the metric value (this must include placeholders for start time, end time, and hostname).
Configure Prometheus


Once all your metrics are defined you can choose whether you require previous analysis or canary analysis to establish a baseline. You can also specify the duration of the verification (default is 15 minutes).

Let Prometheus Verify Your Deployments

With Prometheus integrated and configured you'll now be able to automatically verify all your deployments.

Take the below example that shows a typical deployment that has been successfully verified using Jenkins and Prometheus.

You can see on the right that all Prometheus metrics have been validated by the Harness machine learning algorithms. Green means no anomalies or regressions have been identified and the deployment is operating within its normal range.

Harness With Prometheus



You can get started with Harness and Prometheus today by signing up for your free trial.

Cheers,
Steve.
@BurtonSays

Sign up now

Sign up for our free plan, start building and deploying with Harness, take your software delivery to the next level.

Get a demo

Sign up for a free 14 day trial and take your software development to the next level

Documentation

Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.

Case studies

Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.

We want to hear from you

Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.

Sign up for our monthly newsletter

Subscribe to our newsletter to receive the latest Harness content in your inbox every month.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Continuous Delivery & GitOps