May 4, 2018

Harness Extends Continuous Verification To Datadog

Table of Contents

Key takeaway

These days, it's not uncommon for customers to use Cloud or infrastructure monitoring software across their dev, QA and production environments.

Back in February, we announced support for Dynatrace to accompany our AppDynamics and New Relic APM support. Today, our Datadog support is now GA.

Why Harness + Datadog?

Harness helps customers master Continuous Delivery (CD) so they can deploy new applications and services into production with speed and confidence.

Datadog helps customers monitor the performance of their cloud applications and infrastructure.

Customers typically have one or more deployment pipelines for each application/service. A typical pipeline has several stages that reflect the environments (e.g. dev, QA, staging, production) which are used to test/validate code before production deployment occurs.

Integrating Harness with Datadog now allows customers to automatically verify cloud application and infrastructure performance across their deployment pipelines.

A Simple Deployment Pipeline Example

Let's imagine we have a Docker microservice called 'My Microservice' and our Continuous Integration (CI) tool Jenkins creates a new build (#101) which results in a new artifact version for us.

With Harness, we could pick up that new version and immediately trigger a 4-stage deployment pipeline like this:

Example deployment pipeline

Each pipeline stage deploys our new microservice to a different environment, runs a few tests, verifies everything is good before proceeding to the next stage.

You can see from the above screenshot that stages 1 thru 3 (dev/QA) succeeded and turned green. This is generally good, it means the artifact passed all tests and is ready for production and customers.

The bad news it that stage 4: production turned red and our pipeline status is 'failed'.

Automating Deployment Health Checks With Harness

At Harness, we take CD one step further with something called Continuous Verification.

One of our early customers used to verify production deployments with 5-6 team leads manually analyzing monitoring data and log files. This process took each team lead 60 minutes and occurred 3 times a week. That's 1,080 minutes or 18 hours of team lead time spent on verification. With Harness, reduced verification time to just 15 minutes, and also enabled automatic rollback to occur in production.

With Datadog, Harness is able to deploy and verify the performance of cloud application deployments instantly in every environment. The second a new application or service artifact is deployed, Harness will automatically connect to Datadog and start analyzing the application/service/infrastructure performance data to understand the real business impact of each deployment.

Harness applies unsupervised machine learning (Hidden Markov models & Symbolic Aggregate Representation) to understand whether performance deviated for key services and flags performance regressions accordingly.

Let's take a look at what this verification looks like with Datadog.

Continuous Verification With Datadog

The below deployment workflow relates to the above-failed pipeline, and specifically stage 4 in production.
You can see that phase 1 of the deployment failed. The deployment in terms of deploying the service succeeded, as did the verifications and tests relating to Jenkins. However, the Datadog verification failed which means new performance regressions have been identified. The resulting action, in this case, was an automatic rollback (this is the safety net that Harness provides).

Harness UI

Clicking on the Datadog failed step (red) shows us why the verification failed:


We can see that one key web transaction “get_/todolist/inside/load” has a request duration regression after our new microservice version was deployed. Mousing over the red dot we can get more details – response time actually increased 58% from 840ms to 1330ms post-deployment:

Web Transaction

The default time period for this verification was 15-minutes but this number can be custom. Some deployment pipelines and workflows can take hours to complete because not everything is automated (manual checks/approvals etc.).

You can also configure/filter any Datadog metrics by clicking the Metrics Filter:

Metrics Filter

Configuring Datadog Continuous Verification

Our integration was pretty simple to build using the standard Datadog API.

The first thing you need to do is Setup > Connectors > 'Add New Verification Provider':

Harness Connectors

Then enter your Datadog URL, API Key, and Application Key:

Adding Keys

You can get your API Keys from your Datadog Integrations APIs screen:

Datadog API Harness

Adding Datadog Verification To Deployment Workflows

First, add or edit an existing workflow in Harness.

Simply click 'Add Verification' and select Datadog from the menu.

Add Datadog from Menu

Next, select your Datadog server, enter the service name you want to verify, select the metrics you wish to verify, and enter a verification time period (default is 15 minutes).

Datadog Server

Click submit and your deployment workflow will now automatically verify application performance using Datadog.

More To Come With Datadog

As you can probably imagine there are lots more things we can do with Datadog. Here are few things to expect in the future:

  • Harness deployment markers for Datadog users
  • Support for Datadog log unstructured event data (similar to our Splunk, ELK and Sumo Logic support)

Special thanks to our customers and engineering team for creating this support.


You might also like
No items found.

Similar Blogs

No items found.