Gate your rollout on live metrics with the FME Metric Check step | Harness Blog

Progressive rollouts usually end the same way: someone watches a dashboard, decides the error rate looks fine, and moves the percentage up. That check is the part of the release that depends most on a person being awake and paying attention.

The Metric Check step moves that decision into the pipeline. It is a new step in Harness Feature Management & Experimentation (FME) that reads a metric for a feature flag during a pipeline run, evaluates it against a failure condition you write, and returns pass or fail. If it fails, the pipeline follows whatever failure strategy you configured, such as stopping or rolling back.

Why the measurement should live with the flag

Most teams already have a version of this check. A monitoring tool spots a spike, something calls an API, and a flag gets turned off. The weak link is the middle. The monitoring tool knows a service got worse. It doesn't know which change made it worse. With several flags in flight, an alert is a correlation between a time window and whatever was shipping then, and turning off the nearest flag may fix nothing while hiding the real cause.

FME ties events to the treatment a user was served, using the impression and the user key. The number Metric Check evaluates comes from the users who received the change, and not from the whole service. The decision to continue, pause or roll back is made on that number, in the same pipeline that made the change. That is the difference between "something got worse" and "this flag's users got worse."

How it works

You add the step to a Custom stage in a Harness pipeline. It sits under Feature Management & Experimentation in the Step Library, next to the other FME steps. You configure:

• the environment and the feature flag
• a lookback window, for example 15 minutes or 1 hour
• one or more metrics to evaluate
• a failure condition, written as a JEXL expression, that evaluates to true when the step should fail

For example, this condition fails the step if the average error rate for the "on" treatment goes above 5%:

<+metric("error_rate").treatment("on").mean> > 0.05

You can combine conditions. This one fails if either error rate or p95 latency crosses its limit:

<+metric("error_rate").treatment("on").mean> > 0.05 || 
  <+metric("api_latency").treatment("on").p95> > 500

After the step runs, the execution details show the metric values it evaluated and the pass or fail result. The step also exposes metric and evaluation details as pipeline outputs, so later steps can reference them.

Thresholds, not statistical significance

Metric Check compares values to thresholds you set. It does not wait for statistical significance the way an experiment does. That makes it suited to release decisions, where you want a fast answer to "is this release hurting error rate or latency", and a different tool from experiment analysis, where you want to know whether a change caused a lift. If you want the second answer, keep using experiments.

Using it with the rest of your pipeline

Because it is a standard pipeline step, it works with the things pipelines already do. You can put it between rollout steps, for example after Set Default Allocations raises the percentage, and before the next increase. You can attach notifications and a failure strategy, for example one that stops the pipeline or rolls back a deployment. One point from the docs worth stating plainly: Harness approvals control the flow of a pipeline, and FME environment-level approval settings do not apply to pipeline runs. If your team relies on environment approvals, design the approval into the pipeline.

Get started

1. Check access. Metric Check is in Limited GA and is enabled per account. Request access through the banner on the Pipelines documentation page. You also need pipeline view, edit and execute permissions.
2. Send events from your app. Call track with the same user key and traffic type you pass to getTreatment. An event only counts toward a treatment if it happens after that user's impression. Send events for users in both treatments, and make sure the values vary, ideally with a visible gap between the treatments. A check on a flat metric has nothing to catch. Go to Events and Attribution and exclusion for details.
3. Create the metric. Open Metrics from the FME navigation menu and select + Create metric. For an error rate, set the name to error_rate, the desired impact to decrease, and the traffic type to user. The traffic type must match the flag's traffic type. Measure it as an average of event values per user, and select the error_rate event. Leave the category as None unless you want guardrail alerting, which applies account-wide. Go to Create a metric for every option.
4. Create the pipeline. Open Pipelines from the FME navigation menu, select + Create a Pipeline, enter a name, choose Inline, and select Start. Select Add Stage, then Custom Stage. FME steps run only in a Custom stage.
5. Add the step. In the stage, select + Add Step, then select Metric Check under Feature Management & Experimentation.
6. Fill in the fields. These values work for a checkout flag:
   
   1. Name: Validate checkout metrics
   2. Timeout: 30m, the longest the step waits for results
   3. Environment: the environment where the flag is configured, for example Staging
   4. Feature Flag: checkout-redesign
   5. Lookback Window: 1h. The window counts back from when the step runs, so it must reach back to when your events were sent. If it contains no data, the step returns an error.
   6. Metrics: error_rate. Select + Add a metric to add more, such as api_latency. Each name must match a metric you created.
7. Write the failure condition. The failure condition is a JEXL expression that evaluates to true when the step should fail. Replace on and off with the treatment names on your flag. The step returns an error if a treatment name does not exist. Besides mean and p95, you can use count, sum, max, min, p50, p99 and variance. Go to Metric Check for the full list. Examples:
   
   1. Fail if the average error rate for on is above 5%, with error rate sent as a fraction:
      <+metric("error_rate").treatment("on").mean> > 0.05
   2. Fail if the error rate is above 5% or p95 latency is above 500 ms:
      <+metric("error_rate").treatment("on").mean> > 0.05 || <+metric("api_latency").treatment("on").p95> > 500
   3. Fail if the error rate for on is more than 10% higher than for off. Both treatments need data in the lookback window:
      <+metric("error_rate").treatment("on").mean> > 1.1 * <+metric("error_rate").treatment("off").mean>
8. Set a failure strategy. On the Advanced tab, choose what happens when the check fails, for example stopping the pipeline or rolling back a deployment. Go to failure strategies for the options.
9. Save and run. Select Apply Changes, save the pipeline, and run it. The step details show the configured inputs, the metric values it evaluated and the result under Details, Input, Output and Execution Context.

If the step returns an error instead of a result, check that the lookback window contains events and that the treatment and metric names match your flag. Go to Troubleshooting in the documentation.