DevOps Platform Explained: Why Unified Wins Over Siloed Tools | Harness Blog

DevOps Meets AI: Evaluating the Performance of Leading LLMs

Modern DevOps processes are essential for ensuring efficient, reliable, and scalable software delivery. However, managing infrastructure, CI/CD pipelines, monitoring, and incident response remains a complex and time-consuming challenge for many organizations. These tasks require continuous tuning, configuration management, and rapid troubleshooting, making DevOps resource-intensive. As software systems grow in complexity, manual intervention becomes a bottleneck, increasing the risk of human error, inefficiencies, and slower deployments. This is where automation becomes a necessity, helping teams streamline workflows, reduce operational overhead, and improve deployment velocity.

‍

The rise of artificial intelligence, particularly large language models (LLMs), has opened new possibilities for automating various aspects of software development and operations. By leveraging AI, organizations can enhance efficiency, reduce manual effort, and accelerate software delivery. LLMs bring the potential to transform DevOps by enabling intelligent automation, improving decision-making, and making systems more adaptive to changing requirements.

Our AI engineering team has been at the forefront of integrating AI into DevOps workflows. From AI-powered CI/CD optimizations to intelligent deployment strategies, we continuously explore ways to leverage AI for greater efficiency. In this blog, we share our journey in evaluating LLMs for DevOps automation, benchmarking their performance, and understanding their impact on software delivery workflows.

Harnessing LLMs for DevOps Automation

Before diving into the evaluation, let’s first outline the specific problem we aim to solve using large language models. (Note: In this post, I won’t go into the underlying architecture of the Harness AI DevOps Agent — stay tuned for a future blog post on that!)

Our exploration begins with the task of pipeline generation. Specifically, the AI DevOps Agent takes a user command describing the desired pipeline as input, along with relevant context information. The expected output is a pipeline YAML file generated by the AI DevOps agent, which is composed of multiple sub-agents, automating the configuration process and streamlining DevOps workflows. An example user command and the resulting YAML pipeline would be:

“Create an IACM pipeline to do create a IACM init and plan”

Response:

For simplicity, we conducted the first phase of our evaluations by focusing on generating a single step of the pipeline. Additionally, we explored two different solution designs for utilizing LLMs:

1. Direct Single LLM Calls: In this approach, we send the user command along with the relevant context (e.g., stage type, pipeline schema) in a single request to the LLM under evaluation.
2. Agentic Framework Approach: This approach leverages an agentic framework to distribute sub-tasks — such as context generation, schema verification, and step generation — among multiple AI agents. We implemented this framework using AutoGen.

Performance Metrics: How We Measure Success

In this blog post, we focus on the generation use case — specifically, creating pipeline steps, stages, and related configurations — and introduce the metrics used to evaluate the performance of different models for this task. Our evaluations are conducted against a benchmark dataset with a known ground truth. Specifically, we have curated a dataset consisting of user commands for creating pipeline steps and their corresponding YAML configurations. Using this benchmark data, we have developed a set of metrics to assess the quality of AI-generated YAML outputs in response to user prompts.

Since we are evaluating AI-generated pipelines against known, predefined pipelines, the comparison ultimately involves measuring the differences between two YAML files. To accomplish this, we leverage and build upon DeepDiff, a framework for computing the structural differences between key-value objects. DeepDiff is conceptually inspired by Levenshtein Edit Distance, making it well-suited for quantifying variations between YAML configurations and assessing how closely the generated output matches the expected pipeline definition.

At its core, DeepDiff quantifies the difference between two objects by determining the number of operations required to transform one into the other. This difference is then normalized to produce a similarity score between 0 and 1, providing a structured way to compare data. While we utilize the standard DeepDiff library as one of our evaluation metrics, we have also developed two modified versions tailored specifically for comparing step YAMLs. These adaptations address the unique challenges of our use case, ensuring a more precise and meaningful assessment of AI-generated pipeline configurations.

In particular, we have introduced:

• DeepDiff 2: This metric first applies schema verification before computing the similarity score, assigning a score of zero if the generated YAML fails validation. Additionally, it does not penalize differences in optional fields such as name, identifier, and description, ensuring that minor variations do not disproportionately impact the similarity score. Moreover, as long as the generated solution adheres to schema validation, this metric allows additional keys in the step without penalizing the score.
• DeepDiff 3: This metric builds upon DeepDiff 2 but introduces a penalty for any additional key that does not exist in the reference solution. This stricter approach provides a more precise comparison to the ground truth, considering that extra keys with default values may impact the user experience. Users may not expect to see default values for optional fields in the UI, making it essential to account for such differences in evaluation.

Benchmarking LLMs: Evaluating the Leading Models

Benchmark Dataset

Let’s first introduce the benchmark data used for this study.

At Harness, our QA team generates numerous sample pipelines using automation tools such as APIs and Terraform Providers to simulate customer use cases and various Harness configurations. These pipelines play a crucial role in sanity testing, ensuring that when a new version of Harness is released, all steps, stages, and pipelines continue to function as expected.

For this study, we leveraged this data to create a benchmark dataset of 115 step YAMLs. For each example, we manually added a potential user command that could generate the corresponding step. The same user command was then used to generate a step YAML using an LLM. The AI-generated solutions were subsequently compared against the original YAML file to evaluate accuracy and quality.

Below is an example of a user command and its corresponding YAML file, which serves as the ground truth in our evaluation:

User Command:“Please add a Terraform plan step to the pipeline.”

Ground Truth YAML:

This YAML structure represents the expected output when an LLM generates a pipeline step based on the given user command. The AI-generated YAML will be evaluated against this reference to assess its accuracy and quality.

Models Compared

We evaluated both an agentic framework and direct model calls for utilizing LLMs in pipeline generation. The selection of models for each approach was based on the technical adaptability of the frameworks we used. For example, AutoGen supports only a limited set of LLMs, which influenced our model choices for the agentic framework.

As a result, there isn’t a one-to-one correspondence between the models used in the agentic framework and those used in direct calls. However, there is significant overlap between the two sets.

Agentic Framework: Models operating within an agent-driven setup

• GPT-4o
• O3-mini-medium
• Claude-3.7

Direct Model Calls: Models queried directly without an agentic framework

• GPT-4o
• O3-mini-medium
• Claude-3.7
• DeepSeek R1
• DeepSeek V3

This comparison allows us to assess how different models and methodologies perform in generating high-quality DevOps pipeline configurations.

Results

The figure below illustrates the performance of each model based on the three evaluation metrics introduced earlier. Models that are called using an agentic framework are prefixed with “Autogen_” in the results.

Our findings indicate that using an agentic framework significantly improves response quality across all three metrics. However, AutoGen does not yet support DeepSeek models, so for these models, we only report their performance when called directly.

LLM Performance Comparison for Pipeline Step Generation

In order to gain deeper insights into the scores, we also visualize the number of samples that failed the schema verification step, where a zero score is assigned to such cases. This highlights instances where models struggle to generate valid YAML structures:

Schema Verification Failures Across Models

The plot above clearly demonstrates the effectiveness of an agentic framework with a dedicated schema verification agent. Notably, none of the models within the agentic framework produced outputs that failed schema validation.

Takeaways

Our evaluation of LLMs for DevOps automation provided valuable insights into their strengths, limitations, and practical applications. Below are some key takeaways:

• LLMs demonstrate strong potential for automating DevOps workflows, particularly in generating pipeline YAMLs from user commands — achieving a pass rate of over 95% for the best models. This reduces manual effort, increases efficiency, and streamlines software delivery.
• Leveraging an agentic framework that breaks tasks into smaller sub-tasks and distributes them among sub-agents significantly improves accuracy. This approach reduces schema verification failures and minimizes model hallucinations, leading to more reliable and structured pipeline generation.

DevOps Audit Trails

Audit trails are important for maintaining regulatory compliance, ensuring security, and improving operational efficiency. This blog post will discuss why audit trails are crucial, how they are implemented within Harness, and the various benefits they offer. 

An audit trail is a chronological set of records documenting activity changes made to a system or data. In Harness, the audit trail displays a record of each event that changes the setup of your Harness account, modules, or entities. Users can view the audit trail data, where each event record displays the date, time, user, and action (created/changed/deleted). It also provides information on the resource, Harness entity affected, project, module, and event summary with the YAML difference. Users can filter the audit records by criteria such as user, organization, project, resource, and action, and can exclude events like 2FA and unsuccessful login attempts or system events.

Benefits of Audit Trail

Audit trails provide a comprehensive record of system and user activities, which are essential for several key areas:

Security

Audit trails play a crucial role in detecting security violations. By maintaining detailed records, they ensure compliance with defined regulations and restrictions. These records help identify security breaches, ensure data integrity, and monitor unauthorized access. Additionally, audit trails assist in detecting internal fraud by tracking user actions and changes to sensitive data. In the event of a security breach, audit logs provide vital information for investigating the issue, understanding its impact, and preventing future occurrences. Within Harness, audit trails capture all actions taken within the account, such as user logins, configuration changes, and deployments. This helps identify unusual activities and potential security threats, offering a robust mechanism for tracking and responding to security incidents.

Fraud Prevention

Audit trails can detect and prevent both internal and external fraud by closely monitoring user actions and changes to sensitive data. They uncover discrepancies and enforce controls to reduce the potential for cybersecurity breaches. By logging every action and change, audit trails make it easier to identify unusual patterns that may indicate fraudulent activity. This comprehensive tracking ensures that any signs of fraud are promptly detected and addressed, helping to maintain the integrity and security of the system.

Accountability

Audit trails hold users accountable for their actions by recording who made changes and when. This promotes responsible behavior and helps managers understand the flow of activities, which is crucial for maintaining operational integrity. In the Harness platform, every change is logged with detailed information about the user, action, resource, and context, ensuring that team members are accountable for their actions and reducing the risk of unauthorized activities.

Monitoring user activity is also key to individual accountability. By tracking user access, audit trails ensure that only authorized users can perform sensitive operations. Analyzing user actions and patterns is valuable not only for detecting suspicious behavior but also for ensuring that Role-Based Access Control (RBAC) is properly implemented. This allows administrators to monitor who accessed what information and when, ensuring that authorized users have access to the resources they need to perform their jobs effectively.

User Activity Monitoring

Monitoring user access ensures that only authorized individuals can perform sensitive operations. By analyzing user actions and patterns, audit trails help detect suspicious behavior and verify that Role-Based Access Control (RBAC) is properly implemented. This allows administrators to monitor who accessed specific information and ensures that users have the appropriate level of access to perform their jobs effectively.

Troubleshooting

Audit trails are also invaluable for resolving issues by providing a chronological record of actions. When troubleshooting a system failure, audit trails can help identify the root cause of the problem and distinguish between user errors and system failures. At Harness, the audit trail feature offers users detailed logs of all actions taken, making it easier to trace back steps. This reduces the time required for diagnosing and fixing issues, ultimately enhancing the system’s reliability and performance.

Change Management

Audit trails can track changes to configurations, ensuring that any unauthorized actions are identified immediately. By monitoring changes to code and deployments throughout the development cycle, audit trails promote accountability. Every modification to the deployment pipeline, configurations, resource allocations, and more is recorded, maintaining a clear and detailed history of what was changed, by whom, and when. This leads to improved change management and helps prevent issues before they escalate.

Legal Discovery and Regulatory Investigations

Audit trails create a chain of evidence, revealing the root source of security breaches and documenting the chain of custody for how files were altered. These logs provide a verifiable and transparent record of all actions within a system, ensuring accountability and helping trace issues back to their origin. The detailed record of actions, including who performed them and when, is invaluable in resolving legal disputes and regulatory investigations. By maintaining a clear trail of evidence, audit trails help ensure accountability and support thorough investigations into any security incidents.

Disaster Recovery

Audit trails ensure that records are securely backed up and can be recovered in the event of a crisis. In disaster recovery efforts, having an audit trail is crucial for maintaining business continuity and data integrity. Harness’s audit logs provide a detailed record of actions and changes, which can be used for recovery and analysis if a system failure occurs. This enables a faster and more efficient restoration of operations to their pre-disaster state.

Operational Efficiency

Audit trails provide visibility into the progress and changes of documents and tasks, enhancing workflow efficiency. They track the status of projects, offering users transparency and enabling teams to stay informed about any changes or progress. This visibility allows for easy monitoring of ongoing tasks, ensuring that project deadlines are met and any delays are promptly addressed, ultimately optimizing workflow efficiency.

Error Prevention

Audit trails track actions to identify errors promptly, reducing the chances of repeated mistakes. Knowing that their actions are consistently logged incentivizes users to be more careful, reducing errors altogether.

Legal Compliance

For many organizations, maintaining audit trails is a legal requirement. For example, in the United States, HIPAA mandates that healthcare organizations maintain and regularly review secure audit-trail logs for access to electronic protected health information (ePHI) for at least six years to ensure data integrity and traceability. Similarly, the Sarbanes-Oxley Act (SOX) requires public companies to retain accurate and complete audit-trail logs related to financial reporting for a minimum of seven years to ensure compliance and prevent corporate fraud. These regulations ensure that companies provide verifiable records of all activities, in accordance with legal and regulatory standards.

‍

Using Harness Audit Trails

Now let’s dive into how to access Audit Trails. 

‍

1. Click on the Account Settings option. Go under Account Settings located under Account Overview. 

‍

‍

2. Click on ‘Audit Trail’ under Security and Governance. 

‍

3. This should display an Audit Trail. 

‍

‍

Harness Audit Trail Features

‍

1. Time Range Selection

‍

You can filter audit logs based on different time ranges as shown in the dropdown menu. You can include today, yesterday, past 7 days, or select a customized date range in the calendar view. This helps pinpoint specific periods to review logs. 

‍

2. Basic Event Filtering

‍

‍

Users can choose to exclude specific types of events like ‘Login Events’ or ‘System Events’ to eliminate clutter/pin their focus on particular events. The purpose of this is to concentrate on specific actions.

Exclude Login Events: This removes all login-related activities from the log. For example, any user who is logging in/simply accessing.

Exclude System Events: This removes any system-generated actions like automated updates/notifications.

Then, you can review the logs to focus on the remaining events.

‍

3. Filter Choices Based on Criteria

‍

This feature offers multiple fields for creating specific filters based on criteria such as User, Organization, Project, Resource Type, and Action. These filtering options enable users to focus on particular activities within the system to monitor important events. By using these detailed filters, users can more quickly identify and investigate the root causes of issues. It also simplifies the process of reviewing events according to specific criteria.

User: Select specific user/users whose events you’d like to filter.

Organization: Choose the organization related to events you’re interested in.

Project: Choose the projects to filter actions within the project context.

Resource Type: Select the type of resource by the actions.

Action: Select specific actions (updated/created/deleted) to filter events.

Then, Click on Apply to display the events based on your new filtration choices.

‍

‍

This also allows for multiple selections of the resource type for the new filter. You can also allow yourself/others to view and edit the filters you create.

‍

4. Audit Logs:

‍

‍

Audit logs provide a comprehensive record of all activities and events within the Harness.io account. They enhance transparency, support accountability, and bolster security by recording the time of occurrence, the type of action, the affected resources, and the identity of the user involved. These logs are valuable for identifying any unauthorized access and serve as a historical record for tracking changes to configurations, pipelines, and other resources. They also assist in troubleshooting by helping users quickly identify the root cause of issues. Additionally, users can easily view role assignments, resource updates, and new user invitations.

‍Time: Displays the exact timestamp of when the event occurred.

User: The user who acted (taarini.dang@harness.io).

Action: Description of the action (created/updated/deleted).

Resource: Resource affected by the action (Shared Folder, Module).

Organization: Project/Organization affected by the action.

Module: Specific module where the action took place (Dashboard Folder, Service). 

‍

5. Audit Log Streaming

Audit Log Streaming allows users to continuously stream audit logs from Harness.io to an external destination (Amazon S3, SIEM systems, etc). This enables the audit log data to be available for real-time analysis and long-term storage. This helps with the immediate detection of suspicious events. Streaming logs to a centralized SIEM allows for advanced security analytics. It ensures audit logs are stored securely for extended periods. 

Harness will retain your audit data for two years, but you can configure a streaming destination in Harness to send audit log data to another location for processing. You can integrate this data with SIEM tools for more security and compliance.

‍

Create a New Streaming Destination:

‍

‍

Name it:

‍

Choose Connector Options:

‍

How quickly can your team deliver a new project?

Why can’t you take on new work?

How often are we deploying?

How quickly are we able to recover from production issues?

These are all questions that engineering leaders must discuss with management, as well as internally. Engineering organizations are increasingly expected to deliver software more frequently and with better quality. To meet this demand, software teams are embracing Agile development methodologies and DevOps to varying degrees. As they try to get more efficient in their agile practices, software delivery teams need software metrics to measure their development speed and stability. 

The complexity of distributed teams, outsourced projects and remote work makes it even more critical to have the right metrics defined to measure the overall DevOps performance.

Dora Metrics: The 4 Key Indicators of Elite DevOps Performance 

There are 4 key metrics that many organizations use to measure their delivery performance. These metrics were suggested by the DevOps Research & Assessment (DORA) research program, and hence are known as DORA metrics.  The 4 DORA Metrics are:

Software Delivery Lead Time

Software Delivery Lead Time (also known as Lead Time for Changes) is the amount of time it takes a commit to get into production - from the time the software development team starts working on a task to the time the customer gets the feature. This includes the time spent in build, test, integration, implementation and deployment, and does not include the time spent in design, prioritization or backlog

Deployment Frequency

Deployment Frequency is the number of times code or software is deployed to production or “shipped”. This metric helps organizations determine and set their delivery cadence.

Mean Time To Restore (MTTR) 

MTTR is the time it takes to restore a failure in production. A failure can be an unplanned outage or a service failure.

Change Failure Percentage

This is the percentage of deployment causing a failure in production. It is the measure of the number of times “a hotfix, a rollback, a fix-forward, or a patch” is required after a software deployment or a service change.

‍

How to Measure DevOps Success: Why DORA Metrics Are Important

The four DORA metrics provide a great baseline to measure the tempo, rhythm and responsiveness of an engineering organization.

A.  Development Velocity

The first two metrics are a measure of software delivery performance “tempo”, also known as Development Velocity. It is important for organizations to understand their development velocity. 

Lead Time

Lead Time helps organizations understand how quickly they can deliver software. It gives you a sense of the efficiency of the development teams. With shorter lead times, you can deploy to production in smaller deployments and more often. This enables faster feedback on what is getting built and allows for quicker course correction. Conversely, longer lead times signify bottlenecks in the development process.

Lead Time is a great metric to track, especially if looking at the trend over time. It shows whether there are any issues, and if things are getting better or worse. And at the end of the day, how quickly you are able to respond to the business. However, what is more important is to get further breakdown of the different stages. 

• Are there any bottlenecks? What stage are they in? 
• Are PRs taking long to Review? If so, why? 
• What is the Merge Time? 
  

These are all good questions to ask. With a combination of data from the Project Management, SCM and the CI-CD and Deployment systems, this breakdown can be achieved.

Deployment Frequency

Deployment frequency helps organizations set their release schedules. Many SaaS organizations chose to deploy builds frequently - some even on a daily basis. Smaller, more frequent builds help reduce risk. However, not every organization will want to or need to deploy very quickly or frequently. SaaS organizations may need to deploy very frequently. On the other hand, for certain business applications, deployment frequency of once or twice a year might be sufficient - their customers may not be happy with frequent changes. 

‍

‍

Software teams should evaluate the needs of their business and ensure that the velocity of their development process (the Lead Time and Deployment Frequency) matches their business need. The principles of Lean and Agile can still be applied by delivering software in small batches rather than delivering as large monoliths.

B.  Software Quality

Software development organizations often put a lot of emphasis on improving the development velocity. However, perhaps even more important is the quality of their output. 

Change Failure Percentage

Change Failure rate can give organizations a sense of how frequently they are shipping out code that causes issues. Ideally, the Change Failure Percentage should be as low as possible indicating good quality code.

Mean Time To Resolution (MTTR)

MTTR is the time it takes to restore a failure in production. A failure can be an unplanned outage or a service failure.

‍

The above 2 metrics measure the reliability and stability of the software that is delivered. Together, these are a good indicator of the quality of the development output.

As teams grow, it is critical to find a balance between how much and how often to deploy vs how stable the product is. A higher development velocity is important but should not come at the expense of quality or the stability of the delivered software.

What is a DORA Report?

Engineering leaders need to be able to evaluate the performance of their organizations on an ongoing basis. The DORA report is a great way to start getting some initial insight into the development velocity and software quality. With these metrics, you can start to see if there are any bottlenecks in the development process, and the quality of their output.

It is however important to not view the numbers as absolute. Being able to track DORA metrics on a regular basis helps you see the trends - this could be a better indicator of issues.

The DORA report measures five key metrics:

• Deployment frequency
• Lead time for changes
• Mean time to recovery
• Change failure rate

The DORA report finds that high-performing organizations are able to deploy software more frequently, with shorter lead times, and with lower change failure rates. High-performing organizations also have better operational performance, meaning that they are able to resolve incidents more quickly and with less impact on their customers.

‍

Getting Started with DORA Metrics

Accurately measuring DORA metrics can be a challenge for most organizations. Much of the data that is needed to calculate these DevOps metrics lies in various systems across the DevOps toolchain - project management, SCM, CI/CD, service desk, issue tracking, and other systems. The data must be parsed, broken into spreadsheets, and then correlated to get the right DORA metrics.

As an example, consider an organization that uses Jira for their planning, GitHub for SCM, Harness for CI/CD, ServiceNow for service desk, PagerDuty for production monitoring, and various other tools for testing, security, etc. This is a very common scenario in many organizations where they select different tools that meet their needs for different purposes.

To get a metric such as Lead Time, you need to correlate the data from Jira with data from GitHub and Harness, so you can accurately understand how much time was required from the start of the task till it made it to Production.

Similarly, for accurate MTTR, you need to correlate data from PagerDuty back to ServiceNow and then to Jira. This can be quite a challenge.

Beyond DORA: Other Agile DevOps Metrics 

In addition to the 4 DORA metrics, there are several other metrics that can help engineering teams determine their efficiency, productivity and their bottlenecks. Some of these are:

PM Hygiene

• Requirements Clarity
• Sprint Distribution
• Details of User Stories 
• Prioritization
• Acceptance Criteria & Test Cases
  

Sprint Hygiene

• Story Point Calculation
• Estimation Accuracy
• Story Point Mix
• Dependencies & Interactions
• Distribution of workload between teams
  

PR Lifecycle

• Time to First Commit,
• Time to Review,
• Time to merge,
• Number of reviewers
• Items stuck in various stages
  

CI-CD Effectiveness

• Number of failed builds
• Number of failed deployments
• Number of rollbacks
• Number of hotfixes
  

Incident Management

• Number of Hops
• Bounce backs
• Time spent on resolution
  

Code Hotspots, Technical Debt, Security Considerations and a lot more.

Summary

Metrics are the foundation to understanding the efficiency and effectiveness of your engineering organization. With the right software metrics, you can make data-driven decisions and demonstrate alignment with the business toward customer-centric outcomes.

DORA metrics provide a good foundation to start measuring development velocity and software quality. Tracking DORA metrics regularly helps you see trends and point out problem areas. However, DORA metrics can be hard to obtain since data resides in different tools deployed across the DevOps toolchain. You need to correlate data from various sources such as GitHub, Jira, PagerDuty, etc, which can be difficult, time-consuming and frustrating.

Accelerating DevOps with DORA Metrics and Harness 

The Harness AI DLC Insights module can help organizations solve this challenge. AI DLC Insights enables engineering leaders to proactively manage and demonstrate the performance of the engineering organization to business stakeholders with established success metrics, such as DORA and the SPACE Framework. With over 40 out-of-box integrated data sources (including GitHub, GitLab, BitBucket, Jira, PagerDuty, and Azure), you can find and remove software delivery bottlenecks while gaining a better understanding of engineering operations. 

Interested in learning more about how Harness AI DLC Insights can help improve your engineering outputs? Request a demo. 

‍

Learn more: What Is MTTR?: The DORA Metric You Need To Know

Checkout Best Practices for Governance in DevOps