DevOps Solutions: How to Pick the Right Stack for Your Team | Harness Blog

In the last decade, application DevOps has revolutionized, and automated testing, continuous integration, and seamless deployment pipelines have become standard practice. But when it comes to Database DevOps, progress has lagged. Migrations still require manual scripts, schema changes often rely on human-crafted SQL Scripts, and production rollouts are high-risk events.

Harness has brought AI-native automation to the last mile of DevOps - the database. With AI-Powered Database Migration Authoring, developers can now describe schema changes in natural language and instantly receive compliant, production-ready migrations complete with rollback and governance

That’s where AI comes in. By harnessing natural language processing, intelligent schema parsing, and predictive analytics, AI can now generate, modify, and optimize database changes automatically - cutting down delivery times and reducing the risk of human error.

In this blog, we’ll explore the emerging role of AI in Database DevOps, showcase real-world AI-powered tooling like the AI Changeset Generator, and discuss how teams can prepare for a future of autonomous change management.

The Challenges of Traditional Database DevOps

Despite significant advances in DevOps culture, the database remains one of the least automated components of modern software delivery. Common challenges include:

1. Manual Changelog Authoring - Writing migration changelogs by hand is time-consuming, requires deep syntax knowledge, and is prone to typos or semantic errors.
2. Slow Feedback Loops - Developers often wait for DBAs to review changes, creating bottlenecks that slow the entire CI/CD pipeline.
3. High Risk of Production Failures - A single incorrect migration can bring down critical systems, and rollback scripts are often an afterthought.
4. Limited Tool Intelligence - Popular open-source tools like Liquibase OSS are excellent for structured change tracking, but they lack native AI capabilities, meaning the developer is still responsible for authoring every migration.
5. Complex Multi-Environment Management - Coordinating schema changes across dev, staging, and production environments introduces drift, conflicts, and unpredictable behaviors.

These pain points become especially acute in AI -driven projects where schema adjustments can be needed multiple times a day.

How AI is Reshaping Database DevOps ?

The introduction of AI into Database DevOps workflows unlocks entirely new capabilities:

• Natural Language to Changelog - Describe a schema change in plain English,  e.g., “Add a column named email to the users table”,  and get a production-ready changeset instantly.
• Context-Aware Suggestions -  AI can analyze the existing schema and changelog history to ensure new migrations are consistent and avoid conflicts.
• Environment-Specific Changes - Target changesets for dev, staging, or prod environments using Liquibase-style contexts, ensuring precise deployments.
• Predictive Rollback Strategies - LLM can suggest the most likely rollback steps in case of a failed deployment.
• Automated Compliance Checks - AI can flag non-compliant changes before they reach production, helping meet regulatory requirements without extra manual review.

These capabilities allow teams to move away from reactive, manual processes and toward proactive, automated, and safer database change management.

AI-Powered Database Migration Authoring in Harness Database DevOps

AI-Powered Database Migration Authoring within Harness Database DevOps :

1. Describe changes in plain language. The AI interprets your intent and generates precise, production-ready migrations instantly.
2. Augment existing changelogs automatically. Harness AI analyzes your existing migration history from Git and seamlessly appends new changes in the correct order, no manual copy-paste required.
3. Target specific environments. Generate migrations with Liquibase style contexts like dev, staging, or prod for selective and controlled rollouts.
4. Stay CI/CD-ready. Every AI-authored migration aligns with Harness pipelines, governance policies, and deployment workflows for end-to-end reliabilit

Ready for CI/CD - The generated changelogs are fully compatible with modern pipelines, enabling automated deployments without extra formatting work.

Example Workflow:

• Step 1 - Developer describes a change in natural language in Harness.
• Step 2 - Harness AI analyzes the schema and governance rules.
• Step 3 - Generates a compliant migration with rollback.
• Step 4 - Test and Preview the generated changelog with the new changeset tagged specifically for production.

In the above example:

• The left panel is where you describe the desired change and optionally add existing changelog content.
• The right panel displays the AI-generated, ready-to-use changeset.

Comparison with Traditional OSS Tools

Unlike Liquibase/Flyway, which require manual authoring and lack governance context, Harness embeds AI generation, rollback, and policy validation directly into the CI/CD pipeline This means:

In other words, AI doesn’t replace the robust migration frameworks we know and trust. it helps user by empowering them with easy.

The Road Ahead – Fully Autonomous Database DevOps

The next frontier is self-managing databases, where AI doesn’t just write migrations - it deploys them, monitors for issues, and rolls back or fixes changes automatically.

We envision:

• Continuous Schema Learning - AI models that adapt to your organization’s coding and data patterns over time.
• Self-Healing Deployments - Automatic rollback or schema patching when anomalies are detected in production.
• Integrated Data Governance - AI that ensures every schema change aligns with security, compliance, and business rules in real time.

This isn’t science fiction - the building blocks already exist, and tools like the Harness AI-Powered Database Migration Authoring are the first step toward that reality.

Conclusion

Harness Database DevOps turns the database from a bottleneck into an accelerator. With AI-Powered Database Migration Authoring, every change is safe, compliant, and fully auditable bringing the same automation and confidence to the database that CI/CD brought to applications.

If you’re ready to experience the benefits firsthand, try the Harness Database DevOps

Frequently Asked Questions

1. Can the AI Changeset Generator work with my existing changelog files?‍

Absolutely. You can paste your current changelog content into the tool, and the AI will intelligently insert the new changeset while preserving your existing migration history. This prevents duplication and ensures that the migration order remains intact.

2. How does environment-specific changeset generation work?

When you specify a target environment (for example- dev, staging, or prod), the tool automatically adds a context attribute to the changeset. This enables selective execution of migrations depending on the deployment target, ensuring that environment-specific changes don’t inadvertently impact other environments.

3. What database technologies are supported?‍

The generator produces changesets in a Liquibase-compatible format (YAML/XML/JSON/SQL), making it suitable for any database that Liquibase OSS supports. If you’re already using Liquibase OSS, you can drop the generated files directly into your workflow with minimal setup.

4. Does the tool validate or detect conflicts in changelogs?‍

Yes. When working with an existing changelog, the AI will scan for similar changes (e.g., duplicate table creation, repeated column additions) and adjust the new changeset to avoid conflicts. This makes it safer to iterate in multi-developer environments where changes are happening in parallel.

5. Does Harness support governance and rollback automatically?

‍Yes. Every AI-authored migration includes rollback logic and passes through governance checks before deployment.

‍

Request a demo

Learn more: Harness in Seattle at PASS Data Community Summit 2025

DevOps Meets AI: Evaluating the Performance of Leading LLMs

Modern DevOps processes are essential for ensuring efficient, reliable, and scalable software delivery. However, managing infrastructure, CI/CD pipelines, monitoring, and incident response remains a complex and time-consuming challenge for many organizations. These tasks require continuous tuning, configuration management, and rapid troubleshooting, making DevOps resource-intensive. As software systems grow in complexity, manual intervention becomes a bottleneck, increasing the risk of human error, inefficiencies, and slower deployments. This is where automation becomes a necessity, helping teams streamline workflows, reduce operational overhead, and improve deployment velocity.

‍

The rise of artificial intelligence, particularly large language models (LLMs), has opened new possibilities for automating various aspects of software development and operations. By leveraging AI, organizations can enhance efficiency, reduce manual effort, and accelerate software delivery. LLMs bring the potential to transform DevOps by enabling intelligent automation, improving decision-making, and making systems more adaptive to changing requirements.

Our AI engineering team has been at the forefront of integrating AI into DevOps workflows. From AI-powered CI/CD optimizations to intelligent deployment strategies, we continuously explore ways to leverage AI for greater efficiency. In this blog, we share our journey in evaluating LLMs for DevOps automation, benchmarking their performance, and understanding their impact on software delivery workflows.

Harnessing LLMs for DevOps Automation

Before diving into the evaluation, let’s first outline the specific problem we aim to solve using large language models. (Note: In this post, I won’t go into the underlying architecture of the Harness AI DevOps Agent — stay tuned for a future blog post on that!)

Our exploration begins with the task of pipeline generation. Specifically, the AI DevOps Agent takes a user command describing the desired pipeline as input, along with relevant context information. The expected output is a pipeline YAML file generated by the AI DevOps agent, which is composed of multiple sub-agents, automating the configuration process and streamlining DevOps workflows. An example user command and the resulting YAML pipeline would be:

“Create an IACM pipeline to do create a IACM init and plan”

Response:

For simplicity, we conducted the first phase of our evaluations by focusing on generating a single step of the pipeline. Additionally, we explored two different solution designs for utilizing LLMs:

1. Direct Single LLM Calls: In this approach, we send the user command along with the relevant context (e.g., stage type, pipeline schema) in a single request to the LLM under evaluation.
2. Agentic Framework Approach: This approach leverages an agentic framework to distribute sub-tasks — such as context generation, schema verification, and step generation — among multiple AI agents. We implemented this framework using AutoGen.

Performance Metrics: How We Measure Success

In this blog post, we focus on the generation use case — specifically, creating pipeline steps, stages, and related configurations — and introduce the metrics used to evaluate the performance of different models for this task. Our evaluations are conducted against a benchmark dataset with a known ground truth. Specifically, we have curated a dataset consisting of user commands for creating pipeline steps and their corresponding YAML configurations. Using this benchmark data, we have developed a set of metrics to assess the quality of AI-generated YAML outputs in response to user prompts.

Since we are evaluating AI-generated pipelines against known, predefined pipelines, the comparison ultimately involves measuring the differences between two YAML files. To accomplish this, we leverage and build upon DeepDiff, a framework for computing the structural differences between key-value objects. DeepDiff is conceptually inspired by Levenshtein Edit Distance, making it well-suited for quantifying variations between YAML configurations and assessing how closely the generated output matches the expected pipeline definition.

At its core, DeepDiff quantifies the difference between two objects by determining the number of operations required to transform one into the other. This difference is then normalized to produce a similarity score between 0 and 1, providing a structured way to compare data. While we utilize the standard DeepDiff library as one of our evaluation metrics, we have also developed two modified versions tailored specifically for comparing step YAMLs. These adaptations address the unique challenges of our use case, ensuring a more precise and meaningful assessment of AI-generated pipeline configurations.

In particular, we have introduced:

• DeepDiff 2: This metric first applies schema verification before computing the similarity score, assigning a score of zero if the generated YAML fails validation. Additionally, it does not penalize differences in optional fields such as name, identifier, and description, ensuring that minor variations do not disproportionately impact the similarity score. Moreover, as long as the generated solution adheres to schema validation, this metric allows additional keys in the step without penalizing the score.
• DeepDiff 3: This metric builds upon DeepDiff 2 but introduces a penalty for any additional key that does not exist in the reference solution. This stricter approach provides a more precise comparison to the ground truth, considering that extra keys with default values may impact the user experience. Users may not expect to see default values for optional fields in the UI, making it essential to account for such differences in evaluation.

Benchmarking LLMs: Evaluating the Leading Models

Benchmark Dataset

Let’s first introduce the benchmark data used for this study.

At Harness, our QA team generates numerous sample pipelines using automation tools such as APIs and Terraform Providers to simulate customer use cases and various Harness configurations. These pipelines play a crucial role in sanity testing, ensuring that when a new version of Harness is released, all steps, stages, and pipelines continue to function as expected.

For this study, we leveraged this data to create a benchmark dataset of 115 step YAMLs. For each example, we manually added a potential user command that could generate the corresponding step. The same user command was then used to generate a step YAML using an LLM. The AI-generated solutions were subsequently compared against the original YAML file to evaluate accuracy and quality.

Below is an example of a user command and its corresponding YAML file, which serves as the ground truth in our evaluation:

User Command:“Please add a Terraform plan step to the pipeline.”

Ground Truth YAML:

This YAML structure represents the expected output when an LLM generates a pipeline step based on the given user command. The AI-generated YAML will be evaluated against this reference to assess its accuracy and quality.

Models Compared

We evaluated both an agentic framework and direct model calls for utilizing LLMs in pipeline generation. The selection of models for each approach was based on the technical adaptability of the frameworks we used. For example, AutoGen supports only a limited set of LLMs, which influenced our model choices for the agentic framework.

As a result, there isn’t a one-to-one correspondence between the models used in the agentic framework and those used in direct calls. However, there is significant overlap between the two sets.

Agentic Framework: Models operating within an agent-driven setup

• GPT-4o
• O3-mini-medium
• Claude-3.7

Direct Model Calls: Models queried directly without an agentic framework

• GPT-4o
• O3-mini-medium
• Claude-3.7
• DeepSeek R1
• DeepSeek V3

This comparison allows us to assess how different models and methodologies perform in generating high-quality DevOps pipeline configurations.

Results

The figure below illustrates the performance of each model based on the three evaluation metrics introduced earlier. Models that are called using an agentic framework are prefixed with “Autogen_” in the results.

Our findings indicate that using an agentic framework significantly improves response quality across all three metrics. However, AutoGen does not yet support DeepSeek models, so for these models, we only report their performance when called directly.

LLM Performance Comparison for Pipeline Step Generation

In order to gain deeper insights into the scores, we also visualize the number of samples that failed the schema verification step, where a zero score is assigned to such cases. This highlights instances where models struggle to generate valid YAML structures:

Schema Verification Failures Across Models

The plot above clearly demonstrates the effectiveness of an agentic framework with a dedicated schema verification agent. Notably, none of the models within the agentic framework produced outputs that failed schema validation.

Takeaways

Our evaluation of LLMs for DevOps automation provided valuable insights into their strengths, limitations, and practical applications. Below are some key takeaways:

• LLMs demonstrate strong potential for automating DevOps workflows, particularly in generating pipeline YAMLs from user commands — achieving a pass rate of over 95% for the best models. This reduces manual effort, increases efficiency, and streamlines software delivery.
• Leveraging an agentic framework that breaks tasks into smaller sub-tasks and distributes them among sub-agents significantly improves accuracy. This approach reduces schema verification failures and minimizes model hallucinations, leading to more reliable and structured pipeline generation.