How Harness Ships Harness to Our Users: A Look Inside Our Release Process
At Harness, we are committed to delivering high-quality software with every release. Quality isn’t just a final step — it’s embedded into every stage of our development and release pipeline to ensure a seamless, reliable, and efficient user experience.
In this post, I’ll give you an exclusive look into our quality and release process. From well-defined development cycles to thorough validation checks and comprehensive testing strategies, here’s how we ensure the highest standards are met before deploying code to production.
A Day in the life of a Harness developer
- Coding — Developers write and test code locally.
- Local Testing — Developer test their change in the local environment.
- Build Service Image (Dev image) -They build the image using the Harness CI tool.
- Deploy Code — The image is deployed to a personal space using the Harness CD tool.
- Sanity Testing — Relevant sanity suites are run post-deployment.
- PR Sanity Checks — Once validated, the developer merges changes via a PR, triggering our Smart PR Check pipeline.
- Code Merge — Once the changes are validated and passed they will merge into the Develop branch in the harness code repo
- Develop Branch — Tested codes are merged into to develop branch
- Build Service Image (Snapshot testing image) -We build the images using the Harness CI tool for snapshot testing.
- Snapshot Environments: From the development, we leverage Snapshot Builds, an automated process that runs every 6 hours.
- PR Sanity Checks — Snapshot builds will go through the automated sanity testing
- Release branch cut: From the develop we will create a new Release branch cut every week and that is called system release.
Smart PR Check Pipeline: Enhancing Code Quality & Efficiency
We replaced individual PR-check pipelines with Smart PR Checks powered by GitExperience to optimize resources and cost.
PR Check Workflow
This pipeline includes:
✅ Common Checks — The Common Checks consist of essential PR Checks that are required to pass regardless of the changes made in the file. There are mandatory conditions for every commit, including detecting merge conflicts and ensuring the latest changes are considered.
✅ Build Number Verification — Verifies if the build number and patch version in a build.properties file are correctly incremented based on the target branch and raises errors if they are not.
✅ Recency Check — This stage checks for breaking commits in a harness repository verifies the recency of unmerged commits currently set for 3 days and raises errors if necessary.
✅ Feature Flag Validation/ Check Feature — In this stage, we download a patch for a pull request, check for the addition of a new feature flag, parse a Jira case from the commit message, retrieve the feature flag name from the Jira case, and raise errors if necessary.
✅ File Permission & Copyright Checks — Verifies no unintended changes in file permissions and copyright statements.
✅ CompileCheck — This stage checks if compilation is required for the files modified in a Git merge request, and if not, it skips certain checks and sends success statuses to Harness code repo.
✅ Security & Compliance — GitLeaks prevents secrets from getting committed.
Smart PR Check Pipeline Studio View In Harness CI Tool
✅ Unit Test Intelligence -
Harness Unit Test Intelligence (JavaUnitTests*) uses CallGraph analysis and dynamic instrumentation to select tests to run based on changed files in a Pull Request (PR). It’s a well-structured chained pipeline. By examining the trigger ID, you can verify that SmartPR (parent job) initiates the execution of the UT (chained pipeline). This thoughtful design enables us to effortlessly track and analyze the cost attributed to the UT pipeline individually.
✅ Static Code Analysis & Code Formatting — PMD, Checkstyle, Sonar Scan, and Code Format validations ensure high coding standards. The script utilizes a static source code analyzer, specifically the PMD (Project Mess Detector) tool, to analyze the source code for potential issues and provide feedback on coding practices, code quality, and possible bugs.
✅ MessageCheck — This message-check.sh stage validates if a commit message in a Pull Request (PR) title follows a specific format and displays an error message if it doesn’t.
✅ CodebashHashCheck — This stage runs codebase-hash-check.sh to check for changes in a codebase hash during a Git merge, generates error messages if there are discrepancies, and provides instructions for backward compatibility and resolving the issue.
✅ CodeFormatCheckStyle
- Checkstyle → It will initiate a Bazel build and run tests for designated modules while also enabling the Checkstyle tool for code style checking and enforcement.
- CodeFormat → This stage validates code formatting for GraphQL, Java, and Protocol Buffers (Proto) files, checks for changes in harness code repo, and identifies linting issues using Buf.
✅ Sonar Scan — This stage checks for code smells in the code and prevents the PR from merging if the code smells exceed the configured thresholds in the Sonar tool. Code coverage is also included in the Sonar scan. Every PR for the harness-core codebase is analyzed for both coverage and code smells, with a corresponding report published to the SonarQube server. Any PR that does not meet the required thresholds for coverage and code smells will be blocked from merging.
✅ Go Build — This stage will compile the go language code in the harness-core repo and will run only if the PR contains anything related to .go files.
✅ Git Leaks — Mitigate the risk of data breaches and security incidents by proactively identifying and securely removing sensitive information
✅ Validate Reviews — To check PR title for specific keywords (CDS, PIE, or IDP), fetch PR reviews from harness code API, and verify if there are at least 2 “APPROVED” reviews before proceeding.
✅ Merge Validation — Confirms backward compatibility before merging.
To ensure backward compatibility and maintain a stable development branch, we perform a compile build after merging a PR and proactively notify the developer in case of any failed builds to address them promptly
Snapshot Builds: Catching Issues Early with Shift-Left Testing
To detect issues early in the cycle, we leverage Snapshot Builds, an automated process that runs every 6 hours. This ensures all services remain buildable, deployable, and continuously tested, significantly reducing last-minute surprises in QA.
- SNAPSHOT: upgrades the existing devspace with a new build version of the service.
- VANILA SNAPSHOT: a brand new deployment with a new version of the service.
The System Release Train
Harness has 16 services/products, each with its own release cadence based on product-specific requirements. However, our pipeline modules — CD, CI, IACM and STO along with Platform micro-services follow a weekly release cycle known as the System Release.
The remaining products follow independent release cadences, but the overall release process remains consistent across all.
Prerequisite for System Release QA Branch Cut
- Every Friday, we generate a system release snapshot build and deploy it.
- The team conducts a sanity check on this snapshot build.
- If the sanity test is successful, we will proceed with the branch cut on Monday.
System Release Snapshot Success Message
QA Release Branch Cut, Build, and Deployment
- The branch cut, build, and deployment process is fully automated with Harness CI/CD and takes just 90 mins— no manual intervention is required.
- Once deployed, the QA POC team runs automated tests. If issues are found, Alpha fixes are made, and another round of sanity testing is conducted.
Signoff and Deployment
🔹 Wednesday: A release audit is conducted and a go/no-go decision is made
🔹 If the artifacts are approved, the build is deployed to our first production zone — Prod 0. Harness has multiple production zones and our release process always starts with Prod 0.
Production Deployment: A Structured Release Rollout
Our release process follows a carefully planned schedule to ensure each version undergoes rigorous testing before reaching production environments. Here’s how it works:
Day 1 (Monday): QA Validation Begins
- A new weekly release branch is cut, built, and deployed to the QA environment.
- Automated and manual testing begins to identify any issues early.
Days 1–2: Intensive Testing Phase
- Stability checks, regression tests, and performance evaluations are performed.
- Engineers work on addressing any detected issues to maintain release quality.
Day 3: Deployment and Validation in PROD-0
- The build moves to PROD-0, our internal production zone.
- Select customers and internal teams validate the new version in real-world scenarios.
Day 4: Production Deployment Begins with Prod-1
- The release is deployed to PROD-1, our first external production zone.
Day 5: Expansion to PROD-2
- After successful validation, the release moves to PROD-2, covering more users and use cases.
Day 6–7: Wider Rollout to Additional Zones
- Deployment extends to PROD-3, PROD-4, and our EU clusters.
Day 8: Harness-0
At the core of our release strategy is Harness-0, our internal cluster that packages and deploys our own products. This means we are always running the latest, most polished, and stable version of Harness — to build Harness itself. By doing so, we validate our own tools in a production-like environment before shipping them to customers.
The final version is deployed to Harness-0, completing the cycle.
The cycle repeats every week ! 🚀
Stay tuned for more insights into how we refine our release processes to deliver the best software, every time.
