Harnessing The Power of Secrets Management

While the whole DevOps practices talk about automation and speed, it is also important to ensure that security is taken care of. Most organizations that ignored security are now rethinking it and embracing the DevSecOps principles of shifting left and adding security checkpoints at every step of the CI/CD pipeline. Security is getting more attention as it gets the highest points when considering any tool. Hence, security professionals are in high demand to ensure the systems and processes work as expected without getting attacked by hackers and security vulnerabilities. The software delivery pipeline consists of a CI/CD pipeline that requires a lot of connections from third-party tools for a smooth software delivery workflow. These external tools and platforms, such as your Docker Hub, Artifactory, Cloud providers, Security testing tools, etc, are connected to your CI/CD pipeline through secrets.

Storing and managing these secrets is crucial so the credentials are not compromised. It is also one of the essential and basic traits when considering a CI/CD platform to understand how the secrets are managed securely. 

Today, we will see how secrets are managed in the Harness platform.

What is Secrets Management?

Secrets management in the context of DevOps refers to securely storing and managing sensitive information such as passwords, API keys, database credentials, and other secrets used in the software development and deployment process. These secrets are typically required by various components of an application, including infrastructure configuration, code repositories, build scripts, deployment pipelines, and runtime environments.

The goal of secrets management is to protect sensitive information from unauthorized access, misuse, or exposure, while still providing controlled access to authorized users, processes, or systems. In a DevOps and CI/CD environment, secrets management becomes crucial because automation tools and processes often require access to these secrets to perform tasks such as building, testing, and deploying applications.

Where are secrets involved in a CI/CD pipeline in DevOps?

Secrets are integral to a CI/CD pipeline and can be present at every step, from code to deployment environment. In a typical CI/CD workflow, a developer pushes his/her code to a source control management (SCM) tool like GitHub or BitBucket, and then a continuous integration (CI) tool gets triggered, where it builds and tests the code. Next, the artifacts get created and pushed to a repository, where an artifact repository connector is required. Then the code gets deployed to production, and you need to connect the cloud provider or wherever you are deploying the code (the target environment). Finally, the monitoring tool gets connected to the CI/CD pipeline for observability purposes to keep the application running as expected. 

In a typical CI/CD workflow/pipeline, these types of connectors are involved where managing secrets is very important. 

• Code repository connections - GitHub, GitLab, and BitBucket.
• Continuous Integration servers and Continuous Delivery platforms – Jenkins, Bamboo, JFrog, TeamCity, Drone, Harness, GitLab CI/CD, CircleCI, etc.
• Database connections - MongoDB, Redis, ElasticSearch etc
• Cloud Provider connections - AWS, GCP, Azure, Kubernetes (as keys, configuration files, and other information).
• Container Registry Connections - DockerHub, JFrog Artifactory, Sonatype Nexus, etc
• Infrastructure – (SSH keys, load balancer credentials).
• Collaboration providers – Slack, SMTP, Jira, ServiceNow.
• Verification providers – AppDynamics, New Relic, Prometheus, Splunk, etc

Challenges of Managing Secrets in a DevOps Pipeline

Managing secrets in a DevOps pipeline poses significant challenges in today's technology landscape. One of the primary concerns revolves around maintaining the security and integrity of sensitive information, such as API keys, passwords, and encryption keys, throughout the development and deployment processes. As DevOps emphasizes rapid and continuous delivery, secrets management becomes crucial to prevent unauthorized access and data breaches. Additionally, the distributed nature of DevOps, with multiple teams and various tools involved, adds complexity to secret distribution, rotation, and revocation. Ensuring proper access controls and permissions across different environments, including development, staging, and production, further compounds the challenges. 

Here are some of the current challenges associated with secret management in DevOps:

1. Security: Secrets, such as passwords, API keys, and cryptographic keys, are sensitive pieces of information that need to be protected. One of the main challenges is ensuring the security of secrets throughout the pipeline. Unauthorized access to secrets can lead to data breaches and compromise the entire system.

‍

2. Access Control: Balancing the need for access to secrets with the principle of least privilege is a challenge. Different team members, including developers, operations staff, and third-party integrations, may require access to certain secrets. Managing and enforcing appropriate access controls to secrets is crucial to prevent unauthorized access.

‍

3. Storage and Distribution: Secrets need to be securely stored and distributed across the pipeline. Finding a secure and scalable storage solution for secrets, such as a centralized vault or key management system, is essential. Additionally, ensuring secure distribution of secrets to relevant components, such as containers or cloud instances, requires careful consideration.

‍

4. Rotation and Expiration: Secrets have a limited lifespan due to security best practices. Regularly rotating secrets, such as passwords or API keys, is necessary to mitigate the impact of a potential breach. Coordinating and automating secret rotation processes across different stages of the pipeline can be challenging.

‍

5. Versioning and Auditing: Maintaining a history of secret changes and auditing access to secrets is critical for compliance and security purposes. Tracking and managing versions of secrets used at different stages of the pipeline, as well as auditing who accessed or modified secrets, can be complex.

‍

6. Integration with CI/CD Tools: Secrets need to be seamlessly integrated into the continuous integration and deployment (CI/CD) pipeline. This integration ensures that secrets are securely accessed by the pipeline, allowing automated processes to retrieve and use secrets without exposing them. Achieving this integration while maintaining security can be a challenge.

‍

7. Developer Experience: Striking a balance between security and developer experience is crucial. Developers need an efficient and easy-to-use way to manage secrets without compromising security. Providing developers with tools and processes that abstract away the complexity of secret management can enhance their productivity.

‍

8. Compliance: Compliance requirements, such as GDPR, HIPAA, or PCI-DSS, often have specific rules regarding secret management. Ensuring that the secrets management practices align with relevant compliance regulations can be challenging, requiring organizations to stay current with the latest requirements.

‍

Addressing these challenges requires a combination of robust security practices, effective access controls, automation, and secure secret management solutions. 

How are Secrets Managed in Harness?

You can manage your secrets in Harness using a key management service (KMS) or a third-party secret manager. Google Cloud Key Management Service is the default Secret Manager in Harness and is named Harness Secret Manager Google KMS.

The Key Management Service (Google Cloud KMS or AWS KMS) only stores the key. Harness uses envelope encryption to encrypt and decrypt secrets. The encrypted secret and the encrypted Data Encryption Key (used for envelope encryption) are stored in the Harness database. 

Using Third-Party Secret Managers​

You can also use third-party Secret Managers, for example, HashiCorp Vault, Azure Key Vault, and AWS Secrets Manager.

These Secret Managers store the key, perform encryption and decryption, and also store the secrets (encrypted key pair). Neither the keys nor the secrets are stored in the Harness database. A reference to the secret is stored in the Harness database.

Let’s see how to add a Google KMS. 

Go to your project and click on ‘Connectors’ 

‍

‍

Click on the ‘New Connector’, and you will be presented with all the possible secret managers present. 

‍

Click on the ‘GCP KMS’ and continue. Mention a name and start adding the details. 

Go to your GCP console and click on ‘Security’ and then ‘Key Management’.

Create a key ring and open the Actions menu (⋮), then click Copy Resource Name.

[Each KMS [key management system] is a little different. Google KMS follows this pattern for folks to retrieve the key]

A reference to the key is now on your clipboard.

Paste the reference into an editor.

You can now copy and paste its substrings into each of the Harness Secret Manager’s Details settings as shown below.

‍

Next, you need to attach the GCP KMS Credentials File. 

You can follow this guide and connect your GCP KMS to Harness. 

Similarly, you can add 

• AWS Secrets Manager
• AWS KMS Secrets Manager
• Azure Key Vault Secrets
• HashiCorp Vault Secrets Manager

You can also add encrypted test secrets, file secrets or SSH keys to reference in your CI/CD pipeline. Harness includes a built-in Secrets Management feature that enables you to store encrypted secrets, such as access keys, and use them in your Harness connectors and pipelines.

‍

This is how we add our GitHub Personal Access Token, Docker Hub credentials, Database connection string, etc.

‍

One thing to note here is that you need to make sure that your Delegate is up and running.

Harness Delegate is a service you run in your local network or VPC to connect your artifacts, infrastructure, collaboration, verification, and other providers, with Harness Manager. The first time you connect Harness to a third-party resource, Harness Delegate is installed in your target infrastructure, for example, a Kubernetes cluster. Know more about Harness Delegate here. 

Docker Registry Secret Connector Example

When you want to connect your Docker Hub to Harness, this is the step you need to take.

Click on the new connector and then select ‘Docker Registry’ from the list.

Mention a name and continue.

Next, you will add the Docker Registry URL and authenticate with your Docker Hub credentials as shown in the screenshot below. 

If you see, the password is using an encrypted text secret that nobody can see. 

Finally, you will make sure to connect this with Harness Delegate.

You need to make sure that the connection between your Docker Hub and Delegate is successful. 

This is how secrets are used and managed at Harness. You can also add and store SSH keys securely in Harness. 

Take a deep dive at Harness secret management overview.