Before we dive into the technical details, let’s first briefly discuss what Audit Trails are.
An Audit Trail (or Audit Log) is an account of the operations that someone has performed, in a chronological order. Each audit record captures at least this basic information:
As you can see, each audit record is essentially about who did what, and when.
Auditing helps in:
Audit Trails are an inbuilt feature of the Harness platform. Most of the actions that are performed on the Harness platform generate an audit record. See the below screenshot.
Note: The Audit Trails UI will be released in Q4 ending Jan 2022. Audit Trails can be accessed via API at present.
If you want more information on the basics of Audit Trails, please feel free to read our 101-level blog post. The rest of this post is a technical deep dive, and talks about the high-level design of how we implemented Audit Trails into Harness.
In Harness, we have a microservices-based architecture where there are multiple domain-specific microservices. For example, we have a CI microservice, a CD microservice, etc. that wants to maintain this kind of an audit trail. For this reason, Audit Trails is built as a platform-level feature inside Harness as it’s a common functionality required by many microservices.
At a high level, the auditing system consists of a central dedicated microservice, which is responsible for everything related to audits. On the other side, there are clients (other domain specific microservices, such as CI, CD etc. in this case) who record the operations happening inside them in the form of an audit with the auditing service.
This microservice is primarily responsible for:
Some of the salient features of this service are:
Data storage is the most interesting aspect of this service. The Auditing service uses MongoDB as its storage backend. These are the few reasons for this choice:
As mentioned earlier, clients are other microservices that want to capture the operations being executed inside them as an audit trail.
Registering the audit is easy: just call the API of the auditing service. However, the most important technical consideration on the client side is how to capture the operation as an audit record in the first place.
Let’s look at few options:
We would prefer a standard way to capture audits across microservices even though they might be built upon different tech stacks. Enter Domain Events.
Domain Events are a very important tool in Domain-Driven Design. Domain Events record business significant occurrences. ProjectCreated, AccountDeleted, SecretUpdated are examples of Domain Events. Other business workflows can be triggered based on these domain events.
Domain Events contains all relevant information. For example: ProjectCreated events would have accountId, id, name, description properties. Other common properties such as userId, timestamp etc. can be populated from request context.
It’s extremely important that changes and corresponding Domain Events are saved together in the same transaction in DB. Otherwise, there exists a possibility that changes are persisted in DB, but before Domain Events could be persisted, the client crashes and Domain Events are lost.
Once Domain Events are persisted, a background job notifies interested listeners about their occurrence so that they can be executed. Interested listeners can be within the same microservice or other microservices. If another microservice is interested in the event, a dedicated listener can publish the event into some topic of a message broker, to which other services can subscribe to.
Once you have Domain Events in place, it’s very easy to leverage and generate audit records from them. Have a listener, which listens to the Domain events, convert them to audit records and finally save them with the auditing service.
This approach implements the Transactional Outbox Pattern. You can read more about it here: https://microservices.io/patterns/data/transactional-outbox.html.
We hope you enjoyed this technical Audit Trails walkthrough. Audit Trails are necessary in many industries and simply as a Best Practice in CI/CD. Again, feel free to read our entry-level post on Audit Trails to find out more about why you need them.
Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.