Keep Looker DRY

Introduction

Here at Harness, one of our core values is to build trust through transparency. That is why we have prioritized building a best-in-class custom dashboard experience upon Looker. Efficiency can rarely be leveraged without sound architecture; that is why we have recently refactored our Looker architecture towards a more modular, scalable format to provide greater development agility and accuracy to get ship done even faster. The architecture we have embraced is the Hub & Spoke architecture.

In this blog, we will detail what the Hub & Spoke architecture is, how it is integrated within Harness’ Custom Dashboard service and how it has further optimized the Looker development cycle.

Hub & Spoke Architecture

Within our Custom Dashboard service, we employ a Hub & Spoke architecture. This means we house all our core business explores within the Hub project and import those explores into the appropriate Spokes. Each Spoke represents a production cluster (prod1, prod2, prod3, etc.). Within each Spoke, a model file is declared that points to the respective database connection for that cluster, and the Spoke imports the explores housed within the Hub project. For further clarity, see deployment diagram below:

The benefits of this architecture is apparent: code need only be modified within the Hub project before it propagates across the respective production clusters; additionally, this allows flexibility when new production clusters are added to quickly service them with Looker capabilities.

Below is further detail of each component:

Hub

As mentioned above, the Hub project houses all core business logic, views, etc. Below is a screenshot of the Hub project in Looker to give an idea of what kind of data is housed within this project:

As one can see, there are folders for dashboards, explores and views. Each of these folders holds core files, which means they will, by default, be imported into each Spoke project via Looker’s local import and assume the respective Spoke’s connection string. All core logic for any data that can be applied to each cluster should be developed and stored in the Hub project.

Spoke

Each Spoke imports the Hub project. Below is a screenshot of how this is cared for in an example Spoke project:

As one will notice, there is an imported_projects folder that imports all Hub files into the respective Spoke project. To take advantage of these files, each model file needs to import the respective files, as one can see in lines 5-6 in the above screenshot. Note the connection string on line 1 is routed to the appropriate connection for the production cluster, and a distinguishing label is declared to differentiate each connection for internal developers.

Project Manifest Import

To import the Hub’s files, a project manifest must be instantiated, with similar code to below:

project_name: "spoke_prod"

local_dependency: {
  project: "hub"
  override_constant: env_cluster {
    value: "PROD"
  }
}

The local_dependency allows a local import of the Hub project. Local import just means that the project is housed within the same Looker instance. If a remote import is needed (i.e., hosted on GitHub), then that can be done here; however, local project import satisfies the requirements for Harness’s use cases.

The override_constant overrides the env_cluster found within the core Hub project. One example use case is core dashboard files, as one can see below:

In this case, the model in the OOTB dashboard “Deployments Dashboard” will be overridden to read SPOKE_PROD_CD, allowing dynamic model routing across Spokes.

Data Tests

Data tests are essentially unit tests for data to ensure congruence and accuracy. We do not employ data tests at this time, but we plan on engineering data tests and codifying a data test policy in the coming quarters.

Git Repos

The Hub project is in its own Git repository. Each Spoke is housed within a single repository, assuming their own production branch within the repo.

Benefits from Refactor

Before, Harness had separate model files for each production cluster for each module within a single Looker project. While sufficient during the beginning stages of Looker development, Harness eventually outgrew this V1 architecture. The evolved rendition of this initial architecture required maintaining the same code in multiple places, with our developers essentially copying and pasting LookML feature additions and bug fixes to each model file.

To give a concrete example, if a new feature was to be shipped for Continuous Deployment–such as a new table for pipeline visibility being added–the developer would have to copy and paste all new changes for each model file for visibility in each explore across all production clusters. This is not best practice. Employing Hub & Spoke allows the developer to make the change in one place, with the changes then propagating to the connected Spokes.

Furthermore, the Hub & Spoke architecture allows us to be more agile in our Out of the Box Dashboard development. Hosting a single dashboard LookML file within the Hub, coupled with using the project manifest override in each Spoke, allows a core dashboard file to be constructed and dynamic propagation to be employed for each production cluster. Three versions of the same dashboard need not be maintained anymore.

Conclusion

Hub & Spoke’s agility positions Harness to continue to provide best-in-class custom dashboard services for our customers. With this new modular architecture, code quality will increase and development times will be cut, resulting in faster shipment of feature requests and bug fixes.