Introducing new Catalog Ingestion APIs to make Harness IDP truly yours

Updated

Lately we have been working hard to support some advanced use-cases for our customers. I am glad to write about an exciting set of new capabilities in Harness IDP (Internal Developer Portal) across Software Catalog, Scorecards and Workflows. I will take you through a couple of use-cases and show how this can be useful for you too.

These capabilities allow you to update or ingest additional metadata in the Software Catalog for components and use it to customize your Catalog pages UI, build new Scorecard checks and provide dropdown pickers in the Workflows UI.

One of our main considerations when building this has been the ease of use - that this should not require manual and expensive YAML updates from developers. We have heard you loud and clear.

Now let us dive into these features by taking a couple of use-cases.

‍

GIF demonstrating the Catalog ingestion feature and use-cases

‍

Use Case - View and measure test coverage score for microservices

Assume you have a system which keeps track of test coverage scores for your microservices and other software components. This could be information that you may be getting from your testing tools or systems like Sonarqube. We do not care about the data source or whether you process the data yourself e.g. you might want to generate a mean score based on unit test coverage as well as end to end tests.

‍

For our use-case, let’s assume you want to accomplish two goals -

Show on the Software Catalog the test coverage score for any given component e.g. payment-service
Build a new Scorecard check in your existing or a new Service Maturity scorecard

‍

As you might be aware, every software component is ingested into the catalog using a catalog-info.yaml definition file. The challenge is that there are hundreds or thousands of microservices and the test coverage score is computed multiple times a day. So there is no way a developer or a platform engineer can keep all the YAMLs up to date.

This leads to the need of updating the component’s definition without updating its YAML definition file. As the Catalog in Harness IDP is powered by Backstage and Backstage does not have an API to update an entity, we had to create a new one!

Using the new Catalog Metadata Ingestion APIs, you can add new metadata fields like metadata.testCoverage for any or all software components in the Software Catalog. This is updated directly in the Catalog without updating its YAML definition file in git. Check out the docs to explore the endpoint in detail.

‍

The entity data is now updated in Catalog, great! But how do we see it? Especially, how would developers find their test coverage score for their services, websites and libraries that they own?

This is where our new Entity Additional Info Card comes to the rescue. Without writing any custom plugin or JS/TS - you can update the Layout of the Catalog page from Admin -> Layout section, add this new Card and start displaying the test coverage score on the Catalog Overview page for the components. This is what it will look like -

‍

‍

Brilliant! There’s just one last thing - we need to update our default “Service Maturity” scorecard as well and utilize test coverage score as a check. Let’s assume that the architects in your organization have decided that for backend components, the score should be more than 70% and for UI components, it should be more than 60%.

Scorecards support Catalog as a native data source, where you can query the entity definition to write your own check. In this case, we’ll use the JEXL expression data point and build a check for backend components (filtered using catalog’s tags) to compare the score with the desired number.

‍

‍

And that’s it! The last missing piece is that you can now run the ingestion as per your preference and the data will keep on updating.

ProTip - You can use Harness Pipeline triggers and run the Catalog Ingestion as an hourly or daily cron job, using the Run step in IDP Stage.

‍

This approach can be used for many other similar use-cases, for example tracking team leads and on-call members in catalog or tracking cost budget utilization per service, team or system. Now let’s take a look at another example.

Use Case - Integrating Harness IDP with ServiceNow CMDB

Many of our customers use the ServiceNow CMDB (Configuration Management Database System) to keep track of their software assets. Each application has a unique business application ID which is their unique identifier. It is used anytime developers have to make a new request to the central IT or DevOps teams.

Using the Catalog Ingestion APIs and Additional Info Card as described above, we can ingest and display the Business Application IDs in the IDP Software Catalog as well. However, when it comes to IDP Workflows, these IDs have a role to play there as well, for example when submitting an infrastructure provisioning request.

This is why we have launched a new Autocomplete Picker UI field which allows developers to pick from all available catalog entities and any of its metadata. In this case, we would like developers to pick from all values of metadata.serviceNowAppId.