Rearchitecting Pantomath's Extractor Fleet

How we consolidated two dozen sprawling metadata extractors into one runtime for maintainability, reliability, and self-service setup.

Pantomath is the Data Operations Center that prevents bad data from impacting business outcomes by proactively catching issues, pinpointing their root cause, and autonomously resolving them with operational context from across your entire stack.

Each platform has its own extractor for pulling relevant data from the platform that we then transform into the customer's data context: pipeline lineage graphs, monitoring events, and data assets and changes across their data estate.

These extractors started as one microservice per platform, each in its own repo, stamped out from a service template. Early on this was reasonable as teams could iterate and deploy independently. However as the number of platforms expanded to two dozen and more, the pain started to show.

Pain Points

Developer productivity and change friction - The sprawl of these extractors was making platform level refactoring and improvements a non-starter. For example, to enhance extractor monitoring across our fleet, developers would have to pull 25+ repos, make updates to each and submit a PR each. Even with agentic AI coding, this was becoming hard to manage. 

Release friction - Each standalone extractor had its own CI pipeline. Tracking the deployments and releases across these individual services was becoming difficult. A hotfix in shared extraction code would mean coordinating a release for every extractor that consumed it.

Sub-optimal utilization of provisioned infrastructure - Each extraction is run on a schedule and for extractors that run on our cloud infrastructure, provisioning a HA (high availability) set up for each extractor meant low utilization.

Dependency upgrades and CVE Patches - A vulnerability in a shared dependency meant auditing and patching 25+ services, each on its own release cadence. Independent versioning also let shared libraries drift: at one point we had three live versions of the same internal package across the fleet.

Re-architecture Roadmap

Move Extractors Into A Monorepo 

The first step was to move individual extractors into a monorepo. 

Extractors are built on a node-typescript stack. We first established a “pnpm” repo and migrated standalone extractors one at a time as a package in the pnpm workspace. Shared dependencies were set up as workspace dependencies.

We used Lerna for versioning and package management, and NX for task orchestration and build optimization. Nx Cache particularly helped to reduce build times. We also built a change detection mechanism on the CI/CD pipeline so that a package in the monorepo is built and deployed only if there is direct change or a change in the dependencies.

1. We prioritized low risk extractors first to establish a proof of concept.
2. After 1-2 migrations, we used a Claude skill to capture the steps, the gotchas and automate moving other extractors
3. Drift from changes made to standalone repos before the switch over were automated using a Claude skill
4. Once deployments and regression tests were verified, we moved the others incrementally.
5. After the cutover of each extractor, the standalone extractor repo was archived.

The migration became a repeatable playbook: each extractor was cheaper to move than the last. By the end, moving an extractor was a routine ticket rather than a project.

A unified extractor runtime

Once we had all the ducks (extractors) in a row (monorepo), we started building a unified runtime where we established an interface that each extractor would implement. The extractors were then added as package dependencies. The unified runtime would accept requests and delegate to the platform specific extractor package.

While most platforms were easy to move, some extractors that needed native dependencies were slower to migrate. Oracle, for example, needed thick client dependencies for native network encryption, and Kerberos authentication also needed specific native libraries. We baked these dependencies also into the unified image. The regression test suite and the migration approach provided a good safety net.

Migration Approach

Feature flag to switch the runtime incrementally

Our in-house feature flag mechanism allows us to scope a feature flag across multiple levels of granularity. This was extremely useful as we rolled out the unified runtime. 

1. Enable the unified runtime for a particular platform (say MySQL) starting with one customer
2. After observation, expand customer base and make default

Ensure rollback-ability by preserving the existing standalone services

Standalone extractor Docker image builds and deployments were preserved in the mono repo - to allow for rollback in case of undetected issues in the unified runtime.

Run the regression suite against both runtimes
We updated our regression specs to execute against the standalone extractor and the unified runtime in parallel on any change in the mono repo so that any divergences like extraction output and performance could be measured. This gave us the rollout confidence as we switched over each extractor.

Benefits Unlocked

Much lower resistance to make platform-wide changes and refactoring

The change that used to mean pulling 25+ repos and shepherding 25+ PRs is now a single pull request against the monorepo. The mental friction was dissolved and improvements like fleet-wide extractor monitoring became normal sprint work instead of big sized projects.

Higher developer productivity

One repo to clone, one dependency graph to reason about, one CI pipeline to watch. NX's build cache and change detection mean a PR builds only what it touches, and a shared-code hotfix that once required coordinating ~25 releases now ships as one PR and one release, fleet-wide.

What’s good for the devs is good for their AI agents

Previously, changing a shared library was a multi-repo dance: a developer would first have to make a change in the shared package repo, wait for PR to merge and then use that published package in the extractor. This bottleneck hamstrung the mightiest AI models. With the monorepo, everything about the fleet is one repo change and one PR.

Higher utilization of cloud and on-premise infrastructure

With a single runtime, we no longer provision infra per platform. Extractions run on schedules, so pooled capacity in a unified runtime leads to better load distribution at meaningfully lower cost. And onboarding a new platform no longer requires provisioning a new service deployment.

Simplified on-premise extractor deployment

Some customers prefer on-premise deployments of these extractors. A unified extractor runtime vastly simplified the deployments by collapsing the number of deployment artifacts into one. This single artifact meant unlocked a self service setup for the on premise extractors since the single runtime supports all platforms out of the box.

In the next blog in this series we will cover the scalability and reliability improvements we unlocked with this change under our belt.