From Lambda to Kubernetes: Re-Architecting the Pantomath Monitoring Bus

At Pantomath, our job is to understand customers' data platform well enough to act on them, which means the layer that captures that understanding has to be more dependable than the pipelines it draws from. The Pantomath Monitoring Bus is the backbone of that promise: it's the system that ingests every event from customer environments, processes them, and feeds the incident analysis that powers alerting and lineage.

Over the past several months, we migrated the Monitoring Bus from AWS Lambda to Kubernetes with Karpenter. This post covers why we made the move, how we did it without disrupting customers, and what changed as a result.

The original architecture: serverless end to end

The first version of the Monitoring Bus was a classic serverless event pipeline:

SNS → SQS → Lambda → S3 + Postgres

Events from customer nodes were published to SNS, fanned out into SQS queues, and consumed by Lambda functions that wrote results to S3 and Postgres, where our incident analysis engine picked them up.

For an early-stage product, this was the right call. Lambda let us ship the Monitoring Bus quickly with zero infrastructure to operate: no servers to patch, no capacity to plan, no clusters to babysit. Scaling was automatic, and we paid only for what we used. A small engineering team got to spend its time on the product; anomaly detection, lineage, incident analysis — rather than on operations. SQS gave us durable buffering; Lambda gave us hands-off consumption. It worked well for years.

Where we hit the walls

As Pantomath grew, the shape of our traffic changed. Event volume didn't just increase — it became spikier and more uneven across customers. And that's where the seams in a shared, serverless consumption model started to show.

Queues fill faster than Lambda can drain them

The core issue was a mismatch between how our queues filled and how Lambda let us drain them.

When a customer significantly ramped up usage, whether from a backfill, a batch window, or an incident cascade in their own platform, event volume into SQS surged. But our ability to consume that surge was capped by Lambda concurrency limits. The queues would backlog faster than the functions could drain them, and processing fell behind exactly when customers most needed us to be current.

Lambda's scaling model gave us limited control here. We couldn't say "scale consumers to match queue depth" — we could only raise concurrency ceilings, tune batch sizes, and hope the math worked out. During large spikes, it increasingly didn't. A growing backlog meant stale monitoring: events waiting in queues aren't events being analyzed.

The noisy neighbor problem

Because customers shared the same Lambda concurrency pool and queue infrastructure, one tenant's surge consumed capacity that everyone depended on. A single large customer's traffic spike could push up processing latency for unrelated customers. For a monitoring product, that's exactly backwards — the moment things get busy is the moment we need to be fastest.

We mitigated with reserved concurrency and queue tuning, but these were band-aids. Tenants sharing a single elastic compute pool will always be able to affect each other at the margins.

Tail latency

The compounding effect of both problems showed up in our p95. Between cold starts, per-invocation overhead, and backlog contention during bursts, tail latencies were higher and less predictable than we wanted. Median performance looked fine on a dashboard, but p95 is what a customer actually feels when they're waiting on an alert. We were running out of levers to pull within Lambda's execution model.

Why Kubernetes — and why Karpenter

We evaluated a few paths: staying on Lambda with heavier partitioning, ECS, and EKS. We landed on Kubernetes for three reasons.

First, we could scale consumers directly on queue depth. On Kubernetes, our SQS consumers are long-running services whose replica counts scale as a direct function of queue backlog. When a queue starts to grow, consumers scale out immediately and drain it; when the surge passes, they scale back down. The drain rate is now coupled to the fill rate — the exact control Lambda's model didn't give us.

Second, we could give every customer isolated processing capacity. The Monitoring Bus now runs as per-tenant workloads with their own queues and consumers. A surge from one customer scales that customer's resources — it doesn't touch anyone else's. Isolation stopped being a tuning exercise and became a property of the architecture.

Third, we could scale each stage of the monitoring workflow independently. The Monitoring Bus isn't one homogeneous workload — it's a pipeline of stages: grouping incoming node events, evaluating them, and upserting the resulting monitoring events, incidents, and alerts before dispatch. These stages have radically different performance profiles. Some complete in tens of milliseconds; others take seconds and require strict FIFO ordering. A single shared consumer pool can never be sized correctly for all of them at once — it's simultaneously over-provisioned for the fast stages and starved for the slow ones.

So we partitioned queues along both dimensions: per customer and per stage. Each (customer, stage) pair gets its own queue and its own consumers, sized from a simple throughput model — a pod's drain rate is its worker concurrency times the inverse of that stage's mean processing time, measured from our production Lambda data. Each queue then scales on its own depth. A burst of alert dispatches for one customer scales exactly that: their dispatch consumers. Nothing else moves.

The classic objection to leaving serverless is elasticity: with Kubernetes, don't you end up paying for idle nodes and managing capacity by hand? This is where Karpenter changed the math for us.

Karpenter provisions nodes just-in-time based on actual pending workload, rather than scaling pre-defined node groups. When queue-depth-driven scaling asks for more consumer pods, Karpenter watches for unschedulable pods, when queue depth pushes our consumer deployment to scale out and the resulting pods can't be placed on existing nodes, Karpenter provisions new capacity within seconds, not the minutes it takes to add nodes to a pre-warmed group.; when the surge passes, it consolidates workloads and removes nodes. In practice, we kept much of the elasticity that made Lambda attractive, while gaining:

• Queue-depth-driven scaling — consumer capacity tracks the backlog itself, not indirect concurrency settings
• Stage-aware capacity — fast and slow stages of the workflow scale independently, each at its own natural rate
• Long-running consumers — no cold starts, warm connections to S3 and Postgres, no per-invocation setup cost
• Right-sized, bin-packed compute — Karpenter picks instance types to fit the actual mix of workloads
• Predictable performance — dedicated per-tenant capacity means tail latency no longer depends on what other customers are doing

The ingestion edge (SNS → SQS) stays exactly as it was — durable, decoupled buffering is still the right pattern. What changed is everything downstream of the queue.

How we migrated

The Monitoring Bus is not something we can take down for a weekend, so the migration was deliberately boring by design.

SLO-first. Before moving any traffic, we defined the SLOs the new path had to meet — data correctness, latency, and error rate — at each stage of the cutover. No slice of traffic advanced until those SLOs were met, not on a calendar deadline.

Gradual cutover. We moved traffic incrementally, starting with a small slice, expanding customer by customer and workload by workload as confidence grew. At every stage, we could route traffic back to the Lambda path instantly if anything looked off.

Constant validation. Throughout the cutover, we continuously compared the new path against the old one — verifying that the Kubernetes-based consumers produced identical results in S3 and Postgres, with better performance, before shifting more traffic. Our own monitoring watched our monitoring.

Patch, then proceed. We treated every discrepancy or regression as a gate: fix it, validate the fix, then continue the rollout. Progress was measured in verified traffic shifted, not calendar time. It made the migration slower than a big-bang cutover — and completely uneventful for customers, which was the point.

What changed for our customers

The headline result: every customer is now isolated. One tenant's traffic burst can no longer affect another tenant's processing. Surges are absorbed by that customer's own consumers, scoped to exactly the stage of the workflow that's under load, scaled on queue depth and backed by capacity Karpenter provisions on demand. Everyone else's experience stays flat.

Just as important, the Monitoring Bus now keeps pace with spikes instead of falling behind them. Backlogs that used to build during heavy traffic are drained as they form, so monitoring stays current precisely when customer platforms are at their busiest.

For customers, this translates to something simple: Pantomath is fast, current, and consistent even when your data platform is at its busiest.

Lessons and tradeoffs

We'd be lying if we said this was free. We now operate Kubernetes clusters, which means new operational surface area: upgrades, observability for the platform itself, and Karpenter configuration to maintain. A few things we learned:

1. Serverless-first was still the right call. Lambda got us to market fast and carried us further than we expected. Migrate when the architecture fights your requirements, not before.
2. Scale on the signal that matters. Our real scaling signal was always queue depth. Lambda made us scale on proxies for it; Kubernetes let us scale on the thing itself.
3. Granularity is leverage. Partitioning by customer and by workflow stage meant every scaling decision became small, local, and cheap. Big shared pools force big blunt decisions.
4. Isolation is an architectural property, not a configuration. We spent real effort trying to tune our way to tenant isolation on shared infrastructure. In hindsight, the tuning was a signal that we'd outgrown the model.
5. Karpenter closes most of the elasticity gap. The biggest reason teams hesitate to leave serverless is the fear of losing scale-to-demand, and that fear is far smaller than it used to be.
6. Boring migrations are good migrations. Gradual cutover plus continuous validation cost us time but saved our customers from ever noticing.

What's next

If problems like this sound fun to you, we’re hiring.