Staff Engineer - Distributed Systems

Job Summary

Thousands of pods. 50+ kinds of deployments. Four databases. Three teams shipping fast. And until now, nobody whose whole job is the system itself.

Our teams are excellent at building products. Each service has owners, each feature has engineers, each database has experts. What we dont have is the person who holds the entire distributed system in their head, who sees that the retry policy in one service and the queue configuration two hops away are, together, a cascading failure waiting for Black Friday.

Thats the seat. Your job is to find the failure before it finds production.

You start with Workflows, HighLevels automation engine, and one of the largest systems in the company: 3.1 billion enrollments and 21.5 billion action executions every month, traffic peaking at 28,000+ requests per second, running across thousands of pods on GCP with Pub/Sub, Cloud Tasks, Redis, and multiple database engines underneath. From there, your blast radius grows, into Conversations (2.6 billion messages a month) and a brand-new Ticketing system being built right now, where you get to make sure its born right instead of fixed later.

This is not an architect role where you draw boxes and hand them to someone else. And its not a feature role where you own a backlog. Its the role in between that most companies never create, and most staff engineers spend their careers wishing existed.

Responsibilities

The terrain:

Requirements