SDE IV - Data Engineering

The core responsibilities for the job include the following:

Data Stack:

• Design and own end-to-end data pipelines for batch and real-time handling of bid requests, win/loss events, impression logs, click streams, and conversion signals at scale.
• Build and maintain our data lakehouse architecture (ingestion, storage, and serving layer) optimized for low-latency DSP analytics and ML feature generation.
• Define standards for data quality, lineage, observability, and SLA adherence across all data products.
• Own and evolve our StarRocks deployment for real-time analytical queries, schema design, ingestion patterns, and query optimization for DSP reporting workloads.
• Drive FinOps discipline across the data platform: track compute and storage costs, enforce resource quotas, identify optimisation opportunities, and report on unit economics.
• Establish and lead Data Governance practices: metadata management, data cataloguing, access control policies, PII classification, and data retention standards compliant with GDPR and CCPA.

Backend Microservices:

• Contribute to and review the design of backend services that power DSP functionality, such as bid shading, pacing, budget management, and reporting APIs.
• Architect data contracts and event schemas between microservices, ensuring consistency across the bid request lifecycle.
• Champion best practices in API design, service reliability (SLOs, circuit breakers, retries), and observability (distributed tracing, structured logging, metrics).
• Collaborate with the bidder team on low-latency data reads (Aerospike, Cassandra, or similar) for real-time targeting signal lookups.

Technical Leadership:

• Lead architecture reviews, set engineering standards, and drive adoption of best practices across the data and backend engineering teams.
• Mentor SDE II/III engineers through code reviews, design sessions, and pair programming; grow the next generation of technical leaders.
• Partner with ML engineers on the full model lifecycle: feature engineering pipelines, offline training data generation, experiment tracking, and model serving infrastructure.
• Translate complex business requirements (campaign KPIs, ROAS, viewability, and brand safety) into robust technical designs.
• Own the technical roadmap for data infrastructure capacity planning, migration strategies, and build vs. buy decisions.

Requirements:

• 8+ years of software engineering experience, with at least 4 years focused on data engineering or large-scale data systems.
• Prior experience in ad-tech, programmatic advertising, or real-time bidding (RTB) environments is strongly preferred.
• Experience in ML engineering, feature engineering, building offline training pipelines, or collaborating closely on model productionization is preferred.
• Bachelor's or master's degree in computer science, engineering, or a related technical field. Data Engineering (Core).
• Deep expertise in distributed stream processing frameworks, Apache Kafka, Apache Flink, or Apache Spark Structured Streaming.
• Strong command of SQL and experience optimizing complex analytical queries at petabyte scale.
• Hands-on experience with real-time analytical databases, particularly StarRocks; familiarity with Snowflake or ClickHouse is a plus.
• Proficiency with workflow orchestration tools: Apache Airflow, Prefect, or Dagster.
• Solid understanding of data modelling, dimensional modelling, OBT patterns, and event-driven schemas relevant to ad impression and attribution data.
• Familiarity with open table formats: Apache Iceberg, Delta Lake, or Apache Hudi.

Backend and Systems:

• Proficiency in Python (primary) and/or Java/Scala for building production-grade services and data processing jobs.
• Experience designing and building RESTful APIs and/or gRPC-based microservices.
• Working knowledge of low-latency data stores (Aerospike, Cassandra) used for real-time lookups in the bidding path.
• Comfortable with containerized deployments on Kubernetes and CI/CD pipelines (GitHub Actions, ArgoCD, or similar).

Ad-Tech Domain Knowledge:

• Understanding of the RTB ecosystem DSPs, SSPs, ad exchanges, bid request/response lifecycle, and OpenRTB protocol.
• Familiarity with identity resolution approaches: cookie IDs, device IDs, UID2 / RampID, and privacy-preserving identity.
• Exposure to audience segmentation, frequency capping, pacing algorithms, or attribution models is a significant plus.
• Awareness of privacy regulations (GDPR, CCPA) and their implications for data pipelines and user-level data handling.

Leadership and Collaboration:

• Demonstrated experience leading technical squads or functioning as a technical anchor on cross-functional projects.
• Strong written and verbal communication skills; ability to translate technical complexity for nonengineering stakeholders.
• Track record of driving projects from ambiguous requirements to production with high engineering quality.