Data Platform Engineer

Job Description:

Data Platform Engineer

Location: Gurugram
Employment Type: Full-Time — Hybrid
Experience: 5 to 8 years

Role Description

We are hiring a senior individual contributor to help scale the platform behind our first-party (1P) marketplace data operations. We run high-volume authenticated crawls across 220+ global marketplaces. As that footprint grows, the engineering challenge shifts from writing individual crawlers to building the platform that runs thousands of them predictably. This role sits at that layer. You will work on orchestration, data quality, observability, and service integration — the systems that let a large crawler fleet run at a known standard rather than case by case. It is a hands-on engineering role with real architectural ownership.

What You Will Own

• Orchestration and job reliability
  • Own the scheduling and orchestration layer that coordinates 1P crawl jobs across marketplaces, accounts, and regions.
  • Design retry, backoff, dead-letter, and idempotent rerun semantics so repeated execution is always safe.
  • Build heartbeat, timeout, and automated recovery patterns into long-running distributed jobs.
  • Manage concurrency, rate limiting, session lifecycle, and credential rotation for authenticated portals.
  • Define completion SLAs by job type and design the platform to meet them.
• Data integrity
  • Design schema contracts and validation gates that data passes through before publication.
  • Build deduplication into the pipeline using natural keys, content hashing, and idempotency keys.
  • Implement completeness checks across marketplace, account, and date dimensions.
  • Apply anomaly detection to volume, field-fill rates, and value distributions.
  • Design quarantine and replay paths so questionable batches are held and reprocessed cleanly.
• Observability
  • Instrument the platform with structured logging, metrics, and distributed tracing.
  • Propagate correlation identifiers across job, task, request, and output record, so any row can be traced to its source fetch.
  • Design artifact capture and retention so runs are reproducible and auditable.
  • Build dashboards, alerting, and runbooks that scale with the fleet.
  • Treat time-to-diagnosis as a tracked engineering metric.
• Service integration
  • Integrate the pipeline with authentication, relational, and analytical query services.
  • Apply timeouts, circuit breakers, bulkheads, and graceful degradation across service boundaries.
  • Design health checks and readiness signals that make system state explicit.
  • Build for partial-dependency conditions, so the platform degrades predictably rather than unevenly.

Requirements

• 5 to 8 years building production Python systems, including async work with asyncio or equivalent.
• Deep hands-on experience with Scrapy and Playwright or Selenium at scale.
• Proven work on authenticated portals. Session handling, cookie and token lifecycle, proxy rotation, and anti-bot mitigation.
• Strong distributed queueing experience with RabbitMQ or Kafka. Consumer groups, redelivery, ordering, and dead-letter queues.
• Hands-on workflow orchestration experience. Airflow, Temporal, Prefect, Kestra, or a comparable system.
• Production experience with MongoDB and PostgreSQL, including index design and query tuning on large datasets.
• Practical observability experience. OpenTelemetry, Prometheus and Grafana or similar, structured logging, and distributed tracing.
• Solid grounding in HTTP and HTTPS, HTML, DOM, XPath, and CSS selectors.
• Comfortable with Linux, Docker, and CI/CD pipelines.
• Able to reason about a distributed failure across multiple services independently.

Nice to Have

• Trino, Presto, or comparable distributed query engines.
• Experience with e-commerce vendor and seller portals.
• Data quality tooling such as Great Expectations or Soda, or a custom equivalent.
• Infrastructure as code.
• Failure injection or chaos testing practice.
• Experience mentoring engineers through code review and design review.

What We Are Looking For

• Ownership. You take responsibility for outcomes across system boundaries.
• Systems thinking. You look for the class of problem, not just the instance.
• Curiosity to analyse and reverse-engineer websites and their defences.
• Evidence-driven engineering. You reach for logs, traces, and data first.
• Clear written communication with data, analytics, and client-facing teams.
• Comfort in a fast-paced environment with shifting marketplace behaviour