Senior Data Engineer (Platform & Streaming)

Level: Senior / SDE-III equivalent

Experience: 58 years

Location: India (Hybrid) — or Pune / Remote-India

Type: Full-time

Reports to: Engineering Manager, Data Platform About the Role We're hiring a Senior Data Engineer to own the design and delivery of production data pipelines that power analytics, reconciliation, and AI workloads.

This is a hands-on build role — you'll write Spark and Python daily, make architecture calls on ingestion and storage design, and take responsibility for pipelines that run under real reliability and compliance pressure. You'll work across batch and streaming, structured and unstructured data, and increasingly on the data layer that feeds retrieval and LLM applications. What You'll Do Design and ship end-to-end data pipelines — ingestion, transformation, storage, and serving — for both batch and near-real-time workloads. Build distributed processing jobs in Spark (PySpark/Scala) and optimize them for cost and latency: partitioning strategy, shuffle reduction, file sizing, broadcast joins. Model and maintain a layered data lake (raw processed curated) on S3 or GCS, with clear contracts between zones. Build streaming ingestion on Kafka / Kinesis / MSK for event-driven data, including schema evolution and replay handling. Own orchestration in Airflow (or Dagster) — idempotent DAGs, sensible retries, backfill support, and alerting that fires on the right things. Implement data quality and validation frameworks so bad data fails loudly upstream rather than silently downstream. Write and maintain Terraform for the infrastructure your pipelines depend on, and ship through CI/CD rather than manual deploys. Apply access controls, PII masking, and encryption appropriate to regulated data (HIPAA, PCI, or equivalent). Build ingestion and processing paths for unstructured data — documents, logs, semi-structured JSON — including chunking, embedding generation, and vector store integration. Review designs and code from other engineers, and mentor mid-level engineers on distributed systems fundamentals. Required Qualifications 5+ years building production data pipelines, with at least 2 years at a senior or lead level of ownership. Strong Python — modular, tested, production-grade code, not just scripts. Deep Spark experience (PySpark and/or Scala), including performance tuning of real jobs at scale. Advanced SQL: query optimization, indexing, partitioning, and data modeling for analytical workloads. Production experience with a workflow orchestrator (Airflow, Dagster, or similar). Hands-on experience with at least one major cloud (AWS or GCP) — including storage, compute, networking basics (VPC, subnets, routing), and IAM. Docker in production, and working familiarity with Kubernetes deployments. Infrastructure as Code with Terraform, and CI/CD for data systems. Experience with streaming ingestion using Kafka or a managed equivalent. A track record of delivering under compliance or governance constraints in a regulated domain. Nice to Have Scala for Spark, or Go for data services and tooling. dbt for transformation and testing in the warehouse layer. Databricks, BigQuery, or Snowflake at production scale. Vector databases (pgvector, Pinecone, or similar) and embedding pipelines for RAG systems. Migration experience — legacy batch to Spark, or cross-language platform rewrites. Exposure to fintech, healthcare, or another domain with strict data handling requirements. Our Stack Languages: Python, Scala, SQL, Go Processing: Spark (Databricks), Kafka, Kinesis Orchestration: Airflow, dbt Storage: S3 / GCS data lake, BigQuery, Postgres, pgvector Infrastructure: AWS, GCP, Docker, Kubernetes, Terraform, GitHub Actions