BENGALURU · FULLTIME
Sr Data Engineer
Lifesight
Bengaluru · onsite · Posted 3d ago
Sourced from
Undisclosed6–9 yrsfulltimeBengaluru
Your match
?
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
We are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP), processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers, drive cloud cost optimisation across our data workloads, and build AI-driven workflows that change how we support and operate the platform.
This is a product engineering role, not a services or consulting engagement. You will own systems end to end, live with your design decisions in production, and help define data engineering culture in India.
Key Responsibilities
Build and scale the platform
- Design, build and operate highly available, fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake and warehouse.
- Write and tune large Spark workloads (PySpark, Spark SQL, Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning, shuffle and join optimization, skew handling, memory and executor sizing.
- Build real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam), with clear guarantees on latency, ordering and exactly-once or idempotent processing.
- Model data in BigQuery and the lakehouse (Parquet, Avro, Delta Lake or Apache Iceberg) into clean, simplified datasets that teams can self-serve.
- Build data cleaning, validation and reconciliation pipelines that stay correct when source data is late, duplicated, malformed or missing.
- Put data contracts, schema evolution rules, quality checks and alerting in place, and own data mapping, business logic and transformations end to end.
- Own the cost of our GCP data workloads: measure cost per pipeline and per customer, then reduce it through right-sizing, autoscaling, preemptible/Spot VMs, storage tiering, BigQuery slot and query optimization, and job consolidation.
- Set up cost visibility (billing export, labels, dashboards) and FinOps practices so cost is tracked and owned, not discovered after the fact.
- Lead and mentor 2-4 junior and mid-level data engineers: plan and break down work, review code and designs, and raise the team's engineering bar.
- Lead system design and architecture discussions, write design docs, and make trade-offs on scalability, reliability, latency and cost.
- Design AI-driven workflows for technical support and operations: LLM-assisted triage of pipeline failures and data incidents, automated root-cause analysis, runbook automation, and agents that answer data questions from internal teams.
- Turn repeated support work into automated, process-driven workflows, and measure the time and tickets they save.
- Own production reliability: on-call, monitoring and observability, incident response, postmortems, and low-level debugging and performance tuning on large clusters.
- Orchestrate pipelines with Airflow / Cloud Composer and ship through CI/CD and infrastructure as code (Terraform).
Experience
- 6 plus years in data engineering or backend/software engineering, with at least 4 years building big data pipelines in production.
- Minimum 1-2 years as a tech lead or team lead, mentoring junior engineers and owning delivery for a team.
- Most of your experience at product or SaaS companies, owning platforms you built and ran in production, rather than consulting or IT services delivery.
- Deep hands-on PySpark and Spark SQL; Spark internals (DAG, stages, shuffle, catalyst, AQE), performance tuning, and handling data skew on multi-terabyte jobs. Scala or Java Spark is a strong plus.
- Production experience with BigQuery, Dataproc, Cloud Storage (GCS), Pub/Sub, and Cloud Composer or Airflow. Dataflow, GKE and Bigtable are a plus.
- Designed and run both, with a solid grasp of windowing, watermarks, late data, checkpointing, backfills and idempotency. Kafka or Pub/Sub plus Spark Structured Streaming, Flink or Dataflow.
- Expert-level, including window functions, complex joins and query optimization on large analytical warehouses.
- Strong Python and Java with object-oriented design, design patterns and clean, testable, reusable code.
- Built data cleaning, validation, deduplication and reconciliation pipelines, with automated quality checks and alerting.
- Can design a scalable, fault-tolerant distributed data system and defend the trade-offs in consistency, latency, cost and failure handling.
- A track record of measurably reducing cloud or data platform costs (state the numbers on your CV).
Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
AutomationApache SparkBatch processingclaudeapisApacheCatalystclusteringCloud StorageBigtableapache airflowbigqueryCloud ComposerClickHouseApache BeamADTECHCloud Cost OptimizationcloudArchitectureBig DataArtificial Intelligence
Section · Company
About Lifesight
Lifesight
₹14.4L PA avg
Avg at Lifesight
About
Linesight provides professional consultancy services, management support and strategic advice to the global construction industry. From initial concept to project completion, we have been helping clients to build their businesses since 1974
Employee ratings
10 reviews
4.0/ 5
Culture
4.3
Career growth
3.2
Work-life
4.6
Employees rate it well for
Work-Life Balance47Skill Development44
Find them on