Senior Data Engineering Lead
Optum Global Solutions · Pune · onsite · Posted 29d ago
Undisclosed5–10 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
Responsibilities
- Build reliable batch, micro-batch, and event-driven pipelines on AWS and Databricks
- Develop reusable ingestion frameworks for REST APIs, FHIR Bulk Export, HL7 interfaces, databases, SFTP/file exchange, JSON/NDJSON, CSV, XML, PDFs, and clinical text
- Implement scalable Spark/PySpark and SQL transformations, including schema inference/evolution, checkpointing, idempotency, retries, backfills, and replay
- Design and maintain Delta Lake and Apache Iceberg tables, including physical design, partitioning/clustering, compaction, file sizing, incremental reads/writes, performance tuning, and cost optimization
- Build curated healthcare data models and transformations using FHIR, HL7, OMOP CDM, and clinical terminology mappings
- Implement data-quality frameworks: schema validation, referential-integrity checks, business-rule testing, anomaly detection, reconciliation, completeness checks, and data-quality observability
- Build metadata and lineage capture across source systems, pipeline runs, code versions, transformation rules, mappings, and published data products
- Implement privacy-aware data processing for PHI, including access controls, masking, tokenization/pseudonymization, de-identification, and auditable handling patterns
- Deliver CI/CD pipelines, automated unit/integration/data tests, Terraform or CloudFormation, containerized services, monitoring, alerting, runbooks, and incident-recovery procedures
- Integrate and optimize governed data consumption through Databricks SQL, Athena, Snowflake, Trino, or equivalent engines
- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
- The candidate should have completed 12 months in the current role
- The candidate should not be on any active CAP/ PIP
- The performance review of the candidate must be ME Above in the last common review
- Graduate degree or equivalent experience
- 5+ years of production data-engineering experience
- Hands-on AWS experience with S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS
- Experience with Git, pull requests, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and incident response
- Experience processing semi-structured data and building resilient ingestion pipelines with quality controls, error handling, and replay capability
- Production Databricks experience with Auto Loader, Delta Lake, Workflows, Unity Catalog, notebooks/jobs, SQL Warehouses, and cluster/job optimization
- Advanced Python, SQL, and Apache Spark/PySpark; strong understanding of distributed processing and performance tuning
- Practical Apache Iceberg knowledge, including tables, catalogs, snapshots, schema/partition evolution, compaction, and interoperability with query engines
- Healthcare data knowledge: FHIR R4 and/or HL7 v2, OMOP CDM, clinical terminologies, PHI, HIPAA-aligned engineering controls, and de-identification concepts
- Demonstrated responsible use of AI coding tools and the ability to critically review, test, and productionize generated code
- Healthcare or regulated-data experience
Sourced from naukri
Skills
DockerCapApache SparkCI/CD PipelinesExportapisApacheautomated testingclusteringamazon web servicesaws cloudformationFHIReventbridgeCSVAWS CloudWatchError HandlingAnomaly DetectionDatabricksData EngineeringData ProcessingGITArtificial Intelligence
About Optum Global Solutions
Optum Global Solutions
IT Services & Consulting
33.0k+
employees
2002
24 years old
Hyderabad / Secunderabad, Telangana
India
₹11L PA avg
Avg at Optum Global Solutions
About
We are at the helm of a digital health care revolution, which is being accelerated by a combination of technology and talent. We build and deliver end-to-end digital health solutions for our clients.
Industries
IT Services & ConsultingHardware & NetworkingHealthcareSoftware Product
Employee ratings
10 reviews
4.0/ 5
Culture
3.6
Career growth
3.0
Work-life
3.6
Employees rate it well for
Job Security42Work-Life Balance41
Find them on