Data Engineering Lead - Python, Pyspark, Azure Databricks
Optum Global Solutions · Gurugram · onsite · Posted 3d ago
Undisclosed6–11 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Primary Responsibilities:
- Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
- Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory, Auto Loader and Lakeflow Spark Declarative Pipelines
- Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
- Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
- Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection
- Build data quality, validation and reconciliation frameworks to keep data accurate and reliable
- Tune Spark jobs and Delta tables for performance and cost (partitioning, liquid clustering, Z-ordering, OPTIMIZE/VACUUM, compute sizing)
- Drive CI/CD and DataOps practices using Azure DevOps or GitHub, Declarative Automation Bundles and automated testing
- Troubleshoot production issues, do root cause analysis and put preventive fixes in place
- Work with architects, analysts, data scientists and business stakeholders, and mentor junior engineers
- Use AI-assisted engineering tools to improve development, testing and documentation
- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Required Qualifications:
- B.E./B.Tech./MCA or equivalent degree in Computer Science, IT or Engineering
- 6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks
- Hands-on experience with Delta Lake, Lakehouse architecture and Unity Catalog
- Experience working with file formats such as CSV, JSON, XML, Parquet, Avro and Delta, including schema inference, schema evolution and nested data handling
- Experience with Azure services: ADF, ADLS Gen2, Key Vault, Azure SQL , Power Bi
- Solid knowledge of data modeling, ETL/ELT design, Git, CI/CD and Agile delivery
- Good understanding of views, temporary/global temporary views, materialized views, streaming tables and when to use each
- Solid proficiency in Python, PySpark and SQL, with a solid understanding of Spark internals and performance tuning
- Proven solid problem-solving and communication skills
Preferred Qualifications:
- Databricks Data Engineer Associate/Professional or Microsoft DP-203/DP-700 certification
- Experience with Structured Streaming, Kafka/Event Hubs, Lakeflow Jobs and Databricks SQL
- Infrastructure as Code experience
- Exposure to healthcare data (claims, prior authorization, HL7/FHIR, EDI 837/278) and HIPAA compliance
- Experience with Power BI or AI/BI dashboards
- Experience mentoring or leading a team, or driving cloud migration and modernization
At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.
Sourced from naukri
Skills
CdcAutomationAzure DevopsAgileautomated testingAzure ServicesclusteringAdls Gen2Agile DeliveryAzure SQLADLScloudCommunication SkillsCloud MigrationArchitectureAzure Data FactoryAzure DatabrickscomplianceMicrosoft AzureArtificial Intelligence
About Optum Global Solutions
Optum Global Solutions
IT Services & Consulting
33.0k+
employees
2002
24 years old
Hyderabad / Secunderabad, Telangana
India
₹11L PA avg
Avg at Optum Global Solutions
About
We are at the helm of a digital health care revolution, which is being accelerated by a combination of technology and talent. We build and deliver end-to-end digital health solutions for our clients.
Industries
IT Services & ConsultingHardware & NetworkingHealthcareSoftware Product
Employee ratings
10 reviews
4.0/ 5
Culture
3.6
Career growth
3.0
Work-life
3.6
Employees rate it well for
Job Security42Work-Life Balance41
Find them on