Lead II - Data Engineering

Job Summary

Lead Databricks Developer (Offshore). Looking for 2 strong Lead Databricks Developers with streaming expertise.

Responsibilities

• Design and develop end-to-end data pipelines using Databricks (PySpark, Delta Lake).
• Build real-time streaming ingestion pipelines from Kafka and other event-driven sources.
• Implement layered data architecture (Bronze, Silver, Gold) to support transformation and reporting.
• Develop data processing logic for transactional data ingestion, validation, and enrichment.
• Apply data quality, deduplication, and business rules for healthcare datasets.
• Support cloud migration from on-premise systems to Databricks + Snowflake ecosystem.
• Optimize pipeline performance to meet SLA-driven processing timelines.
• Implement and support CI/CD pipelines and DevOps integration.
• Collaborate with business and technical stakeholders to deliver scalable solutions.

Required Skills

• Strong expertise in Databricks (Streaming Batch)
• Advanced proficiency in PySpark / Spark SQL/Spark Streaming
• Experience in real-time data streaming (Kafka)
• Strong understanding of data modeling and ETL/ELT pipeline design
• Hands-on experience with AWS (S3, Airflow, Lambda)
• Experience with CI/CD, Git, and deployment automation
• Domain Experience
• Experience in Healthcare Payer domain (Claims, Membership, Coverage)
• Exposure to FHIR / healthcare data standards preferred
• Understanding of data privacy, security, and governance requirements (PII/HIPAA)

Leadership Expectations

• Operate at a Lead level driving design, development, and delivery
• Mentor team members and ensure delivery quality
• Own modules independently and collaborate across distributed teams

Experience

• 8+ years in Data Engineering
• 3+ years hands-on experience with Databricks
• Prior experience in Tech Lead/Senior roles
• Looking for 2 strong Lead Databricks Developers with streaming expertise

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.