Senior Data Engineer - Bangalore - Immediate Joiners only

Job description

Senior Data Engineer

Strictly Immediate Joiners only(0 to 15 days NP)

Exp : Minimum 5+ Years

Skills : AWS , Databricks

Job Description:

We are seeking a highly skilled Data Engineer with expertise in Databricks and AWS Cloud to design, build, and optimize enterprise-scale ETL pipelines and reporting solutions. This role is ideal for someone who thrives at the intersection of data engineering and business intelligence, with a strong focus on transforming complex datasets into inputs for actionable insights. You will play a critical role in enabling spend analytics, patient access, and commercial reporting initiatives.

Key Responsibilities Databricks ETL Development

• Design, develop, and maintain scalable ETL pipelines in Databricks using PySpark and SQL.
• Implement robust data ingestion, transformation, and validation processes to ensure high-quality datasets for analytics.
• Optimize workflows for performance, scalability, and reliability across large healthcare datasets.

Data Product Build

• Transform AWS assets (on EC2 and Redshift) to Databricks by recreating the running ETL jobs and orchestrating via Airflow or Databricks Workflows.
• Hands-on working experience of catalog management using Unity Catalog.
• Conversant with AI capabilities within AWS and Databricks.

Requirements

• 7+ years of experience in Databricks data engineering roles.
• Strong hands-on proficiency in Databricks (PySpark, SQL) and AWS Cloud.
• Proven track record in ETL pipeline development and understanding of BI dashboards.
• Experience with US healthcare datasets.
• Strong SQL skills for data extraction, aggregation, and reporting.

• Excellent problem-solving abilities, with the capacity to work independently and in cross-functional teams.
• Detail-oriented with a commitment to delivering high-quality, reliable data solutions.

Good to Have

Domain Data Knowledge

• Knowledge of IQVIA claims (standard, eLaaD, Remit, Rejection, NBRx, TRx, IGG4 Claims) and MMIT datasets to derive actionable insights.

End-to-End Production Deployment Orchestration

Candidates with exposure to deploying and orchestrating data pipelines in production environments will be preferred. While not a core requirement, the following experience is a strong differentiator:

• Experience deploying ETL or ML pipelines end-to-end in a production environment, including environment promotion across dev, staging, and production.
• Familiarity with CI/CD tooling (GitHub Actions, Azure DevOps, or Jenkins) for automating pipeline deployment and release management.
• Exposure to Databricks Asset Bundles (DABs) or equivalent frameworks for version- controlled, repeatable job deployments.
• Working knowledge of Apache Airflow for DAG authoring, scheduling, dependency management, and monitoring of production workflows.
• Awareness of infrastructure-as-code practices (Terraform or AWS CloudFormation) for managing cloud resources supporting data pipelines.
• Basic understanding of containerization concepts (Docker) for packaging and deploying data pipeline components.
• Experience with pipeline monitoring and alerting tracking job health, SLA adherence, failure notifications, and data freshness in production.
• Familiarity with secrets and configuration management across environments (AWS Secrets Manager, Databricks Secrets, or equivalent).

If Interested, Request you to share me your updated resume to [email removed]

Thanks & Regards,

Geetha Shree

Lead Recruiter

Talent Acquisition Team | Quantzig AI Solutions Pvt. Ltd.
www.quantzig.com

[phone removed]