Senior Data Engineer – Databricks Pipelines

Senior Data Engineer – Databricks Pipelines

Location: Gurgaon (Hybrid / In-office)

Experience: 6–9 Years

Employment Type: Full-Time Job Overview We are looking for a highly skilled

Senior Data Engineer with strong expertise in

Databricks, PySpark, and modern data engineering practices to design, develop, and optimize enterprise-scale data pipelines. The ideal candidate will play a key role in building scalable

Lakehouse architectures , developing reliable ETL/ELT workflows, and enabling AI-driven analytics solutions by ensuring high-quality, governed, and performance-optimized data platforms. You will work closely with data scientists, AI teams, and engineering teams to deliver robust data pipelines that power analytics, chatbot solutions, and intelligent data applications. Key Responsibilities Data Pipeline Development

• Design, develop, and maintain scalable ETL/ELT pipelines using

Databricks, PySpark, Delta Lake, and Databricks Workflows .

• Build production-grade data ingestion frameworks supporting both batch and streaming workloads.
• Develop reusable and optimized data processing components using Python and Spark. Lakehouse Architecture & Data Engineering
• Design and optimize

Databricks Lakehouse architecture using Raw, Bronze, Silver, and Gold data layers.

• Implement efficient storage strategies to support analytics, AI workloads, and high-performance retrieval use cases.
• Optimize Spark jobs for performance, scalability, and reliability. Streaming & Data Integration
• Build real-time data ingestion pipelines using

Delta Live Tables (DLT) and Databricks Auto Loader.

• Integrate multiple enterprise data sources including batch and streaming systems.
• Implement CDC (Change Data Capture) pipelines for incremental data processing. Data Quality & Governance
• Implement data validation frameworks and quality checks within data pipelines.
• Build automated data quality gates, exception handling, and monitoring mechanisms.
• Implement

Unity Catalog for data governance, access control, lineage tracking, and security management. Performance Optimization

• Troubleshoot and optimize Spark workloads involving:
• Data skew
• Shuffle optimization
• Partition strategies
• Join optimization
• Cluster performance tuning
• Utilize Spark UI, Adaptive Query Execution (AQE), caching, broadcast joins, and optimization techniques. DevOps & Deployment
• Manage CI/CD workflows for Databricks deployments using:
• Git
• Azure DevOps / GitHub Actions
• Databricks Asset Bundles (DABs)
• Promote data solutions across Dev, QA, and Production environments following engineering best practices.
• Collaborate on infrastructure automation using tools like Terraform. Required Skills & Qualifications
• 6–9 years of experience in enterprise-scale Data Engineering.
• Strong hands-on experience with:
• Databricks
• PySpark
• Python
• Delta Lake
• SQL
• Strong understanding of Lakehouse architecture and modern data platforms.
• Experience developing production-grade ETL/ELT pipelines.
• Hands-on experience with:
• Delta Live Tables (DLT)
• Databricks Workflows
• Auto Loader
• Unity Catalog
• Delta Change Data Feed (CDF)
• Strong knowledge of Spark optimization techniques.
• Experience working with cloud data platforms and data warehouses.
• Experience implementing CI/CD practices for data engineering projects. Preferred Skills
• Experience supporting AI/ML data platforms and retrieval-based applications.
• Knowledge of vector search or AI-ready data architectures.
• Experience with Terraform or Infrastructure-as-Code practices.
• Strong understanding of data governance and security frameworks. Ideal Candidate Profile A strong candidate should be able to:
• Build scalable and reliable data pipelines independently.
• Optimize complex Spark workloads.
• Design production-grade Databricks solutions.
• Implement automation, governance, and best practices across the data lifecycle.

Work Location: Gurgaon

Work Mode: Hybrid / In-office