SENIOR SOFTWARE ENGINEER - Azure Databricks

Location: Bengaluru (Hybrid)

Experience: 5-8 Years

Employment Type: Full-Time

Job Summary: We are seeking a highly skilled SaaS Databricks Engineer to design, develop, and optimize cloud-native data engineering solutions on Azure Databricks. The ideal candidate will have strong expertise in PySpark, SQL, Delta Lake, Azure Data Factory, and modern data platform architectures. The role involves building scalable ETL/ELT pipelines, supporting SaaS applications, and ensuring reliable data delivery across enterprise environments.

Responsibilities:

1. Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.
2. Build and optimize ETL/ELT workflows for large-scale data processing.
3. Implement and manage Bronze, Silver, and Gold Lakehouse architectures using Delta Lake.
4. Integrate SaaS applications and external data sources into enterprise data platforms.
5. Develop data ingestion frameworks using Azure Data Factory (ADF).
6. Optimize Databricks clusters, notebooks, and Spark workloads for performance and cost efficiency.
7. Implement data quality, validation, audit, and reconciliation controls.
8. Work with Azure Data Lake Storage Gen2 (ADLS), SQL databases, and cloud-native storage solutions.
9. Automate deployment processes using CI/CD pipelines (Azure DevOps or GitHub Actions).
10. Perform troubleshooting, root cause analysis, and production support activities.
11. Ensure security, governance, and compliance using RBAC, Key Vault, and managed identities.
12. Monitor pipeline health and implement proactive alerting and observability solutions.
13. Collaborate with architects, analysts, business stakeholders, and product teams.
14. Create technical documentation, data flow diagrams, and operational runbooks.
15. Continuously recommend improvements in platform scalability, reliability, and cost optimization.

Mandatory Skills:

• 5-8 years of Data Engineering experience.
• 3+ years of hands-on Azure Databricks experience.
• Strong expertise in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake and Lakehouse Architecture.
• Familiarity with Azure Data Lake Storage Gen2 (ADLS).
• Proficiency in SQL Server / Azure SQL.
• Python programming skills.
• Experience in ETL/ELT development and optimization.
• CI/CD experience using Azure DevOps or GitHub Actions.
• Understanding of Data Modelling and Data Warehousing concepts.

Preferred Skills:

• Databricks Certified Data Engineer Associate/Professional.
• Microsoft Azure Data Engineer Associate (DP-203).
• Azure Fundamentals (AZ-900).

Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field is preferred. Strong analytical and problem-solving skills, along with excellent communication and collaboration abilities, are essential for success in this role.