Software Developer

Job Summary

We are looking for an experienced Senior Databricks Data Engineer with 710 years of hands-on experience in designing, developing, implementing, and optimizing scalable data engineering solutions. The ideal candidate will have strong expertise in Databricks, PySpark, Python, SQL, Delta Lake, and cloud-based data platforms, preferably Microsoft Azure. The role involves building robust ETL/ELT pipelines, implementing Lakehouse architecture, optimizing data processing workloads, and ensuring high standards of data quality, security, and performance. The candidate will also collaborate with architects, analysts, data scientists, and business stakeholders to deliver reliable, business-driven data solutions.

Key Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize end-to-end ETL/ELT workflows for batch and incremental data processing from multiple data sources.
• Implement Delta Lake and Lakehouse architecture, including Delta tables, schema evolution, ACID transactions, and data versioning.
• Develop and manage data orchestration workflows using Azure Data Factory (ADF), Databricks Workflows, or similar tools.
• Integrate data from various sources into Azure Data Lake Storage (ADLS) and implement efficient data ingestion and transformation strategies.
• Perform data modeling, query optimization, partitioning, and performance tuning to improve processing efficiency and cost-effectiveness.
• Implement data quality checks, validation frameworks, error handling, logging, monitoring, and reconciliation mechanisms.
• Apply security, access control, and governance best practices to ensure data integrity and compliance.
• Implement version control, CI/CD pipelines, and deployment automation using Git and DevOps tools.
• Troubleshoot complex data pipeline issues, optimize existing solutions, and ensure reliability and scalability.
• Collaborate with cross-functional teams, participate in Agile/Scrum ceremonies, and provide technical guidance to junior engineers.

Required Skills & Qualifications

• 7–10 years of overall experience in data engineering, with strong hands-on experience in building enterprise-scale data solutions.
• Extensive practical experience with Azure Databricks, PySpark, Python, and advanced SQL.
• Strong understanding of ETL/ELT design patterns, distributed data processing, and data pipeline architecture.
• Hands-on experience with Delta Lake, Lakehouse architecture, and large-scale data transformation.
• Experience with Azure Data Factory, ADLS Gen2, and cloud-based data integration services.
• Strong knowledge of data modeling, query optimization, Spark performance tuning, and memory management.
• Experience implementing CI/CD, Git-based version control, automated deployments, and DevOps best practices.
• Familiarity with data governance, metadata management, and access control mechanisms.
• Experience working in Agile/Scrum environments with strong analytical, problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Experience with Azure Synapse Analytics, Microsoft Fabric, or other cloud data platforms.
• Knowledge of streaming and near-real-time data processing using Spark Structured Streaming.
• Exposure to Databricks Unity Catalog, job orchestration, and production monitoring.
• Databricks or Azure certifications in data engineering.

Ideal Candidate: A technically strong senior data engineer who can independently own data engineering solutions end to end, drive performance and reliability improvements, and collaborate effectively with technical and business teams.