Data Engineer - Databricks

Key Responsibilities

• Design and develop robust ETL/ELT pipelines using Databricks platform and PySpark
• Implement and maintain data workflows using Databricks Asset Bundles (DABs)
• Build and optimize data processing solutions using Delta Lake architecture
• Develop complex SQL queries and stored procedures for data transformation and analysis
• Ensure data quality through comprehensive testing and validation frameworks
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
• Monitor and optimize pipeline performance and data processing efficiency
• Maintain code quality through version control best practices and code reviews
• Document technical processes and maintain data pipeline documentation

Required Skills & Qualifications

Must-Have Technical Skills:

1. Databricks Platform - DABS
   
• Proficiency in Databricks Asset Bundles (DABs) for CI/CD workflows
• Experience with Databricks workspace management and configuration
2. PySpark Development
   
• Strong programming skills in PySpark for large-scale data processing
• Experience with DataFrame API, RDD operations, and Spark SQL
3. SQL
   
• Advanced SQL skills including complex joins, window functions, and performance optimization
• Experience with data modeling and database design principles
4. Delta Lake Fundamentals
   
• Understanding of Delta Lake architecture and ACID transactions
• Experience with time travel, data versioning, and schema evolution
5. ETL Development
   
• Proven experience in designing and implementing ETL/ELT pipelines
• Knowledge of data integration patterns and best practices
6. Data Quality Assurance
   
• Experience implementing data validation and quality checks
• Knowledge of data profiling and anomaly detection techniques
7. Version Control
   
• Proficiency with Git for code management and collaboration
• Experience with branching strategies and code review processes