Lead Data Engineer

Key Responsibilities:

Data Pipeline Development & Optimization

• Design, develop, and optimize data pipelines using Azure and Databricks.
• Build and manage scalable ELT/ETL workflows with Databricks, Delta Lake, ADF, and other Azure services.
• Optimize Spark, SQL, and Python code for performance and scalability.

CI/CD & Environment Management

• Hands-on work with Databricks Asset Bundles (DABs) for CI/CD and environment management.
• Continuously adopt and apply new Databricks features to accelerate delivery.
• Apply software engineering best practices (TDD, CI/CD, modular design, code reviews) to ensure robust and maintainable pipelines.

Team Mentorship & Collaboration

• Mentor team members to improve technical capability and overall productivity.
• Deliver high-quality solutions in a fast-paced environment with quick turnaround expectations.