Lead Data Engineer

Job Summary

We are looking for an experienced Lead Data Engineer with strong hands-on expertise in Python, SQL, Databricks, and REST APIs to join our Data Analytics team. The ideal candidate should have strong experience in designing and developing scalable data solutions, working with large datasets, and building reliable data pipelines in a cloud/data analytics environment.

Key Responsibilities

• Lead the design, development, and implementation of scalable data engineering and analytics solutions.
• Develop robust and reusable data pipelines using Python, SQL, and Databricks.
• Work with REST APIs to extract, integrate, transform, and consume data from various source systems.
• Design and optimize data models, ETL/ELT pipelines, and data processing workflows.
• Perform complex SQL queries, data transformations, performance tuning, and troubleshooting.
• Develop and maintain Databricks notebooks, workflows, and data pipelines.
• Collaborate with Data Analysts, Data Scientists, BI teams, and business stakeholders to understand data requirements and deliver analytics-ready datasets.
• Ensure data quality, accuracy, security, and reliability across data pipelines.
• Troubleshoot production issues and identify opportunities for performance and process improvements.
• Provide technical leadership and mentor junior and mid-level Data Engineers.
• Participate in solution design, code reviews, technical discussions, and best-practice implementation.

Required Skills

• Strong hands-on experience in Python for data engineering and data processing.
• Strong proficiency in SQL, including complex queries, joins, CTEs, window functions, and performance optimization.
• Strong hands-on experience with Databricks and its data engineering capabilities.
• Good experience working with REST APIs, API integration, authentication, and data extraction.
• Strong understanding of ETL/ELT concepts and data engineering principles.
• Experience working with large datasets and building scalable data pipelines.
• Good understanding of data warehousing, data modeling, and data integration concepts.
• Experience with cloud-based data platforms is preferred.
• Strong problem-solving, communication, and stakeholder management skills.

Good to Have

• Experience with Apache Spark / PySpark.
• Experience with Azure, AWS, or GCP.
• Knowledge of Delta Lake and Databricks Lakehouse architecture.