Lead Data Engineer (Snowflake / Databricks)

Job Summary

We are seeking an accomplished Lead Data Engineer with strong expertise in modern cloud data platforms, with hands-on experience in Snowflake or Databricks. The ideal candidate will be responsible for designing, developing, and optimizing scalable data solutions, building robust data pipelines, and ensuring data quality, reliability, and performance. The candidate should have strong experience in data engineering, data modelling, ETL/ELT processes, and cloud data platforms, with the ability to work across Snowflake and Databricks. Hands-on experience with at least one of these platforms is mandatory, while working knowledge of the other is good to have. This role also involves providing technical leadership, mentoring engineers, collaborating with cross-functional teams, and translating business requirements into scalable technical solutions.

Key Responsibilities

• Lead the design, development, and optimization of scalable, secure, and high-performance data solutions using Snowflake and/or Databricks.
• Design, build, and maintain robust data pipelines using appropriate orchestration and data engineering frameworks, ensuring reliability, scalability, monitoring, retries, and error handling.
• Develop scalable data models and transformations following best practices for data warehousing and modern lakehouse architectures.
• Develop and optimize data solutions on Snowflake or Databricks, including performance tuning, data processing, storage optimization, and workload management.
• Implement data validation, automated testing, monitoring, governance, and security practices to ensure data integrity, reliability, and compliance.
• Identify and resolve performance bottlenecks across data pipelines, queries, jobs, and data processing workloads.
• Build and manage data solutions on cloud platforms such as AWS, Azure, or GCP.
• Partner with data analysts, data scientists, architects, and business stakeholders to translate requirements into scalable technical solutions.
• Provide technical guidance and mentorship to data engineers, conduct code/design reviews, and establish engineering best practices.
• Stay current with advancements in cloud data platforms, data engineering, analytics, and AI/ML technologies and recommend relevant improvements.
• Work with version control, CI/CD, and infrastructure-as-code practices for reliable deployment and management of data platforms.

Qualifications

• Bachelor s or Master s degree in Computer Science, Information Technology, Data Engineering, or a related field.
• 7-10 years of experience in data engineering, with strong hands-on experience in at least one of Snowflake or Databricks.
• Demonstrated experience in leading and mentoring data engineering teams.
• Strong working knowledge of modern data engineering concepts, including:
  • Data modelling and data warehousing
  • ETL/ELT processes
  • Data Lake / Lakehouse architectures
  • Batch and incremental data processing
  • Data quality and validation
• Snowflake: Strong hands-on experience in data modelling, performance tuning, access control, and features such as streams, tasks, external tables, or equivalent capabilities.
• Databricks: Strong hands-on experience with data engineering, Spark-based processing, Delta Lake/Lakehouse concepts, and pipeline development.
• Snowflake or Databricks - one must be hands-on; knowledge of the other is good to have.
• Proficiency in SQL and Python.
• Spark experience is expected for candidates with Databricks exposure.
• Experience building and managing data pipelines on AWS, Azure, or GCP.
• Experience with data orchestration tools such as Apache Airflow, Azure Data Factory, AWS Glue, or equivalent.
• Familiarity with transformation frameworks such as dbt is good to have.
• Familiarity with version control systems such as Git and CI/CD practices.
• Exposure to infrastructure-as-code tools such as Terraform is good to have.
• Strong analytical, problem-solving, collaboration, and communication skills.
• Demonstrated ability to lead technical projects and mentor junior/mid-level engineers.

Preferred Skills

• Hands-on experience with both Snowflake and Databricks.
• Experience with dbt and Apache Airflow.
• Experience with streaming technologies such as Kafka, Kinesis, or Pub/Sub.
• Experience with Generative AI, ML, or advanced analytics applications.
• Familiarity with BI/analytics tools such as Power BI, Tableau, Looker, or similar.
• Knowledge of data governance, security, and compliance frameworks.
• Experience with multiple cloud platforms such as AWS, Azure, or GCP.