Lead Data Engineer

Job Description / Duties and Responsibilities

▪ Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and related Azure services.

▪ Develop complex and optimized T-SQL queries, stored procedures, views, functions, and database objects in SQL Server.

▪ Perform SQL Server database development and administration activities, including performance tuning, troubleshooting, and data integrity management.

▪ Build and maintain data integration pipelines connecting multiple source and target systems.

▪ Work with Azure data platform services to develop and support cloud-based data solutions.

▪ Analyze source systems and changing data structures and adapt data pipelines accordingly.

▪ Apply light-to-medium data modeling techniques to support analytical and operational data requirements.

▪ Work with large volumes of structured and semi-structured data and ensure efficient data processing.

▪ Troubleshoot data pipeline failures, data quality issues, performance problems, and production incidents.

▪ Monitor data pipelines and proactively identify and resolve issues.

▪ Collaborate with developers, architects, QA teams, business analysts, and other stakeholders to deliver data solutions.

▪ Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog refinement, reviews, and retrospectives.

▪ Create and maintain technical documentation covering data flows, mappings, processes, dependencies, and troubleshooting procedures.

▪ Participate in code reviews and follow established development, security, and deployment standards.

▪ Support production deployments, releases, and operational activities as required.

Job Specification / Skills and Competencies

▪ 7+ years of experience in Data Engineering, ETL, data integration, or related roles.

▪ Strong hands-on experience with Microsoft SQL Server development and administration.

▪Advanced knowledge of T-SQL, including:

o Complex queries

o Stored procedures

o Views and functions

o Query optimization

o Indexing and performance tuning

o Error handling and troubleshooting

▪ Strong experience with Azure Data Factory (ADF) for ETL/ELT and data integration.

▪ Practical experience working with Azure data platform services.

▪ Good understanding of data warehousing and data integration concepts.

▪ Experience with data modelling, including dimensional modelling and logical/physical data models.

▪ Ability to understand and work with evolving or changing source data structures.

▪ Strong troubleshooting and problem-solving skills.

▪ Strong technical documentation and communication skills.

▪ Experience working in an Agile/Scrum development environment.

▪ Ability to work independently while collaborating effectively with distributed/onshore-offshore teams.

▪ Experience with CI/CD and source control tools such as Azure DevOps, Git, or Bitbucket.

▪ To adhere to the Information Security Management policies and procedures.