Data Engineer

Responsibilities:

• Design, develop, and maintain data pipelines and ETL processes to efficiently ingest, transform, and load data from various sources into data warehouses and data lakes.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and design data models that facilitate efficient data retrieval and analysis.
• Optimise data pipeline performance, ensuring scalability, reliability, and data integrity.
• Implement data governance and security measures to ensure compliance with data privacy regulations and protect sensitive information.
• Identify and implement appropriate tools and technologies to enhance data engineering capabilities and automate processes.
• Conduct thorough testing and validation of data pipelines to ensure data accuracy and quality.
• Monitor and troubleshoot data pipelines to identify and resolve issues, ensuring minimal downtime.
• Develop and maintain documentation, including data flow diagrams, technical specifications, and user guides.
• Collaborate with software engineers and infrastructure teams to optimise data infrastructure, including storage, processing, and retrieval systems.
• Stay up-to-date with emerging trends and technologies in the field of data engineering, and recommend innovative solutions to improve efficiency and performance.

Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• A master's degree is a plus.
• Proven experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts, practices, and tools.
• Proficiency in programming languages such as Python, Java, or Scala, and experience with data manipulation and transformation frameworks and libraries (e. g., Apache Spark, Pandas, SQL).
• Solid understanding of relational databases, data modelling, and SQL queries.
• Experience with distributed computing frameworks, such as Apache Hadoop, Apache Kafka, or Apache Flink.
• Knowledge of cloud platforms (e. g., AWS, Azure, GCP) and experience with cloud-based data engineering services (e. g., Amazon Redshift, Google BigQuery, Azure Data Factory).
• Familiarity with data warehousing concepts and technologies (e. g., dimensional modelling, columnar databases).
• Strong problem-solving skills and the ability to analyse complex data-related issues.
• Excellent communication and collaboration skills, with the ability to work effectively in cross- functional teams.
• Attention to detail and a commitment to delivering high-quality work within specified timelines.