Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable data pipelines and data engineering solutions.
• Develop data ingestion, transformation, and processing workflows using cloud-based technologies.
• Work with Snowflake and Databricks to build and optimize data processing and analytical solutions.
• Develop and maintain data pipelines using AWS Glue, S3, and Lambda.
• Implement data transformation, cleansing, validation, and integration processes.
• Develop efficient SQL queries and optimize data processing and warehouse workloads.
• Work with business and technical teams to understand data requirements and translate them into technical solutions.
• Participate in solution design, development, testing, and deployment activities.
• Develop reusable components and follow established data engineering and coding standards.
• Troubleshoot data pipeline and application issues and implement appropriate fixes.
• Participate in code reviews, technical discussions, and continuous improvement initiatives.
• Support deployment and release activities across environments.
• Monitor data pipelines and proactively identify issues affecting data processing or availability.
• Provide support for production issues when required, including troubleshooting and root cause analysis.
• Maintain technical documentation and operational procedures.

Preferred candidate profile

• 3-8 years of hands-on experience in data engineering and ETL/ELT development.
• Strong understanding of data engineering concepts, ETL/ELT, and data pipelines.
• Hands-on experience with Snowflake and/or Databricks.
• Experience building data pipelines and data transformation workflows.
• Strong SQL skills.
• Good understanding of data modelling and data warehousing concepts.
• Good experience with cloud-based data platforms and data pipelines.
• Hands-on experience with one or more of the following AWS services:
• AWS Glue
• Amazon S3
• AWS Lambda
• Experience with Databricks data engineering, including:
• Data transformation and processing
• Delta Lake
• PySpark and/or Spark
• Notebook-based development
• Experience with Snowflake development, including:
• Data loading and transformation
• SQL development
• Data warehouse concepts
• Performance optimization
• Experience with any of the following preferred/optional skills will be an advantage:
• ODI (Oracle Data Integrator)
• OAC (Oracle Analytics Cloud)
• PL/SQL
• Amazon MSK / Kafka
• Datadog
• Python
• PySpark
• CI/CD and DevOps practices
• Git
• Cloud monitoring and logging
• Oracle databases
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Strong data engineering and analytical skills.
• Good understanding of cloud data platforms.
• Strong SQL and data transformation capabilities.
• Strong problem-solving and troubleshooting skills.
• Good communication and collaboration skills.
• Ability to work independently as well as part of a team.
• Strong focus on data quality and solution reliability.
• Ability to learn and work across new technologies and platforms.