Data Engineer

Roles and responsibilities

As a Data Engineer specializing in Data Integration, you will design and build solutions to transfer data from operational and external environments to the business intelligence environment. You will utilize various tools to create and implement Extract, Transform, and Load (ETL) processes, ensuring the seamless flow of data throughout the business intelligence solution's lifecycle. Your primary responsibilities will include:

• Design and develop scalable data integration pipelines to ingest data from multiple sources (core systems, third-party systems, and streaming platforms) into the data warehouse. 
• Build and maintain batch and near real-time ETL/ELT pipelines using tools such as AWS Glue, AWS DMS or Apache Kafka. 
• Implement data transformation, cleansing, and validation logic to ensure high data quality and consistency. 
• Optimize data pipelines and queries for performance, scalability, and cost efficiency, especially within Amazon Redshift. 
• Monitor, troubleshoot, and resolve data pipeline issues, ensuring high availability and reliability of data systems. 
• Collaborate with stakeholders to understand data requirements and deliver robust data solutions. 
• Maintain proper documentation and follow best practices in data engineering, including logging, error handling, and version control. 

Required Skills & Qualifications

• 2+ years of experience in Data Engineering or related roles 
• Strong proficiency in Advanced SQL (joins, window functions, query optimization, CTE function) 
• Hands-on experience with Python for data processing 
• Experience building ETL/ELT pipelines using tools like AWS Glue or similar frameworks 
• Working knowledge of streaming systems such as Apache Kafka 
• Familiarity with AWS services such as Amazon S3, Amazon Redshift, and AWS DMS 
• Understanding of data modeling concepts (OLAP vs OLTP, star schema, normalization)
• Experience working with different types of databases, such as Relational (SQL) Databases, NoSQL Databases, Data Warehouses (OLAP) etc with a clear understanding of their use cases, performance characteristics, and data modeling approaches.

Good to Have

• Experience with Apache Spark / PySpark
• Knowledge of data quality frameworks and monitoring tools 
• Understanding of cloud cost optimization techniques
• Some exposure in LLM training, hosting, modelling creation of vector database etc.