Senior Data Engineer

Job Role: Senior Data Engineer AWS 

We are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable, high-performance data pipelines and data platforms. The ideal candidate will have strong hands-on expertise in Python, PySpark, SQL, AWS, AWS Glue, Data Lakes, and Data Warehousing.

The role will primarily focus on building robust ETL/ELT pipelines, developing scalable data processing solutions, optimizing Snowflake workloads, and integrating multiple AWS services to deliver reliable, business-ready datasets. The candidate should have strong experience in designing production-grade data pipelines with a focus on performance, scalability, data quality, monitoring, and cost optimization.

Responsibilities:

1. Data Pipeline Development & Engineering

• Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines for ingesting and transforming data from multiple source systems.
• Develop robust data ingestion and transformation workflows using Python, PySpark, SQL, and AWS Glue.
• Build reusable, production-grade data engineering solutions with appropriate error handling, logging, validation, and monitoring.
• Process large volumes of structured and semi-structured data efficiently.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Collaborate with engineering, analytics, and business teams to deliver reliable, business-ready datasets.

2. Snowflake Data Engineering & Data Warehousing

• Design, develop, and maintain scalable data warehousing solutions using Snowflake/redshift
• Build and optimize databases, schemas, tables, views, and SQL transformations.
• Develop efficient ETL/ELT workflows for loading and transforming data within Snowflake.
• Implement best practices for data modeling, data organization, and performance tuning.
• Troubleshoot and resolve data processing and query performance issues.

3. AWS & Cloud Data Platform

• Design and implement scalable data engineering solutions on Amazon Web Services (AWS).
• Build and maintain data pipelines using AWS Glue for data ingestion, transformation, and processing.
• Utilize Amazon S3 as a scalable storage layer for raw, processed, and curated datasets.
• Integrate AWS services such as AWS Glue, S3, Lambda, and CloudWatch for end-to-end data processing and monitoring.
• Develop automated and serverless data processing workflows where applicable.
• Implement cloud best practices related to security, scalability, reliability, and cost optimization.

4. PySpark & Distributed Data Processing

• Develop and optimize PySpark applications for large-scale, distributed data processing.
• Build scalable batch data processing and transformation workflows.
• Process structured and semi-structured datasets from multiple source systems.
• Implement efficient transformations, partitioning, joins, aggregations, and performance optimization techniques.
• Troubleshoot and optimize PySpark and AWS Glue workloads for improved execution performance and resource utilization.

5. SQL, Data Transformation & Data Modeling

• Develop complex and optimized SQL queries for data transformation, analysis, and reporting requirements.
• Design and implement efficient data models to support analytics and business use cases.
• Work with data warehousing concepts, including dimensional modeling, fact tables, dimension tables, and star/snowflake schemas.
• Build reusable SQL transformations and data processing logic.
• Ensure consistency, accuracy, and reliability of data across the data platform.
• Identify and optimize inefficient queries and data processing workflows.

6. Data Quality, Validation & Reliability

• Implement robust data quality checks, validation frameworks, and reconciliation processes across data pipelines.
• Develop mechanisms for handling data inconsistencies, failures, and processing errors.
• Implement appropriate error handling, retries, logging, and recovery mechanisms.
• Monitor data pipelines and proactively identify data quality and reliability issues.
• Perform root-cause analysis and implement permanent solutions for recurring production issues.
• Ensure pipelines meet required standards for data accuracy, completeness, and reliability.

Skills:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 2+ years of hands-on Data Engineering experience with strong experience building production-grade data pipelines.
• Strong hands-on expertise in Snowflake, including database design, SQL development, query optimization, and performance tuning.
• Strong proficiency in Python and SQL with experience handling large and complex datasets.
• Strong hands-on experience with PySpark for large-scale, distributed data processing.
• Strong experience with AWS, particularly AWS Glue and Amazon S3.
• Experience integrating AWS services such as Lambda and CloudWatch into data engineering workflows.
• Strong understanding of ETL/ELT methodologies and data pipeline architecture.
• Strong knowledge of Data Lakes and Data Warehousing concepts.
• Experience with data modeling, including dimensional modeling, star schema, and snowflake schema.
• Strong understanding of data quality, validation, error handling, monitoring, and production support.
• Experience optimizing Data warehouses, queries, and compute usage for performance and cost efficiency.
• Experience working with structured and semi-structured data formats.
• Familiarity with Git, CI/CD, and automated deployment practices would be an added advantage.
• Strong problem-solving skills and the ability to troubleshoot complex data pipeline and production issues.
• Ability to work collaboratively with business, analytics, and engineering stakeholders.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status