Senior Data Engineer

Senior Data Engineer Job Description Must Have

• Python & Advanced SQL
• Spark / distributed processing / PySpark
• Cloud platforms (AWS) Modules like Lambda, Glue, EMR, Dynamo DB, S3, Redshift, EC2, RDS, Step Functions, Cloud Watch, Systems Manager, Event Bridge, DMS
• SSIS and experience with C# scripting
• Data pipeline automation

Nice to have

• Kafka/Kinesis streaming
• Basic knowledge of AI
• Power BI

Job Summary Responsibilities:

• Able to understand existing jobs in SSIS and ability to maintain them in production including resolving issues and making changes to them as per requirement.
• Basic understanding of C# coding.
• Hands on in Python and advanced SQL.
• Good understanding of Spark eco-system and PySpark programming.
• Able to convert existing SSIS workflows to ETL pipelines running on AWS. That includes knowledge of Lambda, Glue or EMR, Systems Manager, Event Bridge, DMS
• Knowledge of Dynamo DB, Redshift, RDS (Preferably Postgre)
• Awareness of reporting tool like Power BI.
• Monitor and troubleshoot migration processes, resolving issues in a timely manner.
• Work closely with business stakeholders, developers, and administrators to understand data requirements.
• Document migration procedures, processes, and best practices.

Required Skills and Qualifications

• Bachelors degree in Computer Science, Information Technology, or related field.
• 5+ years of experience in data migration or similar roles.
• Strong proficiency in SSIS.
• Proven experience in ETL tools like SSIS and data migration pipelines build on AWS.
• Familiarity with data modeling and schema design principles.
• Ability to analyze and resolve complex data-related issues.
• Strong analytical and problem-solving abilities.
• Excellent communication and collaboration skills.
• Ability to work independently and in a team-oriented environment.

Preferred

• Familiarity with Agile project management methodologies.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Data & AI - Data Engineering - Data Quality & Validation
• Big Data - Big Data - Pyspark
• Database - Database Programming - SQL
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Beh - Communication and collaboration