Senior AWS Data Engineer (Python & Spark)

Key Responsibilities

• Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
• Build and optimize high-performance data pipelines using Python and Spark.
• Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
• Implement and maintain Lakehouse architecture solutions using modern data engineering principles.
• Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
• Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
• Break down solution designs into epics, stories, and technical deliverables.
• Ensure data quality, reliability, scalability, and performance across data platforms.
• Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
• Create and maintain technical documentation, architecture diagrams, and implementation standards.
• Support Agile delivery methodologies and contribute to sprint planning and technical estimations.

Required Skills

• Proven experience as a Senior Data Engineer.
• Strong expertise in Python and Apache Spark.
• Extensive experience building ETL pipelines and data processing frameworks.
• Hands-on experience with AWS services including:
• AWS EMR
• AWS Glue
• AWS Lambda
• AWS Step Functions
• API Gateway
• Amazon Athena
• Experience working with Data Lakehouse architectures.
• Strong understanding of data modelling and enterprise data platform design.
• Experience handling large-scale data processing workloads.
• Knowledge of data engineering best practices including testing, validation, and performance optimization.
• Strong communication and stakeholder management skills.

Preferred Skills

• Financial Services or Capital Markets experience.
• Experience working with market data and reference data platforms.
• Knowledge of Data Governance and Data Management frameworks.
• Experience with:
• Metadata Management
• Master Data Management (MDM)
• Data Quality frameworks
• AWS Certifications such as:
• AWS Certified Solutions Architect
• AWS Certified Data Analytics

Bonus Skills

• Knowledge of Indices and Index Construction methodologies.
• Asset Management domain experience.
• Exposure to modern open-table formats such as Apache Iceberg.

Desired Candidate Profile

• Strong technical leadership and mentoring capabilities.
• Excellent problem-solving and analytical skills.
• Ability to communicate complex technical concepts to technical and non-technical audiences.
• Experience working in Agile product teams.
• Strong ownership mindset with a focus on delivering scalable and maintainable solutions.
• Passion for modern cloud-based data engineering and platform modernization.