Sr Associate-Date Engineer

Job purpose

• We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies.
• The ideal candidate will have experience developing data pipelines using Python and Spark, working with modern data lake platforms such as Databricks and Snowflake, and deploying solutions on AWS. Experience with Dagster for data orchestration is highly desirable.

Key responsibilities

Data Platform Development

• Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.
• Build reusable, production-grade data engineering components and frameworks.
• Develop and optimize high-volume batch and streaming data processing solutions.
• Implement robust testing, monitoring, and observability for data pipelines.
• Troubleshoot complex data processing and performance issues.

Lakehouse & Data Engineering

• Build and maintain modern lakehouse architectures using Databricks, Snowflake, and AWS data services.
• Design scalable data models and storage strategies for structured and semi-structured data.
• Develop ingestion, transformation, validation, and enrichment pipelines.
• Support data quality, lineage, and governance initiatives.

Orchestration & Automation

• Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.
• Automate data operations and deployment processes.
• Monitor pipeline performance and optimize reliability and efficiency.

Cloud Engineering

• Build cloud-native data solutions on AWS.
• Integrate data services across storage, processing, orchestration, and analytics platforms.
• Implement security, scalability, and resiliency best practices.

Engineering Excellence

• Write clean, maintainable, and well-tested code.
• Participate in peer code reviews and technical design discussions.
• Collaborate with product managers, architects, and engineering teams.
• Follow software engineering best practices, including CI/CD and version control.

Key competencies

Experience

• 5+ years of professional experience in Data Engineering.
• Hands-on experience building production-grade data platforms and pipelines.
• Strong software engineering and coding background.

• Python (Expert Level)
• PySpark / Apache Spark
• Data Pipeline Development
• Object-Oriented Programming
• Debugging and Performance Optimization
• REST APIs
• Git

• Experience with one or more:
• Databricks
• Snowflake
• Delta Lake
• Apache Iceberg
• Modern Lakehouse Architectures

• S3
• Lambda
• Glue
• EMR
• ECS/EKS
• Step Functions
• RDS
• CloudWatch

• ETL/ELT Development
• Data Modeling
• Data Quality Frameworks
• Data Validation and Monitoring
• CI/CD for Data Pipelines

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.