Senior Data Engineer

Description

Senior Data Engineer Mindera

At Mindera, we build technology products that create meaningful business impact. We work with modern cloud technologies and encourage engineering ownership, collaboration, continuous learning, and self-organization.

Our Data Engineering teams build scalable, reliable, and high-performance data platforms that enable analytics, reporting, machine learning, and data-driven decision-making.

Requirements

What We’re Looking For

Key Responsibilities

• Design and develop scalable ETL/ELT data pipelines using AWS Glue, PySpark, Python, and SQL.
• Develop and maintain distributed data processing applications using Apache Spark.
• Build and orchestrate data workflows using Apache Airflow.
• Develop AWS Glue jobs for batch and incremental data processing.
• Work with Amazon S3 and other AWS services to build cloud-based data solutions.
• Perform complex data transformations, cleansing, validation, and enrichment.
• Write optimized and maintainable SQL queries for large datasets.
• Optimize Spark and PySpark jobs for performance, scalability, and resource utilization.
• Implement data partitioning, caching, broadcast joins, and other Spark optimization techniques.
• Design reliable and fault-tolerant data pipelines with appropriate error handling and retry mechanisms.
• Implement data quality checks and monitoring across data pipelines.
• Troubleshoot production data pipeline failures and performance issues.
• Work with large-scale structured and semi-structured datasets.
• Participate in technical design and architecture discussions.
• Conduct code reviews and ensure adherence to engineering best practices.

Required Skills

Python

• Strong programming experience in Python.
• Experience writing production-quality, modular, and reusable code.
• Knowledge of exception handling, logging, testing, and debugging.
• Familiarity with Python data-processing libraries.

SQL

• Strong SQL skills with experience handling large datasets.
• Complex joins and subqueries.
• CTEs and window functions.
• Aggregations and analytical queries.
• Query optimization and performance tuning.
• Data deduplication and incremental processing logic.

Apache Spark / PySpark

• Strong hands-on experience with Spark and PySpark.
• Understanding of Spark architecture and execution model.
• DataFrames and Spark SQL.
• Transformations and actions.
• Partitioning and repartitioning.
• Shuffle optimization.
• Broadcast joins.
• Caching and persistence.
• Handling data skew.
• Spark job troubleshooting and performance optimization.

AWS Glue

• Hands-on experience developing AWS Glue ETL jobs.
• Experience with Glue Data Catalog and Crawlers.
• Glue Spark jobs.
• DynamicFrames and DataFrames.
• Job bookmarks and incremental processing.
• Partitioned data processing.
• Schema evolution and data quality.
• Integration with Amazon S3 and other AWS services.

Apache Airflow

• Experience designing and developing Airflow DAGs.
• Task dependencies and scheduling.
• Operators and sensors.
• Retries and failure handling.
• Backfills and catchup.
• XCom and task communication.
• Monitoring and troubleshooting DAGs.
• Designing idempotent and reusable workflows.

AWS

• Strong experience with AWS data services.
• Amazon S3.
• AWS Glue.
• IAM.
• CloudWatch.
• Familiarity with services such as Lambda, Athena, Redshift, or EMR is an advantage.

Preferred Skills

• Experience with modern data lake / data warehouse architectures.
• Experience with Parquet and other columnar storage formats.
• Knowledge of data modeling and dimensional modeling.
• Experience with CI/CD and Git-based development workflows.
• Familiarity with Docker and infrastructure-as-code concepts.
• Experience with Agile/Scrum methodologies.
• Experience working with high-volume and large-scale datasets.
• Understanding of data governance, security, and data quality practices.

Senior-Level Expectations

• Independently own data engineering projects from requirements through production.
• Design scalable and maintainable data architectures.
• Make appropriate technology and architecture decisions.
• Identify and resolve performance bottlenecks.
• Troubleshoot complex production issues.
• Establish coding, testing, and data quality standards.
• Mentor other engineers and contribute to technical decision-making.
• Communicate effectively with both technical and non-technical stakeholders.

Benefits

We offer

• Flexible working hours (self-managed)
• Annual bonus, subject to company performance
• Access to Udemy online training and opportunities to learn and grow within the role

At Mindera we use technology to build products we are proud of, with people we love.

Software Engineering Applications, including Web and Mobile, are at the core of what we do at Mindera.

We partner with our clients, to understand their products and deliver high-performance, resilient and scalable software systems that create an impact on their users and businesses across the world.

You get to work with a bunch of great people, and the whole team owns the project together.

Our culture reflects our lean and self-organisation attitude.

We encourage our colleagues to take risks, make decisions, work in a collaborative way and talk to everyone to enhance communication. We are proud of our work and we love to learn all and everything while navigating through an Agile, Lean and collaborative environment.