Data Engineer

Key Responsibilities

Design and develop scalable Python and PySpark/Spark-based data pipelines for batch data processing and integration.

Build reusable frameworks and components for data ingestion, transformation, validation, and publishing.

Work with structured and semi-structured data from databases, files, APIs, and enterprise data platforms.

Develop and optimize complex SQL and Spark transformations for large-volume datasets.

Implement data-quality checks, exception handling, logging, reconciliation, and operational controls.

Write modular, maintainable, testable Python/PySpark code following enterprise engineering standards.

Use GitHub for source-code management, branching, pull requests, and collaborative development.

Work with GitHub Actions and/or Harness to support CI/CD workflows for building, testing, and deploying data applications.

Develop and maintain workflow orchestration using Apache Airflow or equivalent enterprise scheduling/orchestration platforms.

Troubleshoot pipeline failures, performance issues, data-quality problems, and production defects.

Participate in code reviews, unit/integration testing, release activities, and production support.

Collaborate with data architects, platform engineers, DevOps teams, product owners, and other engineering teams.

Required Technical Skills

Must Have

Strong hands-on programming experience with Python.

Strong experience with Apache Spark / PySpark.

Strong working knowledge of SQL and data transformation techniques.

Experience developing production-grade ETL/ELT and batch data pipelines.

Understanding of distributed data-processing concepts and Spark performance optimization.

Experience working with Git/GitHub, including branching, pull requests, and code reviews.

Practical exposure to CI/CD, preferably using GitHub Actions and/or Harness.

Experience with Apache Airflow or another scheduling/workflow orchestration tool.

Experience with automated testing, logging, error handling, deployment, and production troubleshooting.