Python / Spark Data Engineer

Key Responsibilities

Design and develop scalable Python and PySpark/Spark-based data pipelines for batch data processing and integration.

Build reusable frameworks and components for data ingestion, transformation, validation, and publishing.

Work with structured and semi-structured data from databases, files, APIs, and enterprise data platforms.

Develop and optimize complex SQL and Spark transformations for large-volume datasets.

Implement data-quality checks, exception handling, logging, reconciliation, and operational controls.

Write modular, maintainable, testable Python/PySpark code following enterprise engineering standards.

Use GitHub for source-code management, branching, pull requests, and collaborative development.

Work with GitHub Actions and/or Harness to support CI/CD workflows for building, testing, and deploying data applications.

Develop and maintain workflow orchestration using Apache Airflow or equivalent enterprise scheduling/orchestration platforms.

Troubleshoot pipeline failures, performance issues, data-quality problems, and production defects.

Participate in code reviews, unit/integration testing, release activities, and production support.

Collaborate with data architects, platform engineers, DevOps teams, product owners, and other engineering teams.

Required Technical Skills

Must Have

Strong hands-on programming experience with Python.

Strong experience with Apache Spark / PySpark.

Strong working knowledge of SQL and data transformation techniques.

Experience developing production-grade ETL/ELT and batch data pipelines.

Understanding of distributed data-processing concepts and Spark performance optimization.

Experience working with Git/GitHub, including branching, pull requests, and code reviews.

Practical exposure to CI/CD, preferably using GitHub Actions and/or Harness.

Experience with Apache Airflow or another scheduling/workflow orchestration tool.

Experience with automated testing, logging, error handling, deployment, and production troubleshooting.

Good to Have

Experience working with cloud or enterprise data platforms.

Knowledge of data warehousing and dimensional/data-modeling concepts.

Experience integrating data from databases, REST APIs, files, and other enterprise sources.

Familiarity with metadata, data lineage, data quality, and data governance concepts.

Exposure to containerization and deployment technologies such as Docker/Kubernetes.

Experience working within banking or other highly regulated enterprise environments.

DevOps Expectations