Data Engineer (Data Retention Project)

Job Details

Data Engineer (Data Retention Project)

Bengaluru

5 Days Onsite

Shift Timing - 02:00 PM to 11:00 PM

Job Description

Role & Responsibilities:
• Design and maintain scalable data pipelines for data ingestion, transformation, and processing.
• Build and optimize ETL/ELT workflows using PySpark, Databricks, SQL, and Airflow.
• Develop data models and solutions for retention, reporting, and analytics needs.
• Ensure data quality, integrity, and reliability through validation and monitoring frameworks.
• Integrate data from multiple sources and optimize processing performance.
• Collaborate with cross-functional teams to deliver data solutions and resolve data issues.
• Maintain technical documentation, data lineage, and mapping artifacts.
• Support production deployments, troubleshooting, and continuous improvement through engineering best practices.

Technical Skills Requirements:
1. PySpark & Spark SQL: Hands-on experience writing production-grade PySpark code
2. Databricks: Proven experience working within Databricks environments, including cluster configuration and Delta Lake
3. Apache Airflow: Strong experience building and managing pipelines in Apache Airflow
4. SQL Mastery: Advanced SQL skills (window functions, CTEs, query optimization, indexing strategies)
5. Data Modeling: Solid understanding of modern data warehousing concepts
6. Version Control: Proficiency with Git and CI/CD deployment workflows
7. AWS: Experience in Amazon Web Services (AWS) cloud infrastructure
8. ETL/ELT: Proven track record of designing, building, and optimizing scalable ETL/ELT pipelines and data platforms