Senior Data Engineer

Responsibilities:

• Design, develop, and maintain scalable data architecture and data engineering solutions on Azure and Databricks.
• Build robust batch and streaming data pipelines using Apache Spark, PySpark, Delta Lake, SQL, and related technologies.
• Develop and optimize complex data processing frameworks for structured, semi-structured, and high-volume datasets.
• Implement enterprise data lakehouse solutions aligned to business, analytics, and AI/ML requirements.
• Ensure data pipelines are reliable, performant, reusable, and built with strong engineering standards.
• Collaborate with business teams, data architects, analysts, data scientists, and cloud engineers to translate requirements into technical solutions.
• Support AI and ML initiatives by creating curated, governed, and feature-ready datasets.
• Work on performance tuning, troubleshooting, and optimization of Databricks workloads and Spark jobs.
• Establish and maintain data quality, lineage, security, governance, and compliance controls.
• Drive best practices in CI/CD, code quality, testing, release management, and operational support for data solutions.
• Contribute to platform improvements, architecture reviews, and technology evaluation activities
• Deliver high-quality, scalable, and secure data solutions aligned to business objectives.
• Enable data-driven and AI-ready capabilities through well-engineered data platforms.
• Contribute to Tata Digital's long-term data strategy and platform maturity.

Requirements:

• Demonstrate strong technical leadership in the Azure and Databricks ecosystems.
• Bachelor's or master's degree in computer science, information technology, engineering, or a related field.
• 6-8 years of experience in data engineering, data platform engineering, or related roles.
• Strong hands-on experience with Azure cloud services and Azure Databricks.
• Deep expertise in Apache Spark / PySpark, Delta Lake, SQL, and distributed data processing.
• Strong understanding of data architecture, data modeling, and modern lakehouse patterns.
• Experience building and supporting complex ETL/ELT pipelines in enterprise environments.
• Working exposure to AI and ML data pipelines, feature preparation, or data support for AI-based solutions.
• Experience with batch and streaming data processing.
• Solid understanding of data governance, security, access control, and data quality practices.
• Experience with orchestration tools such as Azure Data Factory, Databricks Workflows, Airflow, or similar.
• Strong debugging, analytical thinking, and problem-solving skills.

Competencies:

• Strong technical ownership and accountability.
• Ability to work on complex, high-impact data problems.
• Collaborative mindset with cross-functional stakeholders.
• Strong communication and documentation skills.
• Adaptability to evolving data and AI technologies.