Senior/Lead Data Engineer

Job Description:

• Job title : Lead / Senior Data Engineer
• Experience: 8+ years
• Location: Chennai

Roles and Responsibilities:

Role Overview

• The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities

• Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.
• Define technical strategy, architecture, and best practices for development and data engineering.
• Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.
• Construct infrastructure for efficient ETL processes from various sources and storage systems.
• Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.
• Lead the implementation of algorithms and prototypes to transform raw data into useful information.
• Review code for quality, scalability, and performance.
• Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.
• Develop analytical tools, programs, and reporting mechanisms.
• Create data validation methods and data analysis tools.
• Interpret data trends and patterns to establish operational alerts.
• Conduct complex data analysis and present results effectively.
• Prepare data for prescriptive and predictive modeling.
• Ensure compliance with data governance and security policies.
• Troubleshoot, debug, and resolve complex technical issues.
• Drive continuous improvement in software and data development processes, tools, and methodologies.
• Mentor and guide engineers through code reviews, technical discussions, and training.
• Ensure timely delivery of projects while maintaining high engineering standards.
• Continuously explore opportunities to enhance data quality and reliability.
• Apply strong programming and problem-solving skills to develop scalable solutions.
• Demonstrate passion for testing strategy, problem-solving, and continuous learning.
• Willingness to acquire new skills and knowledge.
• Possess a product/engineering mindset to drive impactful data solutions.
• Experience working in distributed environments with global teams.
• Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

• Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.
• High proficiency in Python, Scala, and Spark for applied large-scale data processing.
• Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
• Solid understanding of batch and streaming data processing techniques.
• Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
• Expert-level ability to write complex, optimized SQL queries across extensive data volumes.
• Experience with RDBMS and OLAP databases such as MySQL and Snowflake.
• Familiarity with Agile methodologies.
• Obsession for service observability, instrumentation, monitoring, and alerting.
• Knowledge or experience in architectural best practices for building data lakes.
• Qualifications -Bachelor's degree in computer science, Engineering, Information Systems, or related field