Senior Data Engineer

Job Description (JD)

Lead / Senior Data Engineer

Location: Hyderabad, India

Experience: 8+ Years

Employment Type: Full-Time

Notice Period : Immediate / 15 Days.

About the Role

We are looking for a highly skilled and results-driven Lead / Senior Data Engineer with extensive experience in designing, developing, and managing enterprise-scale data platforms across Azure and AWS cloud ecosystems. The ideal candidate will bring strong expertise in Azure Databricks, Snowflake, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Azure Data Factory, Python, PySpark, and SQL, combined with proven leadership experience in delivering modern data warehouse and lakehouse solutions.

This role requires a hands-on technical leader who can drive architecture decisions, build scalable and governed data platforms, mentor engineering teams, and collaborate closely with business stakeholders to deliver data-driven solutions that enable analytics, reporting, and AI/ML initiatives.

Key Responsibilities

Data Architecture & Engineering

• Design, develop, and maintain scalable and high-performance ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Python, PySpark, and SQL.
• Build metadata-driven and parameterized data ingestion frameworks supporting both batch and real-time processing.
• Architect modern Lakehouse and Data Warehouse solutions leveraging Azure and AWS cloud technologies.
• Develop analytics-ready datasets and enterprise data products for business intelligence, reporting, and advanced analytics use cases.
• Implement robust data quality frameworks, reconciliation processes, and validation controls.

Azure Databricks & Lakehouse Development

• Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Build and manage Delta Live Tables (DLT) pipelines for reliable and quality-enforced data processing.
• Develop PySpark and Spark SQL transformations for large-scale data workloads.
• Implement Unity Catalog governance, including data lineage, auditing, metadata management, and fine-grained access controls.
• Optimize Databricks workloads through partitioning strategies, caching, broadcast joins, Adaptive Query Execution (AQE), and data-skew handling.

Snowflake Data Warehousing

• Design and implement Snowflake-based data warehouse solutions.
• Develop and optimize Snowflake schemas, data models, RBAC frameworks, and data ingestion processes using Snowpipe and COPY operations.
• Perform SQL query tuning, warehouse sizing, performance optimization, and cost management.
• Support reporting, analytics, and enterprise BI workloads through scalable and efficient data models.

Cloud & Streaming Solutions

• Develop real-time and event-driven data processing solutions using Azure Event Hubs, AWS Kinesis, Azure Functions, and AWS Lambda.
• Build scalable ingestion frameworks for streaming and batch data sources.
• Support hybrid and multi-cloud data ecosystems across Azure and AWS platforms.

Data Modeling & Governance

• Design dimensional data models using Star Schema and Snowflake Schema methodologies.
• Implement Slowly Changing Dimensions (SCD Type 1, 2, and 3).
• Support Data Mesh and domain-oriented data product initiatives.
• Establish enterprise data governance practices, security controls, and data stewardship standards.

DevOps & Automation

• Develop and maintain CI/CD pipelines using Azure DevOps and GitHub Actions.
• Implement automated deployment, monitoring, testing, and release management processes.
• Promote engineering best practices through code reviews, version control standards, and development frameworks.

Leadership & Stakeholder Management

• Lead technical design discussions, architectural reviews, and project delivery activities.
• Mentor and guide data engineering teams through technical coaching and knowledge sharing.
• Collaborate with business stakeholders, architects, data scientists, and analytics teams to define and implement data solutions.
• Drive technical strategy and ensure successful project execution from design through production support.

Required Qualifications

Education

• Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related discipline.

Experience

• 8+ years of experience in Data Engineering, Data Warehousing, and Cloud Data Platforms.
• Proven experience leading enterprise data transformation and modernization initiatives.
• Strong track record of delivering scalable data solutions in Azure cloud environments.

Technical Skills

Cloud Platforms

• Microsoft Azure
• Amazon Web Services (AWS)

Azure Technologies

• Azure Databricks
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure SQL Database
• Azure Event Hubs
• Azure Functions
• Microsoft Fabric

Databricks & Data Governance

• PySpark
• Spark SQL
• Delta Lake
• Delta Live Tables (DLT)
• Unity Catalog
• Databricks Workflows

Data Warehousing

• Snowflake
• Azure Synapse Analytics
• Amazon Redshift

Programming Languages

• Python
• PySpark
• SQL / T-SQL
• Pandas

AWS Services

• AWS Glue
• Amazon S3
• Amazon Athena
• AWS Lambda
• AWS Step Functions
• AWS IAM
• Amazon CloudWatch

Data Modeling

• Medallion Architecture
• Star Schema & Snowflake Schema
• Data Warehousing
• Data Mesh
• Lakehouse Architecture
• SCD Type 1, Type 2, and Type 3

DevOps & Collaboration

• Azure DevOps
• GitHub Actions
• Git Version Control
• Agile/Scrum Methodologies

Preferred Qualifications

• Databricks Certification (Associate/Professional).
• Snowflake Certification.
• Microsoft Azure Data Engineering Certifications.
• Experience supporting AI/ML and advanced analytics workloads.
• Experience implementing enterprise data governance frameworks.
• Exposure to Microsoft Fabric and modern Lakehouse platforms.

Key Competencies

• Technical Leadership
• Solution Architecture
• Data Engineering Best Practices
• Performance Optimization
• Problem Solving & Root Cause Analysis
• Stakeholder Management
• Team Mentoring & Coaching
• Strategic Decision Making
• Communication & Collaboration