Senior/Lead Data Engineer

Responsibilities-

1. Data Engineering & Pipeline Development

Monitor and optimize compute usage across workloads.

• Evaluate storage patterns, caching techniques, and data partitioning. • Design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).

• Architect data ingestion frameworks to process structured, semi-structured, and unstructured data.

• Develop re-usable frameworks for data quality, data validation, and exception handling.

2. Data bricks & Cloud Platform Expertise

• Optimize Data bricks clusters, jobs, and queries for performance and cost efficiency.

• Manage cluster configurations, auto scaling, job scheduling, and deployment workflows.

• Implement best practices for notebook modularization, CI/CD, and version control.

3. Data Architecture & Modelling

• Design end-to-end data models including lakehouse architecture, medallion layers (bronzesilvergold).

• Collaborate with Data Science team to prepare training datasets and feature engineering pipelines.

• Implement governance, cataloguing, and lineage using cloud-native tools.

4. Performance & Cost Optimization

• Recommend architecture and platform-level improvements to reduce cost.

5. Collaboration & Documentation

• Work closely with cross-functional teams including Data Science, Analytics, Business, and IT Infra.

• Maintain detailed documentation for pipelines, data flows, and technical designs.

• Support production deployments, monitoring, and troubleshooting.

Educational Qualifications Graduation/Post Graduation

Competencies Required: –

• Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.

• Experience in architecting and scaling cloud-based pipelines in Azure • Strong SQL expertise and understanding of distributed data processing.

• Experience with source versioning (Git), CI/CD, and DevOps workflows. • Proficiency with REST APIs, Databricks SQL Warehouses, and Unity Catalog.

• Good understanding of data warehousing concepts, lakehouse patterns, and metadata management.

• Exposure to ML model deployment workflows

• Intermediate proficiency in Power BI (dataset preparation, modeling, dashboards).

Experience - 5+ years of experience as a Data Engineer or in a similar role.