Data Engineer

Role: Lead Data solutions

Experience: 7+ Years

Notice Period: 0-30 days

Location: Bangalore

Work Mode: Hybrid

Job Description:

We are looking for candidates with hands-on design experience in:

ADF, Databricks, SQL, Data analytics, Power bi, Pyspark, Data engineering

Note: Need 2-3 years of experience in leading small size team(2-4)

As a Lead – Data Solution and Analytics, you will  Lead a team responsible for the build, support, and optimization modern data solutions and analytical products across the Azure Data Platform. You’ll design and manage robust data pipelines (Azure Data Factory, Azure Databricks, Data Lake, Spark.SQL), SQL development and Data Analysis. You will collaborate closely with business stakeholders, data engineers, and data scientists to maintain reliable, secure, and scalable data products that drive decision-making—particularly within Supply Chain and CPG contexts.

Note: This role includes production support, monitoring, and potential rotational shift coverage (including night shifts if required).

Key Responsibilities:

Required Qualifications:

Design, develop, customize, and manage data integration tools, data lakes, warehouses, and analytical systems on Azure.

Build, scale, and harden data pipelines (internal & external sources) using Azure Data Factory, Azure Databricks (PySpark/Spark SQL), Azure Data Lake, Azure Synapse, and Azure SQL.

Deploy data pipelines into production environments by enriching the model with data stored in a Data Lakes or coming directly from data sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.

Create reusable SQL scripts for reporting, data validation, and ETL processes.

Design, develop, and optimize complex SQL/T-SQL queries, stored procedures, and performance tuning for large-scale datasets.

Perform data profiling, cleansing, and validation to ensure accuracy and reliability. [ Data Analytics]

Develop data models and analytical frameworks to support reporting and decision-making.

Mentor and Build a Data Engineering Support team composed of Data Engineering Resources

Own automation, observability, and monitoring frameworks—capturing operational KPIs and data quality metrics; set up alerting and runbooks.

Monitor performance & stability, triage incidents, and adapt pipelines as data/models/requirements evolve.

Implement best practices for systems integration, security (RBAC, Key Vault, RLS/OLS), performance, cost optimization, and data governance.

Governance and Compliance

Establish clear policies and procedures for support operations.

Ensure compliance with industry standards and regulations (e.g., ITIL, ISO).

Regularly review and update governance frameworks.

CAPA (Corrective and Preventive Action)

Define processes for identifying, documenting, and investigating non-conformances.

Implement corrective actions to address identified issues.

Develop preventive measures to mitigate future risks.

Monitor effectiveness of CAPA implementation.

Major Incident Process

Establish criteria for what constitutes a major incident.

Define roles and responsibilities during major incidents (e.g., incident commander, communication lead).

Create a communication plan for stakeholders during incidents.

Conduct post-incident reviews to capture lessons learned and improve processes.

Root Cause Analysis (RCA)

Develop a standardized approach for conducting RCA.

Utilize tools and techniques (e.g., 5 Whys, Fishbone Diagram) to identify root causes.

Document findings and ensure accountability for corrective actions.

Communicate insights and preventive measures across teams.

Failure Mode and Effects Analysis (FMEA)

Identify potential failure modes in digital products or processes.

Assess the impact and likelihood of each failure mode.

Prioritize risks based on severity and probability.

Develop action plans to mitigate high-priority risks.