SENIOR SOFTWARE ENGINEER - Azure Databricks

Azure Data Engineer

Experience Required: 5 6 Years

Role Overview

We are looking for an experienced Azure Data Engineer to design, build, and support end-to-end data pipelines on the Azure ecosystem. The role involves extracting and transforming data from multiple source systems, building robust ADF/Synapse/Databricks pipelines, and supporting business-critical reporting and reconciliation processes.

Skills Required

• Azure Data Factory (ADF)
• Azure Databricks (ADB)
• Azure Active Directory
• Azure Storage
• Azure Data Lake Services (ADLS)
• Azure Key Vault
• Azure Logic Apps
• Azure SQL DB
• Azure HDInsight
• Good knowledge of Azure DevOps (ADO) services Repos, Boards, Build Pipelines (CI/CD)
• Good knowledge of developing data engineering frameworks and notebooks using Azure Databricks with Spark SQL, PySpark, and Python
• Good knowledge of Azure Synapse Analytics

Good-to-Have Skills

• GitHub
• PowerApps
• Azure Monitor and Log Analytics
• Power BI
• REST APIs

Daily Activities

• Extracting data from source systems and files in any format from SFTP locations.
• Processing the extracted data and storing it into staging tables in Azure Blob Storage.
• Transforming data based on the mapping (STTM) document to meet given requirements in ADB using SQL, Python, and PySpark code.
• Generating a final TLM file or report and loading it into the Smartstream environment.
• Building pipelines end-to-end using ADF and Synapse.
• Monitoring pipelines and alerts in ADF.
• Implementing new enhancements for the LBU.
• Analyzing and fixing issues raised by the business.
• Updating JIRA and documenting in Confluence.

Ideal Candidate Profile

• 56 years of hands-on experience in Azure data engineering, with strong exposure to ADF, Databricks, and Synapse.
• Solid programming skills in SQL, Python, and PySpark for building transformation logic and data engineering frameworks.
• Experience working with SFTP-based data extraction and file-based integrations.
• Familiarity with CI/CD practices using Azure DevOps.
• Strong analytical and issue-resolution skills, with experience supporting business/production issues.
• Good communication skills for documentation (Confluence) and task tracking (JIRA).