Walk-in || Azure Data Engineer - Face To Face in Hyderabad

Role & responsibilities

Job Description:

1. Skills - ADB + Pyspark + Kafka

Grade - (G5-7+ years, G6- 9+ years )

Design, develop and optimise scalable ETL/ELT pipelines using Databricks.

• Develop production-grade data transformations using Python, PySpark and SQL.
• Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
• Implement batch and, where required, streaming ingestion patterns.
• Work with Databricks Workflows and Unity Catalog for orchestration and governance.
• Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
• Implement data-quality checks, error handling, monitoring and performance optimisation.
• Develop automated deployments using CI/CD and follow software engineering best practices.
• Troubleshoot production data pipelines and ensure reliability and scalability.

Mandatory skills:

• Strong hands-on Databricks, Python/PySpark and SQL.
• Delta Lake and Databricks Workflows.
• ETL/ELT development and performance tuning.
• ADLS Gen2 and Azure data ecosystem.
• Unity Catalog exposure.
• Git and CI/CD experience.
• The existing Allianz-aligned platform JD similarly calls for strong Databricks/PySpark, Delta Lake, Unity Catalog and Synapse capabilities, making these good baseline screening criteria

2. JD :

Skills Azure Data Factory + Testing

Exp 9+ years

Job Description: Data Test Lead

Key Responsibilities

• Define and lead the end-to-end data testing strategy for large-scale EDW migration programmes from legacy ETL platforms to cloud-native architectures.
• Drive source-to-target reconciliation, parallel testing, and transformation validation to ensure Azure Data Factory and dbt outputs align with existing Ab Initio processing logic and business rules.
• Design, develop, and automate robust data validation frameworks using:
• dbt testing capabilities
• SQL-based reconciliation techniques
• ADF validation activities
• Custom validation and audit frameworks
• Validate complex transformation logic, aggregations, business rules, and dimensional models across staging, curated, and reporting layers.
• Lead and mentor distributed QA and testing teams across onsite and offshore delivery models.
• Coordinate SIT, UAT, regression, reconciliation, and production validation activities.
• Track test execution progress, defect management, data quality KPIs, and testing metrics for stakeholder and programme reporting.
• Build reusable frameworks for:
• Continuous data quality monitoring
• Automated reconciliation
• Data lineage validation
• Source-to-target validation
• Regression testing
• Collaborate closely with Data Architects, ETL Developers, Business Analysts, and Platform Teams to ensure successful migration delivery.
• Support root cause analysis and defect triage activities for data discrepancies and pipeline failures.
• Ensure adherence to enterprise governance, audit, compliance, and data quality standards.

Required Skills & Experience

• 10+ years of experience in Data Warehouse and ETL Testing, including leadership responsibilities.
• Strong understanding of ETL/ELT concepts and legacy-to-cloud migration testing methodologies.
• Extensive experience in:
• Data validation
• Transformation testing
• Reconciliation testing
• Source-to-target testing
• Data quality assurance
• Strong knowledge of Ab Initio components and validation of graphs, mappings, and processing workflows.
• Experience testing Azure Data Factory pipelines, orchestration flows, and metadata-driven frameworks.
• Hands-on expertise in dbt including:
• dbt tests
• snapshots
• SQL-based transformation validation
• model validation
• lineage verification
• Advanced SQL expertise with modern cloud databases such as Snowflake.
• Strong understanding of dimensional modelling and enterprise data warehousing concepts.
• Experience with defect management and test management tools such as JIRA, Azure DevOps, or equivalent.
• Strong stakeholder management, communication, and leadership capabilities.

3) JD :

Skills Azure Databricks+ Pyspark

Exp 6+ years

• Experience and Qualification:68 years of experience in data engineering, ETL development, or big data processing.
• Strong hands-on experience with Azure Databricks. Proficiency in Apache Spark and PySpark.
• Strong programming skills in Python and SQL. Experience with Azure Data Lake Storage Gen2.
• Experience developing pipelines using Azure Data Factory.
• Practical knowledge of Delta Lake and lakehouse architecture.
• Experience working with relational and NoSQL databases.
• Strong understanding of data modeling, ETL/ELT concepts, and data warehousing.
• Experience with Git-based source control and CI/CD practices.
• Knowledge of Azure services such as: Azure Key Vault Azure Synapse Analytics Azure SQL Database Azure Event Hubs Azure Monitor Experience with data security, access control, and governance in Azure environments.

Preferred candidate profile