Senior Data Engineer | Azure Databricks & Power BI
Visionet Systems · Bengaluru · onsite · Posted 62d ago
About a minute. You'll make an account on the way through.
About this role
Job Description - Senior Data Engineer | Azure Databricks Power BI
Position
Senior Data Engineer - Azure Databricks Power BI
Experience: 8-10 Years
Role Summary
We are looking for an experienced Senior Data Engineer with strong Azure Databricks and Power BI expertise to design, develop, migrate, and modernize enterprise data and analytics solutions.
The candidate will be responsible for end-to-end data engineering, including data ingestion, transformation, data modelling, migration, performance optimization, data quality, and integration with BI platforms.
The role requires hands-on experience in Azure Databricks, PySpark, SQL, Azure Data Lake, and Power BI, along with practical experience in legacy data-platform and BI migration initiatives.
Experience migrating Informatica workloads to Azure Databricks and IBM Cognos reports/dashboards to Power BI will be highly preferred.
Key Responsibilities
1. Data Engineering Databricks
- Design and develop scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and SQL.
- Develop robust ETL/ELT pipelines for batch and incremental data processing.
- Implement Bronze, Silver, and Gold / Medallion architecture patterns.
- Build and manage Delta Lake tables and optimize Databricks workloads.
- Implement data transformation, cleansing, validation, aggregation, and enrichment processes.
- Apply Spark performance optimization techniques including partitioning, caching, broadcast joins, and optimization of data layouts.
- Develop reusable frameworks, notebooks, libraries, and data engineering components.
- Implement error handling, logging, monitoring, restartability, and operational controls.
2. Azure Data Platform
- Work with Azure services such as:
- Azure Databricks
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL Database
- Azure Key Vault
- Azure DevOps
- Design secure and scalable data ingestion and integration patterns.
- Implement appropriate authentication, authorization, secrets management, and access controls.
- Support CI/CD implementation for Databricks notebooks, code, pipelines, and deployment artifacts.
3. Data Migration Modernization
- Participate in legacy-to-cloud data platform modernization and migration programs.
- Analyze existing ETL workflows, data models, dependencies, business rules, and transformation logic.
- Convert legacy ETL processes into scalable Databricks/PySpark solutions.
- Perform source-to-target mapping and transformation-rule analysis.
- Support data reconciliation, validation, parallel runs, defect resolution, and migration sign-off.
- Develop migration utilities and reusable patterns to accelerate large-scale migration.
- Support migration planning, assessment, estimation, cutover, rollback, and hypercare activities.
4. Informatica Azure Databricks Migration - Good to Have
Hands-on experience migrating Informatica PowerCenter / Informatica ETL workloads to Azure Databricks is highly desirable.
The candidate should ideally understand:
- Informatica mappings, workflows, sessions, transformations, and dependencies.
- Conversion of Informatica mappings into PySpark / Spark SQL.
- Mapping complex transformation logic to Databricks.
- Informatica parameterization and workflow dependencies.
- Incremental load and CDC patterns.
- Source-to-target mapping and data lineage.
- ETL reconciliation and data validation.
- Performance comparison between legacy and modernized pipelines.
- Migration factory approaches for large-scale ETL modernization.
5. Power BI BI Modernization
- Develop enterprise-grade Power BI dashboards and reports.
- Build semantic models and appropriate dimensional models for reporting.
- Develop optimized DAX measures and calculated columns.
- Implement Power BI data security, including Row-Level Security (RLS).
- Optimize report, semantic model, and query performance.
- Work with business stakeholders to translate reporting requirements into effective analytical solutions.
- Support migration of legacy BI reports to Power BI.
6. Cognos Power BI Migration - Good to Have
Experience migrating IBM Cognos Analytics reports to Power BI is highly preferred.
Exposure should include:
- Cognos reports, packages, Framework Manager models, and reporting logic.
- Assessment and inventory of existing Cognos reports.
- Mapping Cognos reports to Power BI equivalents.
- Conversion of prompts, filters, calculations, drill-throughs, and reporting logic.
- Migration of Cognos semantic/reporting models to Power BI semantic models.
- DAX implementation for Cognos calculations.
- Report rationalization and identification of redundant/obsolete reports.
- Reconciliation of Cognos and Power BI outputs.
- Business-user validation and migration sign-off.
Required Technical Skills
Skill Area
Required Expertise
Azure Databricks
Strong hands-on experience
PySpark
Strong hands-on experience
Spark SQL
Strong
SQL
Advanced
Delta Lake
Strong
Azure Data Lake Gen2
Strong
Azure Data Factory
Good
Power BI
Strong hands-on experience
DAX
Strong
Data Modelling
Strong
ETL/ELT
Strong
Data Migration
Strong
Azure DevOps / Git
Good
CI/CD
Good
Python
Good
Data Quality Reconciliation
Strong
Good-to-Have Skills
- Informatica PowerCenter
- Informatica Databricks migration
- IBM Cognos Analytics
- Cognos Power BI migration
- Legacy ETL modernization
- Legacy BI modernization
- Azure Synapse Analytics
- Microsoft Fabric
- CDC and incremental ingestion
- Unity Catalog
- Databricks Workflows
- Azure Functions / Logic Apps
- Data governance and lineage tools
Role Summary
We are looking for an experienced Senior Data Engineer with strong Azure Databricks and Power BI expertise to design, develop, migrate, and modernize enterprise data and analytics solutions.
The candidate will be responsible for end-to-end data engineering, including data ingestion, transformation, data modelling, migration, performance optimization, data quality, and integration with BI platforms.
The role requires hands-on experience in Azure Databricks, PySpark, SQL, Azure Data Lake, and Power BI, along with practical experience in legacy data-platform and BI migration initiatives.
Experience migrating Informatica workloads to Azure Databricks and IBM Cognos reports/dashboards to Power BI will be highly preferred.
Key Responsibilities
1. Data Engineering Databricks
- Design and develop scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and SQL.
- Develop robust ETL/ELT pipelines for batch and incremental data processing.
- Implement Bronze, Silver, and Gold / Medallion architecture patterns.
- Build and manage Delta Lake tables and optimize Databricks workloads.
- Implement data transformation, cleansing, validation, aggregation, and enrichment processes.
- Apply Spark performance optimization techniques including partitioning, caching, broadcast joins, and optimization of data layouts.
- Develop reusable frameworks, notebooks, libraries, and data engineering components.
- Implement error handling, logging, monitoring, restartability, and operational controls.
2. Azure Data Platform
- Work with Azure services such as:
- Azure Databricks
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL Database
- Azure Key Vault
- Azure DevOps
- Design secure and scalable data ingestion and integration patterns.
- Implement appropriate authentication, authorization, secrets management, and access controls.
- Support CI/CD implementation for Databricks notebooks, code, pipelines, and deployment artifacts.
3. Data Migration Modernization
- Participate in legacy-to-cloud data platform modernization and migration programs.
- Analyze existing ETL workflows, data models, dependencies, business rules, and transformation logic.
- Convert legacy ETL processes into scalable Databricks/PySpark solutions.
- Perform source-to-target mapping and transformation-rule analysis.
- Support data reconciliation, validation, parallel runs, defect resolution, and migration sign-off.
- Develop migration utilities and reusable patterns to accelerate large-scale migration.
- Support migration planning, assessment, estimation, cutover, rollback, and hypercare activities.
4. Informatica Azure Databricks Migration - Good to Have
Hands-on experience migrating Informatica PowerCenter / Informatica ETL workloads to Azure Databricks is highly desirable.
The candidate should ideally understand:
- Informatica mappings, workflows, sessions, transformations, and dependencies.
- Conversion of Informatica mappings into PySpark / Spark SQL.
- Mapping complex transformation logic to Databricks.
- Informatica parameterization and workflow dependencies.
- Incremental load and CDC patterns.
- Source-to-target mapping and data lineage.
- ETL reconciliation and data validation.
- Performance comparison between legacy and modernized pipelines.
- Migration factory approaches for large-scale ETL modernization.
5. Power BI BI Modernization
- Develop enterprise-grade Power BI dashboards and reports.
- Build semantic models and appropriate dimensional models for reporting.
- Develop optimized DAX measures and calculated columns.
- Implement Power BI data security, including Row-Level Security (RLS).
- Optimize report, semantic model, and query performance.
- Work with business stakeholders to translate reporting requirements into effective analytical solutions.
- Support migration of legacy BI reports to Power BI.
6. Cognos Power BI Migration - Good to Have
Experience migrating IBM Cognos Analytics reports to Power BI is highly preferred.
Exposure should include:
- Cognos reports, packages, Framework Manager models, and reporting logic.
- Assessment and inventory of existing Cognos reports.
- Mapping Cognos reports to Power BI equivalents.
- Conversion of prompts, filters, calculations, drill-throughs, and reporting logic.
- Migration of Cognos semantic/reporting models to Power BI semantic models.
- DAX implementation for Cognos calculations.
- Report rationalization and identification of redundant/obsolete reports.
- Reconciliation of Cognos and Power BI outputs.
- Business-user validation and migration sign-off.
Good-to-Have Skills
- Informatica PowerCenter
- Informatica Databricks migration
- IBM Cognos Analytics
- Cognos Power BI migration
- Legacy ETL modernization
- Legacy BI modernization
- Azure Synapse Analytics
- Microsoft Fabric
- CDC and incremental ingestion
- Unity Catalog
- Databricks Workflows
- Azure Functions / Logic Apps
- Data governance and lineage tools
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Skills
About Visionet Systems
Visionet Systems
IT Services & Consulting
2.2k+
employees
1995
31 years old
Bangalore / Bengaluru, Karnataka
India
₹5.2L PA avg
Avg at Visionet Systems
About
Industries
Employee ratings
10 reviews
Culture
3.5
Career growth
3.1
Work-life
3.5
Employees rate it well for
Find them on