Senior Data Engineer (Azure Databricks)

Spectral Consultants is Hiring: Senior Data Engineer – Azure Databricks for a leading Accounting & Advisory Firm.

Location: Bangalore

Experience: 5–8 Years

We are looking for a Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, SQL and Delta Lake to build scalable, governed data pipelines and analytics/AI-ready datasets.

Key Responsibilities

• Build scalable ETL/ELT pipelines on Azure Databricks using Medallion Architecture

• Develop Bronze → Silver → Gold → Platinum Delta Lake data models

• Work with PySpark, Spark SQL, Structured Streaming and Auto Loader

• Implement Unity Catalog governance, access controls, lineage and data classification

• Integrate data from ADLS Gen2, AWS, GCP, Snowflake, Redshift, BigQuery and other sources

• Build data-quality, audit, reconciliation and operational-metadata frameworks

• Implement Databricks Workflows/Jobs, DLT/Lakeflow, Git and Databricks Asset Bundles

• Optimize Delta workloads using partitioning, OPTIMIZE, Z-Ordering/Liquid Clustering

• Develop datasets supporting AI/ML, analytics, Power BI and Tableau use cases

Must-Have Skills

Azure Databricks

PySpark / Spark / Advanced SQL

Delta Lake & Medallion Architecture

Unity Catalog

ADLS Gen2

Structured Streaming / Auto Loader

Databricks Workflows / DLT

Git / CI-CD

Data Quality & Governance

Good to Have

• MLflow / Feature Store / Mosaic AI / Vector Search

• Power BI / Tableau data modelling

• Financial