Sr. Azure Data Architect - Databricks

Job Summary

Overview: Job Title: Sr. Data Architect - Azure Databricks

Location: Remote Hybrid

Key Responsibilities

• Design and implement end-to-end data architecture / data ecosystem using Azure-native services and Databricks
• Build scalable, high-performance data platforms and robust data frameworks
• Partner with stakeholders to translate business requirements into reliable data solutions
• Implement data governance principles using Unity Catalog
• Create and implement data ingestion and data integration patterns using Medallion Architecture
• Apply best practices in data engineering
• Develop PySpark notebooks
• Create procedures for advanced data transformation
• Handle performance tuning in Databricks and Azure SQL DBs
• Architect end-to-end data solutions supporting: Yield improvement, Process control, Predictive maintenance, Operational efficiency; Ensure strict governance, security, scalability, and compliance across the platform

Required Experience

• 10+ years of overall experience and 5+ years of architecture experience in data architecture / data engineering roles
• Strong hands-on experience with major enterprise data platforms
• Extensive experience with Azure-native services including:
  • Azure Data Lake Storage (ADLS)
  • Azure Data Factory (ADF)
  • Azure Synapse Analytics
  • Azure SQL
  • Strong hands-on experience with Azure Databricks
• Required Databricks Expertise
  • Should have worked with the latest Databricks capabilities including:
    • Unity Catalog:
      • Catalog and schema design
      • Permissions and security
      • Managed and external tables
      • Lineage and governance
    • Delta Live Tables (DLT):
      • Reliable pipeline development
      • CDC implementation
      • Data transformations
      • Data quality controls
      • Scaling and performance tuning
      • Other modern Databricks capabilities and best practices
    • Data Ingestion & ELT Experience
      • Experience with data ingestion tools such as:
        • Fivetran
      • HVR or equivalent CDC / replication tools
      • Experience with ELT tools such as:
        • DBT
      • Familiarity with:
        • Real-time / streaming data
        • Batch processing
        • Sensor / OT / ET data flows

Technical Skills (Requirements)

Technical Skills

• Strong expertise in data ingestion, transformation, and orchestration frameworks
• Hands-on experience building insurance-specific data models
• Proficiency in Python / PySpark and Spark SQL
• Strong understanding of streaming vs batch processing
• Experience with Spark job performance tuning
• Ability to optimize:
  • Data storage formats
  • Partitioning strategies
  • Schema design
• Experience working with:
  • Object stores
  • Data lakes
  • Cloud compute scaling
  • Databricks cluster management

Governance, Security & Operations

• Deep understanding of:
  • Data governance
  • Security and compliance
  • RBAC and attribute-based access control
  • Encryption and audit logs
  • Data privacy requirements
• Experience with operational excellence practices:
  • Monitoring and observability
  • Reliability and metrics
  • Backup, failover, and disaster recovery strategies

Preferred Qualifications

• Prior experience working on Insurance P&C cloud-based data and analytics projects
• Strong analytical and problem-solving skills
• Experience defining and enforcing best practices for Databricks workloads and Spark-based pipelines
• Solid understanding of distributed data processing concepts
• Strong communication and stakeholder management skills
• Ability to document architecture decisions, trade-offs, and technical recommendations

Job Type

Any

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

Disclaimer: The job location mentioned in this description is based on publicly available information or company headquarters. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.