Architect

Job Summary Architect role requiring 10 to 14 years of experience in designing and governing Databricks based data platforms using Unity Catalog Python Databricks SQL Delta Lake Workflows and PySpark with a strong focus on Property and Casualty Insurance analytics operating in a hybrid work model during day shifts with no travel requirements while ensuring secure and scalable data solutions.

Responsibilities

• Design and implement scalable Databricks platforms that support complex analytics and reporting needs for Property and Casualty Insurance portfolios to improve underwriting and claims insights
• Configure and administer Databricks Unity Catalog to provide consistent data governance access control and lineage across multiple workspaces and hybrid environments
• Develop robust data ingestion and transformation pipelines using PySpark to integrate policy claims billing and external data sources into Delta Lake with high reliability
• Author optimized Databricks SQL queries and data models that support actuarial analysis risk scoring pricing optimization and operational reporting for insurance stakeholders
• Create and manage Databricks Workflows to orchestrate end to end batch and near real time data pipelines ensuring predictable execution and timely data availability
• Build reusable Python utilities and frameworks for data validation monitoring error handling and performance tuning across the Databricks platform to enhance maintainability
• Establish and document standards for Delta Lake design including table formats partitioning strategies schema evolution and data quality rules tailored to insurance data domains
• Collaborate with cross functional teams including data engineers data scientists actuaries and product partners to translate insurance business requirements into technical Databricks solutions
• Implement security and compliance controls in Databricks using Unity Catalog features to protect sensitive insurance data such as policyholder information and regulatory datasets
• Perform performance optimization and cost management activities including cluster configuration job tuning and workload segregation to achieve efficient use of compute resources
• Guide peers on best practices for Databricks CLI usage to automate deployments configuration management and environment setup in controlled and auditable ways
• Create clear architecture diagrams runbooks and technical documentation to support operations support teams and future enhancements of the Databricks platform
• Drive continuous improvement initiatives by analyzing platform metrics incident trends and user feedback to enhance reliability usability and business value of the data platform

Qualifications

• Require extensive experience in Databricks Unity Catalog administration including access policies catalog structures and governance practices for enterprise environments
• Require hands on proficiency with Databricks CLI Python and PySpark for building automated deployment scripts data pipelines and platform management utilities
• Require strong experience with Databricks SQL and Delta Lake for designing high performance data models and queries that serve insurance analytics and reporting needs
• Require proven background in Property and Casualty Insurance data domains covering policy claims exposure rating and regulatory reporting to ensure relevant solution design
• Require experience operating in hybrid work environments and collaborating across distributed teams using modern communication and project tracking tools
• Nice to have familiarity with cloud native services related to storage networking and identity that integrate with Databricks platforms in large multinational organizations
• Nice to have exposure to data quality frameworks observability tools and metadata management solutions that complement Unity Catalog and enhance platform trust

Certifications Required Good to have Databricks Certified Data Engineer Professional or equivalent Databricks Architect level certification is preferred