Databricks Engineer

Senior Databricks Engineer

We are seeking a Senior Databricks Engineer with 10 years of data engineering experience to support a large-scale analytics modernization and refactoring initiative. This role requires a highly independent engineer who can operate effectively with high-level direction, decompose ambiguous objectives into actionable work, and drive delivery with limited day-to-day oversight.

The primary focus of this role will be to build and mature Silver and Gold data layers in Databricks, leveraging existing Bronze-layer data assets while improving data quality, lineage, governance, and trust. The engineer will work closely with Tableau developers and business stakeholders to reverse engineer existing dashboards, trace reporting logic back to source systems, document current-state dependencies, and design curated data products that provide the business with a high degree of confidence in the underlying data.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and curated datasets within the Databricks platform.
• Build upon existing Bronze-layer data assets to create well-defined, reusable, and governed Silver and Gold layers.
• Independently translate high-level business and technical objectives into detailed work plans, technical tasks, dependencies, and deliverables.
• Partner closely with Tableau developers to reverse engineer existing dashboards, metrics, calculations, filters, and data dependencies.
• Trace dashboard outputs back through semantic logic, SQL, views, source tables, and upstream systems to determine how existing reporting solutions are constructed.
• Document current-state data flows, source-system mappings, transformations, business rules, metric definitions, dependencies, and lineage.
• Design Silver-layer datasets that standardize, cleanse, conform, and enrich Bronze data for downstream consumption.
• Design Gold-layer data products optimized for analytics, reporting, and Tableau consumption.
• Define and enforce data quality rules, reconciliation checks, validation processes, and observability controls.
• Implement and maintain data lineage and governance practices so that business users can understand where data originates, how it is transformed, and how key metrics are derived.
• Work with business and technical stakeholders to reconcile discrepancies between legacy dashboard logic and newly curated data products.
• Refactor complex or redundant transformation logic into maintainable, reusable, and scalable data engineering patterns.
• Ensure that curated datasets support consistent metric definitions and reduce duplicated business logic across reporting solutions.
• Optimize Databricks workloads for performance, reliability, scalability, and cost efficiency.
• Proactively identify data quality issues, technical debt, lineage gaps, and architectural risks and drive them toward resolution.
• Develop and maintain technical documentation for pipelines, transformations, data models, lineage, and governance controls.
• Support testing and validation of refactored Tableau dashboards against newly developed Silver and Gold datasets.
• Collaborate across data engineering, analytics, BI, architecture, and business teams throughout the modernization effort.

Required Qualifications

• 10 years of professional data engineering experience, including significant hands-on experience with modern cloud data platforms.
• Strong hands-on experience with Databricks and the Lakehouse architecture.
• Advanced proficiency with SQL and strong experience developing complex data transformations.
• Strong experience with Python and/or PySpark for data engineering workloads.
• Experience designing and implementing Bronze, Silver, and Gold data architectures using Medallion architecture principles.
• Strong understanding of relational and dimensional data modeling concepts.
• Experience developing curated datasets for business intelligence and analytical reporting use cases.
• Demonstrated experience tracing reporting outputs back through multiple layers of transformations and source systems.
• Strong understanding of data lineage, metadata management, data governance, and data quality practices.
• Experience with Delta Lake, including tables, optimization strategies, schema evolution, and incremental processing.
• Strong experience with ETL/ELT development, orchestration, testing, monitoring, and production support.
• Demonstrated ability to work independently with high-level direction rather than detailed task-level instructions.
• Ability to break down ambiguous business or technical problems into a structured delivery plan and independently execute against that plan.
• Strong analytical and problem-solving skills, particularly when working with undocumented or poorly documented legacy data solutions.
• Strong technical documentation skills.
• Strong communication skills and the ability to collaborate effectively with Tableau developers, data architects, analysts, engineers, and business stakeholders.

Preferred Qualifications

• Experience supporting large-scale data modernization, analytics transformation, or BI refactoring programs.
• Experience working directly with Tableau developers or other BI teams to create certified analytics-ready datasets.
• Familiarity with Tableau calculations, extracts, custom SQL, data source relationships, and semantic-layer concepts.
• Experience reverse engineering legacy dashboards and reports to identify source data, transformation logic, and metric definitions.
• Experience with Unity Catalog for governance, access control, lineage, and data discovery.
• Experience implementing data quality frameworks and automated reconciliation processes.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Experience with orchestration technologies such as Databricks Workflows, Airflow, Azure Data Factory, or similar tools.
• Experience with CI/CD, source control, automated testing, and infrastructure-as-code practices.
• Experience working in consulting or other client-facing delivery environments.
• Familiarity with enterprise data governance, master data, metadata management, and data stewardship practices.

Ideal Candidate

The ideal candidate is a self-directed senior data engineer who is comfortable entering a complex existing environment, determining how the current reporting ecosystem works, and turning that understanding into a cleaner and more governed data architecture.

This person should be naturally investigative and comfortable working through ambiguity. They should be equally capable of analyzing existing Tableau dashboards with BI developers, tracing metrics and data back to their sources, documenting what they discover, and then designing trusted Silver and Gold data products that simplify downstream reporting.

Success in this role requires more than strong Databricks development skills. The engineer must be able to establish clear lineage from source to report, create reliable and reusable data products, and provide the business with confidence that analytics and reporting are based on consistent, well-governed, and explainable data.