Data & AI Platform Engineer II

The Commercial Data & AI HYD team is seeking a Data & AI Platform Engineer II to help build and operate the Databricks and Unity Catalog platform infrastructure underpinning the team's data products. This role is central to ensuring the platform remains reliable, secure, and ready to support new Commercial AI initiatives.

The successful candidate will manage cloud engineering tasks for the platform, coordinate with Enterprise IT on systems integration and feature enablement, and help ensure the team's platform configuration remains aligned with enterprise architecture standards. This includes supporting platform reliability and security through access controls, monitoring, and configuration management.

This role offers the opportunity to build deep platform engineering expertise while working closely with both the Enterprise IT and Commercial Data & AI teams, with room to grow into more senior platform and architecture responsibilities over time.

Key Responsibilities:

• Build, configure, and operate Databricks and Unity Catalog platform components supporting the team's data products, including workspace, cluster, and access configuration.
• Manage cloud engineering tasks for the platform, including provisioning, cost management, and performance tuning of the underlying cloud infrastructure.
• Coordinate with Enterprise IT on systems integration and feature enablement, supporting rollout of new platform capabilities to the team.
• Partner with Enterprise IT and the team's data architects on platform architecture standards, ensuring configuration stays aligned with enterprise-wide direction.
• Support platform reliability and security by implementing access controls, monitoring, and configuration management for the Databricks environment.
• Troubleshoot platform-level issues affecting pipeline execution, working with data engineers to isolate platform versus pipeline-specific issues.
• Validate that newly migrated infrastructure meets performance and reliability requirements before cutover to production use.
• Document platform configuration, access patterns, and operational runbooks to support consistent operations across the team.

Skills & Competencies:

• Working proficiency in SQL and Python for platform automation and configuration scripting.
• Hands-on experience with Databricks platform administration (workspace configuration, clusters, jobs, Unity Catalog) and Delta Lake concepts.
• Experience with cloud infrastructure fundamentals (e.g., AWS storage, compute, networking, and IAM concepts) relevant to a Databricks deployment.
• Understanding of access control, security, and compliance considerations for a shared data platform handling sensitive commercial pharma data.
• Familiarity with infrastructure-as-code or configuration management approaches for reproducible platform setup.
• Working knowledge of software engineering practices including Git/version control, code reviews, and CI/CD concepts.
• Good troubleshooting and cross-team communication skills, with the ability to work effectively with both data engineers and Enterprise IT.
• Understanding of data governance and secure data handling practices, including access controls and protection of sensitive data.

Qualifications & Experience:

• A bachelor's or master's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent practical experience).
• 3-5 years of hands-on experience in platform engineering, cloud infrastructure, DevOps, or data platform administration.
• Experience administering or operating a Databricks environment, including Unity Catalog, in a production setting.
• Experience with cloud platforms (AWS preferred) supporting data engineering workloads.
• Preferred: experience coordinating with a central Enterprise IT or platform team on integration and architecture standards.
• Preferred: experience working in a regulated or pharma commercial data environment.

We hire for skills and capabilities, not just credentials - if this role excites you, but doesn't perfectly match your resume, we encourage you to apply anyway.