Lead Data Platform Engineer

Position Summary:

We are expanding our Platform Engineering capability to build, secure, and automate the enterprise Data Platform on cloud and Databricks. This role owns the underlying infrastructure, ingestion frameworks, CI/CD pipelines, orchestration, and observability that enable Data Engineers and Analytics teams to operate at scale. The ideal candidate combines strong platform engineering fundamentals with hands-on DevOps skills across data replication, job scheduling, deployment automation, and cloud operations.

Key Responsibilities:

Infrastructure & Platform Engineering:

• Deploy and maintain Databricks workspaces and cloud infrastructure using Infrastructure-as-Code.
• Manage platform upgrades, patching, new flow setup, and environment refresh support.
• Support enterprise data replication (HVR) and file-based ingestion patterns from operational systems into the data platform.

Orchestration & Job Scheduling:

• Provide monitoring, recovery, and operational support for enterprise job scheduling and orchestration.
• Configure job dependencies and coordinate with source teams on long-running workloads.

CI/CD & Deployment Automation:

• Design and maintain GitLab CI/CD pipelines for data and platform projects with automated deployment workflows.
• Standardize deployment strategies using reusable templates and Databricks-native deployment tooling.
• Implement branching strategies, code review policies, and environment promotion rules.
• Support the Change Request (CR) deployment lifecycle, including validation and ticket closure.

Monitoring, Reliability & Support:

• Configure monitoring, alerting, and logging to ensure platform stability.
• Serve as an escalation point for platform-related incidents and vendor coordination.
• Support year-end activities and compliance reporting requirements.

What Success Looks Like (First 6-12 Months):

• In your first 6-12 months, you'll stabilize CI/CD and monitoring for key platform flows, automate recurring operational tasks, and streamline the change-request and deployment lifecycle.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
• 6+ years of industry experience in Data Engineering, Cloud Infrastructure, or DevOps.
• Hands-on experience with CI/CD tooling (GitLab preferred) - pipeline authoring, release management, and secrets management.
• Strong grounding in cloud platforms (AWS preferred) for data workloads.
• Working knowledge of Databricks platform administration.
• Experience with monitoring and observability tools, proactive alerting, and incident triage.
• Proficient in Python and Bash/Shell scripting for automation.

Preferred Qualifications:

• Experience with enterprise data replication tools (e.g. HVR).
• Advanced Infrastructure-as-Code skills.
• Familiarity with enterprise job orchestration platforms (e.g. Autopilot).
• Exposure to Databricks Serverless Compute and Workflow orchestration.
• Cloud Solutions Architect or Databricks certifications are a plus.

Competencies:

• Reliability-first mindset - focus on stability, automation, and self-healing systems.
• Strong sense of ownership across the platform lifecycle - build, run, and evolve.
• Effective vendor coordination and cross-team collaboration.
• Clear documentation and knowledge-sharing habits.