Senior Data Engineer - Databricks

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer - Databricks based in United States.

This role offers the opportunity to build and operate the data foundation behind an AI-powered revenue intelligence platform. You will own critical Databricks pipelines that transform complex ERP and CRM data into reliable, actionable insights. Working at the intersection of data engineering, cloud platforms, and AI innovation, you will help scale enterprise-grade solutions. You will drive improvements in pipeline reliability, performance, security, and cost efficiency across global environments. Partnering with engineering, product, architecture, and data teams, you will influence how data products evolve and scale. This is a high-impact position for an experienced data engineer who enjoys solving complex challenges and building modern data platforms.

Accountabilities:: In this role, you will own the design, development, optimization, and reliability of large-scale data pipelines that power advanced analytics and AI-driven applications. You will ensure data platforms remain secure, scalable, and efficient while collaborating with cross-functional teams to deliver high-quality solutions.

Own production support for Databricks-based data platforms, including monitoring, alerting, incident response, and SLA management across critical data workflows.

Design and develop high-performance data pipelines using Databricks, PySpark, SQL, and Python to ingest, transform, and deliver ERP and CRM data at scale.

Optimize Databricks workloads by improving processing performance, reducing compute costs, and enhancing pipeline efficiency through measurable improvements.

Lead modernization efforts by migrating legacy ETL/ELT workflows into scalable Databricks-based architectures with automated deployment and operational processes.

Build and maintain Delta Lake architectures, including schema design, partitioning strategies, incremental processing, and data quality enforcement.

Support customer onboarding by provisioning, validating, and securing tenant-specific data pipelines while ensuring reliable data delivery.

Implement strong data governance practices, including access controls, security standards, compliance requirements, and privacy protections for enterprise data.

Establish data observability practices to monitor pipeline health, ensure data integrity, and support accurate machine learning and predictive analytics outcomes.

Partner with engineering, product, architecture, and infrastructure teams to ensure seamless integration across the broader technology ecosystem.

Support globally distributed operations through incident management, on-call rotations, technical documentation, runbooks, and continuous improvement initiatives.

Apply CI/CD best practices for data engineering workflows, including automated testing, version control, and reliable deployment processes.

Contribute to technical strategy by sharing knowledge, improving engineering standards, and helping evolve platform capabilities.

Requirements:

The ideal candidate is an experienced data engineer with strong expertise in modern cloud data platforms, Databricks ecosystems, and production-scale pipeline development. You bring a combination of technical depth, problem-solving ability, and collaboration skills to build reliable data solutions in a fast-moving environment.

4+ years of experience in data engineering roles, building and maintaining production-grade data pipelines.

At least 2 years of hands-on experience with Databricks and the Apache Spark ecosystem across Azure and/or AWS environments.

Strong proficiency in PySpark, SQL, and Python, with experience delivering scalable data solutions under SLA requirements.

Hands-on experience with Delta Lake, including schema evolution, ACID transactions, optimization strategies, and incremental or streaming processing patterns.

Experience tuning Databricks workloads, managing compute resources, and improving production pipeline performance.

Strong knowledge of PostgreSQL, including query optimization, schema design, and production data integration.

Experience supporting legacy ETL technologies such as SSIS, Informatica, or custom SQL/Python-based workflows.

Experience working with multi-tenant architectures, including tenant isolation, performance optimization, and enterprise data privacy requirements.

Solid understanding of data governance, security, compliance, access management, and protection of sensitive business information.

Ability to collaborate effectively with data scientists, product teams, infrastructure engineers, and enterprise architecture stakeholders.

Experience with cloud-based data platforms, automation, and infrastructure practices is highly valuable.

Familiarity with ML feature engineering, feature stores, or predictive analytics workflows is a plus.

Experience with Databricks Serverless optimization, cross-cloud data access, customer onboarding automation, or Infrastructure-as-Code patterns is preferred.

Databricks Data Engineer certification or equivalent industry certification is a plus.

Strong ownership mindset, adaptability, curiosity, and willingness to learn new technologies in a collaborative environment.

Benefits:

Comprehensive healthcare coverage for employees and families.

401(k) savings plan with employer matching contributions.

Unlimited paid time off.

Paid parental leave.

Online legal services support.

Financial planning resources.

Discounted pet insurance options.

Corporate benefit program offering travel, entertainment, and lifestyle discounts.

Health and wellness reimbursement program.

Travel discount opportunities.

Access to educational resources and career development programs.

Employee referral bonus program.

Opportunities for professional growth, learning, and career advancement within a merit-based environment.

Flexible remote work opportunities.

How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best!  Why Apply Through Jobgether?    Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.     #LI-CL1