Sr Data Engineer

What You''ll Do

• Own and evolve our Databricks Lakehouse platform Unity Catalog governance, Delta Lake table management, and Lakeflow Declarative Pipelines.
• Design and build reliable, scalable data pipelines using Apache Spark and dbt for batch and near-real-time processing.
• Stream and ingest data at scale via Kafka; orchestrate workflows with Airflow.
• Partner with the integrations team to onboard new data sources and support AI/ML and analytics use cases.
• Own data quality and observability end-to-end using Datadog; proactively identify and resolve discrepancies across data layers.
• Manage infrastructure as code using Terraform; ensure platform reliability and cost efficiency on AWS.
• Build and maintain MongoDB and PostgreSQL data stores; define schema standards and access patterns.
• Deliver data products surfaced in Tableau; collaborate with analytics and product teams on reporting requirements.
• Mentor engineers, lead code reviews, and drive engineering best practices across the team.
• Participate in hiring and onboarding of new team members. Required
• Deep hands-on expertise with Databricks Unity Catalog, Delta Lake, and Lakeflow Declarative Pipelines.
• Strong Apache Spark experience tuning, optimization, and production operations.
• Proficiency in Scala (required) and Python for data pipeline development.
• Experience with Kafka for real-time data ingestion and streaming architectures.
• Solid dbt skills for SQL-based transformation and data modeling.
• Familiarity with Airflow for workflow orchestration.
• Infrastructure experience with Terraform and AWS (S3, IAM, networking fundamentals).
• Experience with MongoDB and PostgreSQL for operational and analytical workloads.
• Strong communication skills able to translate complex technical concepts for non-technical audiences.

Keywords:

Data Platform: Databricks (Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines), Apache Spark

Databases: MongoDB, PostgreSQL

Cloud: AWS (S3, EMR, Lambda, Glue, IAM, VPC)

Streaming & Messaging: Apache Kafka

Orchestration: Apache Airflow