Senior Data Engineer

Job Description: Senior Data Engineer – Data Ingestion & Platforms

Role Overview

We are seeking a seasoned Senior Data Engineer with a strong software engineering mindset to design,

build, and optimize our next-generation Ingest Factory and Data Processing Frameworks. In this role,

you will go beyond traditional ETL scripting to build scalable, metadata-driven pipelines and reusable

data frameworks.

The ideal candidate possesses deep expertise in Python, PySpark, and the Databricks Lakehouse

ecosystem (including LakeFlow and Delta Lake), combined with rigorous software engineering

discipline (SOLID, CI/CD, and infrastructure as code). You will work both independently and

collaboratively within an Agile environment to build production-grade software that ensures data quality,

governance, and seamless orchestration.

Key Responsibilities

Architecture & Pipeline Engineering

• Ingest Factory Design: Design, develop, and maintain robust data ingestion frameworks
  
  

leveraging Databricks LakeFlow, managed connectors, and declarative pipelines.

• Data Lakehouse Patterns: Implement repeatable ETL/ELT patterns within a Delta Lake
  
  

architecture, ensuring optimized storage, table design, and strict data lineage enforcement.

• Metadata-Driven Orchestration: Build parameterized notebooks and end-to-end orchestration
  
  

flows to automate ingestion across diverse source system patterns.

Software Craftsmanship & Automation

• Advanced Python Development: Write clean, modular, and maintainable Python code applying
  
  

SOLID and DRY principles. Move beyond basic PySpark scripting to contribute to and publish

reusable internal packages (e.g., PyPI).

• Framework Creation: Define reusable functions and framework-level abstractions to
  
  

dramatically improve development efficiency across the data team.

• Testing & Quality Assurance: Implement rigorous data quality checks, monitoring, and alerting
  
  

frameworks. Lead test practices including Unit, Integration, and End-to-End (E2E) testing.

Optimization & Troubleshooting

• Performance Tuning: Optimize complex distributed Spark workloads, Databricks compute
  
  

configurations, and SQL queries (efficient filtering, indexing, and joins) to reduce processing

costs.

• Advanced Troubleshooting: Deep dive into Spark UI and logs to diagnose and resolve
  
  

performance bottlenecks, data skew, and serialization issues.

DevOps & Collaboration

• CI/CD & IaC: Own the deployment lifecycle by building and maintaining GitHub Actions / GitLab
  
  

pipelines and provision infrastructure utilizing Terraform (IaC).

• Agile Delivery: Actively participate in SCRUM ceremonies, design solutions to specific user
  
  

stories, vet architectures with the team, and deliver retro demos prior to production deployment.

Technical Skillset & Qualifications

Must-Have Core Skills:

• Cloud Data Experience: 5+ years of production experience in cloud data engineering and building
  
  

enterprise-grade software.

• Databricks Ecosystem: Deep hands-on experience with Databricks Notebooks, Jobs
  
  

optimization, Delta Lake, Connectors, and LakeFlow (jobs, tasks, flows).

• Advanced Python & Spark: Mastery of Core Python, Data Structures & Algorithms (DSA), and
  
  

package management. Clear understanding of distributed workloads (Spark vs. single-node

processing).

• Software Engineering Disciplines: 3–5+ years of practicing SOLID coding principles, Git controls
  
  

(PR reviews, branching strategies), and modern IDE features (Cursor/VSCode).

• DevOps & Automation: Production experience with Terraform and CI/CD tools (GitHub Actions
  
  

or GitLab CI).

• Advanced SQL: Proficiency in writing and optimizing mid-to-complex queries, ensuring efficient
  
  

data processing and model design.

Soft Skills & Operational Traits

• Ability to work independently as a self-starter while being a highly collaborative team player.
• Strong business and data literacy—understanding not just how to move data, but the business
  
  

purpose behind it.

• Excellent communication skills for vetting solutions with peer engineers and presenting work
  
  

during retro demos.

Skills: python,data engineer,pyspark,spark,metadata,metadata driven pipeline,deployment,customization,configuration,data packages,data products