Senior Data Engineer

We are looking for a hands-on Data Engineer with strong expertise in SQL, PySpark, Apache Spark, and Databricks, along with awareness of GenAI and AI-assisted coding tools. The candidate should be capable of building scalable data pipelines, troubleshooting performance issues, and optimizing data processing for both performance and cost.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using SQL, PySpark, Spark, and Databricks.
• Build reliable batch/incremental data-processing solutions and data products.
• Troubleshoot and optimize Spark workloads for performance, scalability, and cost efficiency.
• Work with Delta Lake and modern data-platform architecture.
• Use AI coding agents to accelerate development, debugging, testing, and documentation.
• Collaborate with application teams, consultants, and managers.

Requirements:

• Strong problem-solving and analytical skills.
• Hands-on, engineering-oriented mindset.
• Ability to understand why a technical approach is selected, not just implement it.
• Clear and structured communication.
• Comfortable working independently and learning new technologies.
• Expert SQL and PySpark development.
• Strong understanding of Apache Spark internals: distributed processing, partitioning, shuffle, data skew, and execution model.
• Strong Databricks experience with Delta Lake, optimization, compute management, and cost optimization.
• Strong data-engineering fundamentals: ETL/ELT, data modelling, incremental processing, and data quality.
• Experience troubleshooting and optimizing large-scale data workloads.

GenAI and AI-Assisted Development:

• LLMs, RAG, embeddings, agents, and agentic AI.
• AI-assisted coding tools such as Claude Code, Codex, Cursor, Antigravity, or similar.
• Ability to review, validate, and improve AI-generated code.

Good to Have:

• DevOps knowledge: Git, CI/CD, Docker, and deployment practices.
• Exposure to NoSQL databases and their use cases.
• Data Engineer/Analytics Certificate in any of the clouds (AWS, Azure, or GCP).