Senior Data Engineer

Senior Data Engineer Role: Senior Data Engineer
Role Overview
We are seeking a Senior Data Engineer to drive AI-led transformation across our analytics and data platforms. This role is designed for a hands-on engineer who can ideate, design, and implement AI-driven solutions, modernize Python and data pipelines, and maintain best-in-class analytics environments.
The role combines agentic AI concepts, cloud-based AI services, strong Python/PySpark engineering, and deep analytics expertise across Databricks and Snowflake, while applying solid ETL and DBA fundamentals in AWS.

Skill Priority Core Responsibilities

1 AI - Agentic Systems, GenAI Intelligent Workflow Automation
Lead AI ideation by identifying opportunities to eliminate manual effort, reduce operational friction, and improve decision-making.
Design and implement agentic AI solutions, including:
Multi-agent orchestration for task delegation and workflow execution
AI agents for monitoring, diagnostics, reconciliation, and data reasoning
Build end-to-end GenAI / LLM-powered workflows, including prompt engineering, chaining, tool use, and evaluation.
Design and maintain data pipelines specifically for AI workloads, supporting:
Training, fine-tuning, and inference
Vector storage, retrieval-augmented generation (RAG), and embeddings
Integrate AI solutions with existing data platforms while ensuring governance, observability, and cost control.
Apply responsible AI principles, access controls, and auditability for AI-driven systems.

2 Python Engineering, APIs PySpark Development
Develop production-grade Python applications supporting data, AI, and automation use cases.
Design and expose REST APIs using Flask or FastAPI for model inference, AI services, and data access.
Improve and refactor existing Python codebases for maintainability, performance, and scalability.
Migrate legacy logic and scripts into PySpark-based implementations on Databricks.
Follow best practices in:
Modular architecture and design patterns
Logging, monitoring, and exception handling
Unit and integration testing

3 Analytics Platforms - Databricks Snowflake Expertise
Act as an expert practitioner for Databricks and Snowflake, supporting both analytics and AI-driven workloads.
Perform advanced problem solving related to:
Performance tuning
Cost optimization
Query optimization and data layout
Define and enforce standards and best practices for analytics and AI workloads on these platforms.
Support administration and daily management activities, including:
Workspace and resource governance
User access and role management
Platform usage optimization
Enable analytics teams through reusable patterns, templates, and documentation.

4 ETL Cloud Data Engineering (AWS)
Architect and implement scalable ETL pipelines using modern cloud-native patterns.
Apply expert knowledge of ETL frameworks and design principles, including incremental processing and fault tolerance.
Build and maintain pipelines using AWS Glue, interacting with S3, IAM, and related AWS services.
Ensure reliable orchestration, monitoring, and error handling across data pipelines.
Optimize ETL workloads for performance, scalability, and cost efficiency.

5 DBA Concepts Data Storage Expertise
Apply strong DBA fundamentals to analytics and operational systems.
Work with relational and NoSQL databases, including:
Amazon RDS (MySQL, PostgreSQL)
MongoDB Atlas
Apply best practices in:
Indexing and query optimization
Backup, recovery, and high availability
Capacity planning and performance troubleshooting
Collaborate with infrastructure and database teams on design and operational improvements.

Cloud-Based AI Services Platforms (Cross-Cutting Skill Area)
Hands-on experience or strong familiarity with cloud AI and compute services, including:
AWS Bedrock for foundation models and GenAI applications
AWS SageMaker for training, experimentation, and deployment
AWS Lambda for event-driven AI and data automation
Databricks AI for ML/GenAI workloads and platform integration
Microsoft Copilot and related AI tooling for productivity enablement and integration scenarios
Evaluate, integrate, and operationalize AI services based on cost, security, scalability, and use-case fit.

Must Have Skills
6+ years of experience in Data Engineering / Analytics Engineering roles.
Strong hands-on experience with GenAI, LLMs, agentic AI, and multi-agent orchestration.
Advanced Python development experience, including Flask or FastAPI.
Strong experience in PySpark and distributed data processing.
Expert-level proficiency with Databricks and Snowflake.
Deep understanding of ETL architecture, AWS Glue, and cloud-native data pipelines.

Good to Have Skills
Experience with vector databases and RAG architectures.
Exposure to ML lifecycle management, MLOps, and experiment tracking.
Infrastructure-as-Code (Terraform or similar).
Working knowledge of DBA concepts and databases on AWS.
Experience working in regulated domains (banking, fintech, healthcare).
CI/CD practices for data, AI, and analytics pipelines.