Senior AI Data Engineer

Project Description:We are looking for AI Engineers for a strategic initiative to build Agentic AI use cases around Trading Surveillance. Ideal candidate is someone with strong pyspark background coupled with proven experience build Agentic AI pipelines for code conversion from Q KDB to Pyspark.

Responsibilities:

• Design, develop, and maintain full-stack Python applications with modern frontend frameworks
• Build and optimize RAG (Retrieval-Augmented Generation) systems for AI applications
• Create and implement efficient vector databases and knowledge stores
• Develop APIs that connect frontend interfaces with backend AI services
• Implement and maintain CI/CD pipelines for AI applications
• Monitor application performance and troubleshoot issues in production
• Design and maintain data pipelines, ETL/ELT workflows, and data infrastructure to support AI and analytics workloads

Mandatory Skills:

• Artificial Intelligence (AI)
• PySpark

Mandatory Skills Description:

• Minimum of 5 years of experience in AI/ML solution implementation (mandatory)
• Minimum of 3 years of experience in Pyspark processing high volume data (mandatory)
• AI & ML Skills:
Multi-Agent System Design & Orchestration
Full-stack AI Integration (Frontend to Backend)
Advanced Prompt Engineering
LLM Observability, Evaluation & Monitoring (LLMOps)
• Data Engineering Skills:
Proficient in Pyspark coding and concepts. Example: Lazy evaluation, window functions, spark architecture and performance finetuning (ex: data skew reduction (salting etc), joins (broadcast, merge sort joins ect)
SQL proficiency and query optimization

• Main Technologies:
Pyspark
Git hub Copilot

Languages:

• English: C2 Proficient