Software Engineer ( Python + PySpark + Agentic AI )

- Write and tune PySpark jobs on EMR (EMR on EKS) and Delta Lake to prepare and transform data at scale
- Design and build LLM-powered agents that plan tasks, call tools, and propose or apply actions against our services and data, under confidence thresholds and human-in-the-loop review
- Define the tools and APIs agents can use, and connect agents to internal systems (e.g., via Model Context Protocol)
- Build retrieval-augmented (RAG) pipelines: chunking, embeddings, vector search, and grounding responses in our data
- Design agent workflows with clear guardrails, confidence gating, human-in-the-loop exception handling, and fallback behavior
- Write prompts and evaluation harnesses; run evaluation in CI and measure agent quality, cost, and latency, then iterate
- Build and maintain Python services and shared libraries, and Kafka producers/consumers for moving data between services
- Serve agents behind APIs and monitor them with LLM/agent observability (quality, cost, latency, failures)
- Write design docs and collaborate with engineers and data scientists on architecture decisions
- Use AI coding agents in your own day-to-day workflow while owning correctness and quality

We are looking for an experienced Python + PySpark Developer with a strong background in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate should have strong expertise in Python, PySpark, SQL, and big data technologies. You will build the large-scale data pipelines and agentic AI features that power our platform on AWS. You will apply strong Python and PySpark skills to data processing on EMR on EKS, and build LLM-based agents that assist and automate work on our data under human oversight.