LLM Engineer - Manager

Role & responsibilities

Mandatory Skills

1. Advanced Prompt Engineering.

• Experience with structured outputs (JSON mode, function/tool calling) and system prompt design.
• Ability to systematically test, version, and optimize prompts for quality and token efficiency.
• Knowledge of prompt injection risks and mitigation techniques.

2. LLM Architecture Design

• Strong understanding of transformer architecture, attention mechanisms, tokenization, and context windows.
• Ability to design scalable LLM system architectures, including agents, memory, tool use, and orchestration layers.
• Experience making trade-off decisions on model selection, latency, cost, and accuracy.

3. RAG System Implementation

• Hands-on experience building production RAG systems end-to-end.
• Proficiency with vector databases such as Pinecone, Weaviate, FAISS, Chroma, Milvus, Qdrant, or pgvector.
• Knowledge of embedding models, chunking strategies, hybrid search (semantic + keyword), re-ranking, and query transformation.
• Experience evaluating RAG pipelines using tools like RAGAS, TruLens, or custom metrics.

4. Fine-tuning Strategy

• Ability to determine when fine-tuning is the right approach compared with prompting or RAG.
• Hands-on experience with PEFT techniques (LoRA, QLoRA), instruction tuning, and alignment methods (RLHF, DPO).
• Experience with dataset curation, cleaning, and preparation for fine-tuning.
• Familiarity with Hugging Face Transformers, PEFT, TRL, Axolotl, or Unsloth.

5. LangChain / LlamaIndex (Advanced)

• Deep, hands-on experience building complex chains, agents, and multi-step workflows.
• Proficiency with LangGraph, LCEL, custom tools, retrievers, memory modules, and callbacks.
• Experience with LlamaIndex indexing strategies, query engines, and data connectors.
• Ability to extend frameworks with custom components when needed.

6. Multi-model Integration

• Experience integrating and orchestrating multiple LLM providers and open-source models.
• Ability to implement model routing, fallback mechanisms, load balancing, and cost optimization.
• Exposure to multimodal models (text, image, audio, vision-language).
• Familiarity with gateways and abstraction tools such as LiteLLM, OpenRouter, or similar.

7. Production Deployment

• Proven experience deploying LLM applications to production at scale.
• Proficiency with model serving frameworks such as vLLM, TGI, Ollama, Triton, or TensorRT-LLM.
• Strong knowledge of Docker, Kubernetes, CI/CD, and cloud platforms (AWS, Azure, or GCP), including services like Bedrock, Azure OpenAI, or Vertex AI.
• Experience with LLMOps and observability tools such as LangSmith, MLflow, Weights & Biases, Langfuse, or Arize.
• Understanding of caching, batching, quantization, rate limiting, and latency optimization.