Gen AI Developer

What You'll Do
Daily health & usage review check agent health endpoints, MCP server status, token usage dashboards, and user feedback queues across AI platforms, Incident Response, Agent Kubectl-AI, and the Gateway; flag anomalies early.
Incident response & RCA triage errors, slow responses, and availability issues; use logs, OpenTelemetry/Langfuse traces, and Kubernetes tooling (kubectl/oc, pod logs, HPA/resource metrics) to find root cause and remediate; document outcomes.
Token/cost optimization monitor Bedrock token consumption per agent/tool, identify high-cost call patterns (e.g., oversized tool results, redundant context), and propose or implement fixes (prompt trimming, tool-result summarization, model tier adjustments, offloading heavy LLM calls to lighter-weight agents/tools).
Backlog delivery implement enhancement-level tickets from the CTO AI backlog: new/adjusted MCP tools, agent config changes, prompt updates, dashboard/report tweaks, minor UI fixes in the ChatOps Hub.
Monitoring & alerting help build out and mature alerting (health checks, error-rate thresholds, token-usage thresholds, MCP server availability) using existing observability stack (Langfuse, OpenTelemetry, Kubernetes-native metrics).
Feedback loop management review end-user feedback submitted through the platform; resolve directly when possible, or route to the right team/agent owner with context.
Operational hygiene keep runbooks, on-call notes, and known-issue documentation current; contribute to release/rollback procedures for dev lab prod promotion via Jenkins/Kustomize.

Must-Have Skills
3+ years professional software engineering or SRE/DevOps experience
Strong Python (FastAPI or similar async web framework) able to read and safely modify a production codebase
Working knowledge of Kubernetes (kubectl, deployments, ConfigMaps/Secrets, logs, basic troubleshooting) ideally OpenShift and/or EKS
Comfortable with CI/CD pipelines (Jenkins preferred) and Git-based workflows (this org uses Bitbucket Server, not GitHub)
Experience debugging production incidents: reading logs/traces, forming hypotheses, verifying fixes
Familiarity with LLM-based systems prompts, tool/function calling, token usage/cost concepts (hands-on experience with LangChain/LangGraph, Bedrock, OpenAI, or similar is a strong plus)
Basic SQL (Postgres) for querying conversation/audit/usage tables
Clear, concise written communication for incident notes, runbooks, and backlog ticket updates

Nice to Have
Experience with LangGraph or another agent-orchestration framework
Experience with MCP (Model Context Protocol) or building/consuming tool servers
Familiarity with OPA (Open Policy Agent), JWT/OIDC auth flows, or Azure AD integration
Exposure to HashiCorp Vault for secret management
Frontend experience (React/TypeScript) to make small ChatOps UI fixes
Experience operating in a regulated financial-services or similarly compliance-heavy environment
Familiarity with observability tools such as Langfuse, Grafana, or OpenTelemetry pipelines
Exposure to service registry/discovery patterns (service meshes, API gateways, package/artifact registries) directly transferable to upcoming MCP registry and agent control-plane integration work

• Communication
• Java
• Java Cloud Service
• JAXP
• Life Insurance

That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.