Senior AI Engineer - LLM Evaluation

Job Requisition ID # 26WD98447

Position Overview If you love building real Artificial Intelligence (AI) systems that solve real customer problems and are passionate about measuring, improving, and scaling AI quality, this role is for you. The Applied AI team within Autodesk's Architecture, Engineering, and Construction (AEC) organization builds cloud-native AI solutions that make industry workflows smarter, faster, and more connected. This role is primarily focused on AI evaluation and quality engineering for Generative AI systems. You will establish evaluation standards, build automated validation frameworks, and drive observability practices that ensure AI systems are reliable, trustworthy, and production-ready. In addition to AI evaluation, you will contribute to the design and development of AI-powered applications, including Retrieval-Augmented Generation (RAG) systems, agentic workflows, and Large Language Model (LLM)-based solutions deployed at scale.

Responsibilities

• Lead AI evaluation and quality strategy for Generative AI and Machine Learning (ML)-powered systems
• Design and implement scalable evaluation frameworks for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, agentic workflows, and traditional Machine Learning (ML) models
• Fine-tune, evaluate, benchmark, and deploy Large Language Models (LLMs) in production environments
• Implement evaluation and observability standards, including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems
• Define and track AI quality metrics, including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates
• Build automated evaluation pipelines and integrate them into Continuous Integration and Continuous Deployment (CI/CD) workflows for continuous model validation
• Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tools
• Build scalable Retrieval-Augmented Generation (RAG) pipelines using structured and unstructured enterprise data sources
• Integrate AI agents with Autodesk platforms and enterprise services using Model Context Protocol (MCP)-based integrations or equivalent architectures
• Partner with Product Managers, Architects, Software Engineers, Experience Designers, and Data Scientists to deliver AI-powered experiences
• Ensure AI systems align with Responsible AI principles and enterprise governance requirements
• Contribute to reusable AI platform capabilities, Software Development Kits (SDKs), orchestration patterns, and engineering best practices
• Leverage AI coding assistants to accelerate feature delivery while maintaining rigorous validation through testing, Continuous Integration and Continuous Deployment (CI/CD), code reviews, and security best practices

Minimum Qualifications

• Bachelor's or Master's degree in Computer Science, Machine Learning, Data Science, Engineering, or equivalent practical experience
• 5+ years of hands-on experience delivering Artificial Intelligence (AI) and Machine Learning (ML)-powered systems into production
• Experience designing evaluation frameworks, automated testing solutions, benchmarking systems, or model validation pipelines
• Strong understanding of Large Language Model (LLM) evaluation, model quality measurement, and AI observability concepts
• Strong proficiency in Python with experience building automation frameworks and testing infrastructure
• Experience deploying and operating AI applications in production, including monitoring, observability, and continuous improvement
• Experience building cloud-native applications on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
• Excellent technical communication and collaboration skills
• Demonstrated experience using AI coding tools to accelerate the delivery of production systems

Preferred Qualifications

• Hands-on experience with AI evaluation frameworks such as LangSmith, DeepEval, TruLens, Promptfoo, OpenAI Evals, or similar tools
• Experience evaluating Retrieval-Augmented Generation (RAG) systems, agentic workflows, AI assistants, and prompt-based applications
• Familiarity with Responsible AI, AI governance, model risk assessment, and AI observability platforms
• Experience deploying cloud-native AI solutions on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
• Familiarity with deep learning architectures such as Transformers and modern Machine Learning (ML) frameworks, including PyTorch, Lightning, Ray, or equivalent technologies
• Experience with enterprise knowledge systems, semantic search, vector search, or contextual retrieval systems
• Experience with emerging agentic AI frameworks such as LangGraph, CrewAI, OpenAI Software Development Kits (SDKs), Claude Software Development Kits (SDKs), Model Context Protocol (MCP), Agent-to-Agent (A2A), or equivalent technologies
• Contributions to open-source AI projects, technical blogs, conference presentations, or published research are an advantage

The Ideal Candidate

• Demonstrates a strong understanding of Artificial Intelligence (AI), Machine Learning (ML), and Generative AI concepts and applies industry best practices to solve complex technical challenges
• Uses analytical thinking, sound judgment, and data-driven decision-making to evaluate model performance and recommend effective solutions
• Connects technical implementation with broader product, customer, and business objectives
• Applies creativity and innovation to improve AI quality, evaluation methodologies, and engineering practices
• Leads projects or key workstreams while contributing to the successful delivery of broader initiatives
• Takes ownership of improving engineering processes, quality standards, and operational excellence
• Builds strong working relationships with senior technical leaders, peers, and cross-functional stakeholders
• Works independently while seeking guidance on highly complex or ambiguous challenges when appropriate
• Demonstrates excellent collaboration and communication skills when working across globally distributed teams
• Shares knowledge, mentors less experienced engineers, and contributes to a culture of continuous learning
• Maintains a passion for learning emerging AI technologies, frameworks, and best practices while continuously improving technical expertise Learn More

About Autodesk Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made. We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world. When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Salary transparency Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global\-belonging

Are you an existing contractor or consultant with Autodesk? Please search for open jobs and apply internally (not on this external site).