Program Analyst Sr - Generative AI, Prompt Engineering & Validation

Job Area: Operations Group, Operations Group > Program Analyst

General Summary:

Assists with program development and implementation through managing processes, procedures, and tools that improve efficiencies. A Program Analyst coordinates across teams and monitors timelines, budgets, risks, and priorities to achieve program progress. Typically, a program needing a Program Analyst will be of significant size and will require expertise related to the development of project management mechanisms.

Minimum Qualifications:

Associate's degree in Business Administration, Management, Computer Science, Engineering, Computer Science, or related field. OR High School Diploma or equivalent and 2+ years of relevant work experience.

Job Title: Program Analyst Sr Generative AI, Prompt Engineering Validation

Location: Bangalore

Employment Type: Full-time

About the Role

We are looking for a proactive and detail-oriented Program Analyst to support Generative AI initiatives focused on prompt engineering, feature validation, test execution, output quality evaluation, and program tracking. The role will support internal GenAI platforms and tools, including search, summarization, RAG-based workflows, analytical agents, and domain-specific AI assistants.

The ideal candidate should combine strong program coordination skills with a practical understanding of GenAI systems, structured prompt-based testing, multi-source data validation, and clear reporting for engineering and leadership stakeholders.

Key Responsibilities1. GenAI Program Tracking Coordination

• Track GenAI feature validation plans, deliverables, owners, dependencies, and closure status across teams.
• Maintain dashboards, trackers, action item logs, risk registers, and validation status reports.
• Coordinate with engineering, product, platform, and domain expert teams to drive timely completion of testing and feedback cycles.
• Consolidate weekly progress updates, blockers, risks, and recommendations for program reviews.
• Ensure validation documentation, prompt libraries, and issue logs are accurate, current, and accessible.

2. Prompt Engineering Prompt Test Design

• Create, refine, and maintain prompts for GenAI features, search workflows, summarization use cases, and analytical agents.
• Develop zero-shot, few-shot, role-based, multi-turn, edge-case, and negative test prompts.
• Build reusable prompt templates and feature-wise prompt libraries.
• Compare prompt variants and document impact on accuracy, relevance, completeness, consistency, and latency.
• Translate domain expert feedback into improved prompts and updated test scenarios.

3. Feature Validation Test Execution

• Design and execute structured test cases for GenAI features, including functional, sanity, regression, negative, and exploratory testing.
• Validate end-to-end behavior from user prompt to retrieved context, generated response, and final output format.
• Map feature requirements to prompts, expected outputs, acceptance criteria, and validation evidence.
• Identify defects, usability gaps, hallucinations, incomplete answers, inconsistent responses, and low-confidence outputs.
• Support release readiness assessment using clear status indicators such as Go / Watch / Risk.

4. Data, Search RAG Validation

• Validate AI outputs against structured, semi-structured, and unstructured data sources such as SharePoint, Confluence, Jira, documents, JSON/XML, and internal knowledge bases.
• Check search relevance, context quality, source completeness, and data freshness before summarization.
• Support validation of RAG-based workflows, semantic search, keyword search, global filters, and domain filters.
• Identify data gaps, indexing issues, retrieval failures, and source-quality problems impacting AI response accuracy.
• Maintain golden datasets, expected-answer sets, and benchmark queries for repeatable validation.

5. Output Quality Evaluation Defect Analysis

• Evaluate GenAI responses for factual correctness, relevance, completeness, clarity, format quality, and consistency across repeated runs.
• Log defects with clear reproduction steps, prompt used, expected output, actual output, severity, and owner.
• Perform first-level root cause analysis to distinguish prompt issues, data issues, retrieval issues, model limitations, and product defects.
• Work with SMEs to validate technical correctness and define acceptance thresholds for complex responses.
• Track quality improvements over time and report trends in accuracy, hallucination reduction, and feature readiness.

6. Reporting, Documentation Stakeholder Communication

• Prepare weekly validation summaries, risk updates, action item trackers, and executive-ready status reports.
• Document prompt libraries, test suites, validation evidence, known issues, and improvement logs.
• Facilitate review meetings, capture minutes of meeting, document decisions, and follow up on action items.
• Communicate clearly with cross-functional stakeholders across engineering, platform, product, and program management teams.
• Support leadership reviews with concise summaries, dashboards, and recommendations.

7. Automation Productivity Support

• Use Excel, Power BI, Jira, Confluence, SharePoint, or similar tools to maintain trackers, dashboards, and reporting views.
• Support basic scripting or automation for batch prompt execution, response capture, comparison, and reporting where applicable.
• Help improve validation efficiency through reusable templates, structured checklists, and repeatable test processes.
• Track productivity metrics such as test coverage, closure rate, defect aging, prompt reuse, and feature readiness.

Required Skills Competencies

• Hands-on exposure to Generative AI tools, LLM-based applications, prompt engineering, or AI-assisted workflows.
• Strong ability to create structured test cases, validation reports, action trackers, and executive summaries.
• Good understanding of prompt-response behavior, hallucinations, context limitations, and output evaluation methods.
• Working knowledge of search, summarization, RAG concepts, information retrieval, or knowledge-base validation.
• Strong Excel skills; Power BI, Jira, Confluence, SharePoint, or dashboarding experience is preferred.
• Basic Python, SQL, or scripting knowledge is preferred but not mandatory.

Preferred QualificationsEducation Background

• Bachelors degree in Engineering, Computer Science, Electronics, Data Science, Information Systems, or a related discipline.
• Prior experience in semiconductor, SoC, embedded systems, platform engineering, enterprise AI, or technical program management is preferred.
• Experience supporting validation of internal