LLM Evaluation Professional

Role Overview

As a Geography Domain Expert, you will evaluate and improve Large Language Models by applying your expertise across physical and human geography, GIS concepts, climate, geopolitics, and demographics. You will design challenging prompts, assess AI-generated responses, identify hallucinations and reasoning issues, and contribute to benchmark datasets and adversarial test cases.

Key Responsibilities

• Create advanced prompts covering physical and human geography, GIS concepts, climate, geopolitics, and demographics.
• Evaluate AI-generated responses for factual accuracy, reasoning quality, completeness, and nuance.
• Identify hallucinations, logical inconsistencies, outdated information, and edge cases.
• Develop benchmark datasets and adversarial test cases.
• Provide evidence-based feedback with reliable references.
• Collaborate with AI researchers to improve model performance.
• Maintain high annotation quality and documentation.

Day-to-Day Tasks

• Write domain-specific prompts of varying difficulty.
• Compare multiple AI responses and rank them.
• Explain why a response is correct or incorrect using authoritative references.
• Identify ambiguous questions and propose improved prompts.

Job Requirements

Minimum Qualifications

• Master's degree or higher in a discipline related to Geography.
• 3+ years of professional, research, or teaching experience.
• Excellent written English and analytical skills.
• Strong attention to detail.

Preferred Qualifications

• Experience with LLMs, Generative AI, prompt engineering, or AI evaluation.
• Published research, industry recognition, or teaching experience is a plus.
• Ability to work independently and provide objective, evidence-backed reviews.

Work Details

Location: Kolkata, India

Employment Type: Contractual

Work Mode: WFH

Leave Benefits: This contractual engagement does not include paid medical leave or other paid leave benefits.

Work Schedule: Full-time, 40 hours per week, with at least 4 hours of PST overlap.

Start Immediate

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.