AI Evaluation Specialist (LLM / Generative AI)

AI Evaluation Specialist (LLM / Generative AI)

Location

Remote (Continental U.S.) Employment type: W-2 contract, hourly Eligibility: U.S. Citizen or Lawful Permanent Resident (Green Card Holder) Background: Must be able to obtain and maintain a Public Trust

About the Role

Elite Talent Hub is seeking an AI Evaluation Specialist to support an AI/ML team on a federal public health program. You will determine whether enterprise AI tools are effective, safe, usable, and ready for business adoption.

This is a hands-on evaluation role, not a model-building role. You will design repeatable evaluation frameworks, test generative AI tools against real-world use cases, including OpenAI products and life sciences models such as GPT-Rosalind, and turn your findings into clear recommendations. You will work with product managers, data scientists, AI and software engineers, security and privacy teams, and business stakeholders.

What You’ll Do

• Plan and run hands-on evaluations of AI tools using representative business scenarios, workflows, prompts, datasets, and user personas
• Assess response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and task effectiveness
• Build and maintain standardized frameworks, scorecards, rubrics, and acceptance criteria for AI tools, models, agents, and AI-enabled applications
• Work with security, privacy, legal, and governance stakeholders to evaluate safety, privacy, security, data handling, and responsible AI considerations
• Conduct comparative evaluations and proofs of concept across AI products, models, configurations, and vendors, including functionality, integration requirements, performance, and cost
• Evaluate AI agents and tool-enabled workflows, including RAG, enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP) capabilities
• Stay current on generative AI, model evaluation, AI safety, and responsible AI, and bring relevant methods into the evaluation program
• Share knowledge and contribute to a strong, collaborative team

What You Bring

• 5+ years of experience in model evaluation or model quality assurance, including hands-on evaluation of AI-enabled products or services
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria
• Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identifying hallucinations and other failure modes
• Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost
• Familiarity with common generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, enterprise search, and tool or connector integrations
• Bachelor’s degree

Nice to Have

• Experience evaluating enterprise AI platforms or assistants, including OpenAI products, life sciences models such as GPT-Rosalind, or comparable generative AI solutions
• Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows
• Experience building dashboards or reports for AI evaluation metrics, quality trends, risk findings, and regression results
• Experience using Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings
• Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI
• Master’s degree

Pay: $80.00 - $90.00 per hour

Expected hours: 40.0 per week

Work Location: Remote