AI Quality Engineer

Job Description

===================

Key Responsibilities

• Design and implement evaluation frameworks (evals) to assess LLM and agentic AI system quality, including accuracy, consistency, safety, and task completion rates.
• Build and maintain automated test pipelines for AI features, covering unit, integration, and end-to-end scenarios across agentic workflows.
• Develop tooling to detect regressions in model behavior, prompt outputs, and agent decision-making across releases.
• Define and track quality metrics for AI systems (e.g., hallucination rates, tool-use accuracy, latency, failure recovery) and surface findings clearly to stakeholders.
• Collaborate with engineers and product managers to identify edge cases, adversarial inputs, and failure modes specific to multi-step agentic pipelines.
• Contribute to prompt evaluation strategies, including red-teaming, adversarial testing, and bias/fairness assessments.
• Participate in design and code reviews with a quality-focused lens, raising concerns about testability and reliability early.
• Help define and document quality standards and best practices for AI/ML features across the team.
• Other duties as assigned.

Qualifications

Required

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
• 3–5 years of professional software engineering or quality engineering experience.
• Hands-on experience working with LLMs or agentic AI systems (e.g., GPT-4, Claude, Gemini, or open-source models).
• Proficiency in Python for scripting, test automation, and data analysis.
• Experience designing and running evaluations (evals) for generative AI or LLM-powered features.
• Solid understanding of software testing principles: unit, integration, regression, and end-to-end testing.
• Familiarity with agentic frameworks and concepts (e.g., tool use, multi-step reasoning, retrieval-augmented generation, memory).
• Experience with CI/CD pipelines and integrating aut...