Product Quality Analyst

Partnering with a global leader in digital personalization and streaming technology, Aquent is seeking a dynamic professional to revolutionize how human intelligence grounds next-generation machine learning and AI systems. In this role, you will have a direct impact on shaping the future of personalized digital experiences, bridging the gap between cutting-edge technology and human nuance to ensure millions of users receive highly intuitive, culturally relevant content.

About the Opportunity

Aquent has partnered with an industry-leading digital streaming and technology company known for delivering world-class personalized experiences. This organization relies on sophisticated machine learning and artificial intelligence to connect users with the content they love. By joining this pioneering team, you will help ground advanced AI intelligence in the subtle nuances of culture, personal taste, and real-world habits, playing a vital role in the evolution of their core technology platform.

The Role and Your Impact

We are looking for an operational leader and program-level owner to drive high-quality human judgments that serve as the ground truth for machine learning models. In this highly collaborative position, you will lead end-to-end data annotation projects, establish scalable workflows, and align large language model (LLM) evaluators. This is not a direct labeling role; instead, you will act as the strategic and operational backbone, designing rubrics, managing vendor partnerships, and freeing up engineering and data science bandwidth by professionalizing the human-in-the-loop lifecycle.

What You'll Do

• Lead the end-to-end execution of human annotation projects, managing everything from initial scoping and intake to final delivery.
• Serve as the primary liaison between data science, engineering, and external partners to map annotation requirements to the most efficient pipelines.
• Maintain and prioritize a comprehensive roadmap of annotation needs, coordinating internal and external resources effectively.
• Manage vendor relationships, including contracts, onboarding, and coordination with legal, security, privacy, finance, and accounting teams.
• Develop, maintain, and consult on reusable annotation frameworks, task designs, and evaluation rubrics.
• Facilitate feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Systematically evaluate annotation quality, resolve disagreements, and continuously improve guidelines and outputs.
• Establish operational documentation and best practices for a newly formalized function, partnering closely with dedicated data scientists.

What You'll Bring

• Proven background in data operations (Data Ops), annotation operations, model evaluation, product quality, or technical program management.
• Hands-on experience managing, coordinating, or designing data annotation workflows at an operational or program level.
• Demonstrated experience working within a scaled technology company or directly collaborating with established data annotation vendors.
• Familiarity with machine learning-driven products and the human evaluation of AI/LLM-powered features.
• Strong ability to translate complex product and modeling needs into clear, scalable annotation guidelines and evaluation rubrics.
• Experience navigating vendor onboarding, procurement coordination, and cross-functional project tracking.
• Exceptional cross-functional communication skills, with a proven ability to align technical teams and navigate ambiguity.
• A self-starter mindset, comfortable stepping into a 0-to-1 environment to build, formalize, and streamline processes.

Bonus Points

• Experience interacting with AI collaboration tools (e.g., Claude, Codex).
• Basic coding abilities.
• Experience scaling annotation pipelines from early-stage setups to operational maturity.