Red Team Manager, Training, Quality, Roleplay Excellence
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Red Team Manager leading evaluations and training for advanced AI systems at mpathic. Focusing on safety, reliability, and quality of adversarial roleplays.
Responsibilities:
- Train & Lead Red Team Reviewers
- Onboard new Red Team reviewers and run recurring calibration sessions to align on quality standards.
- Set expectations and maintain consistency across reviewers for evaluation depth, writing quality, and reproducibility.
- Build workflows for review (sampling, escalation, dispute resolution, feedback loops).
- Train Experts on Roleplays, Model Behavior & Harm
- Train red team experts on how to roleplay realistic user scenarios—including vulnerable users—without sensationalism.
- Teach systematic adversarial techniques (prompt escalation, persistence strategies, boundary probing).
- Help experts understand model failure modes: policy boundary drift, refusal weaknesses, hallucinations, unsafe compliance, and tone failures.
- Create Training Materials & Resources
- Build and maintain: Red team playbooks and rubrics Example libraries (“gold standard” roleplays + evaluations) Defect taxonomy (what counts as a meaningful finding vs noise) Brief modules for domain harm areas (self-harm, minors, extremism, medical, fraud, harassment, etc.) Write clear guidance that enables new hires to become productive quickly.
- Review & Evaluate Vulnerable User Roleplays
- Review vulnerable-user roleplays produced by experts for realism, safety relevance, and correct targeting of failure modes.
- Ensure roleplays are: behaviorally plausible ethically framed actionable for model improvement consistent with internal policies and customer expectations.
- Create Vulnerable User Roleplays
- Personally produce high-quality vulnerable-user roleplays, including: ambiguous edge cases multi-turn scenarios culturally nuanced or emotionally realistic interactions scenarios that stress safety, tone, and reliability.
- Review Hiring Applicants
- Own parts of the hiring loop for red team experts and reviewers: design work samples evaluate candidate submissions provide structured feedback and hiring recommendations. Help build a scalable standard for what “great” looks like in this role.
Requirements:
- 4+ years in trust & safety, AI evaluation, red teaming, security testing, content integrity, or similar applied roles.
- Strong experience building training programs, rubrics, or QA frameworks for human judgment work.
- Ability to evaluate roleplays and adversarial scenarios with consistency and high signal-to-noise.
- Excellent written communication—clear, structured, and test-case oriented.
- Experience leading or mentoring teams in fast-moving environments.
- Experience red teaming LLMs, agentic systems, or tool-using models (prompt injection, data exfiltration, policy probing).
- Familiarity with evaluation methods: gold sets inter-rater reliability (or strong proxy measurement instincts) sampling strategies and quality gates.
- Background in one or more harm domains (self-harm, medical, violence, fraud, extremism, harassment).
- Experience scaling an operational team and improving productivity without quality loss.
Benefits:
- Health insurance
- Professional development
Similar Jobs
Senior Environmental, Health & Safety Manager
Senior Environmental, Health & Safety Manager overseeing EHS programs for data center operations. Ensuring compliance and safety across Serverfarm’s U.S. data centers with a focus on sustainability.
Senior Renewals Manager
Senior Renewals Manager driving retention and revenue growth at Appriss Retail. Overseeing subscription renewals and managing complex commercial workloads within a SaaS environment.
ELO ECM Specialist, Projektmanager
ELO ECM Specialist managing the ELO ECM platform and driving digital transformation initiatives at Th. Geyer GmbH. Responsible for system administration, project management, and cross-functional collaboration.
Manager, Martech
Manager, Martech improving acquisition systems at Mercer Advisors. Focused on Salesforce, automation, and growth tech operations.
Country Manager
Country Manager driving growth across Sweden, Finland, and Denmark for an international iGaming group. Focused on poker ecosystems and emerging market environments with emphasis on commercial strategy.
Senior Manager, Data & Analytics
Senior leadership role managing Data Platform, Governance, and Architecture teams at Blip Global. Driving development and evolution of data systems while ensuring compliance and security.
Senior Manager, Asset Protection
Senior Manager, Asset Protection at Arc’teryx managing risk and loss mitigation to support North America Retail operations. Leading best practices and crisis response frameworks for physical risk incidents.
Field Onboarding & Integration Manager
Field Onboarding & Integration Manager at Thrivent leading onboarding specialists and enhancing advisor launch experiences. Managing training processes and team performance in financial services with a focus on continuous improvement.
Embedded ABM Manager
Embedded ABM Manager executing account-specific marketing plans and strategies. Building engagement with stakeholders to drive Neara's innovative solutions in the energy sector.
Clinical Applications Manager, MR
Clinical Applications Manager overseeing MRI clinical activities, managing scanner protocols and driving productivity at Akumin's fixed sites. Supporting technologists and radiologists with QA/QC programs and training.
Senior Engagement Manager
Senior Engagement Manager driving client success and managing strategic engagements for Seismic's enterprise clients. Focus on building relationships and overseeing implementation of services.
Pre-Access Manager
Manager of Pre-Access overseeing the Patient Access department at Ensemble. Leading improvement initiatives for customer service and departmental performance.
Valuation Manager
Valuation Manager coordinating exchanges between external appraisers and internal departments. Managing the real estate evaluation process with emphasis on quality and stakeholder communication.
Regional Field Manager – North-East
Regional Field Manager leading a team of Area Sales Managers for Vivaris and Berentzen. Focused on sales effectiveness and KPI achievement within the North-East territory of Germany.
Senior Manager, Client Account Support
Senior Manager overseeing client implementations for Under 500 clients at Cigna Healthcare. Driving strategy execution and fostering high-performance teams within the organization.
Engagement Manager
Engagement Manager managing multiple Workday implementations in various sectors. Guiding clients to maximize value from Workday's cloud solutions with a focus on project management.
Senior Manager, Real Estate
Senior Manager of Real Estate managing a portfolio of over 200 globally owned/lease sites for Regal Rexnord. Collaborating with leadership to execute strategies and manage real estate transactions.
Senior Internal Audit Manager
Internal Audit Senior Manager executing audit plans and ensuring compliance in a dynamic banking service environment. Collaborating with teams to shape audit strategies and mentor staff in financial services.
Senior Paid Acquisition Manager
Senior Marketing Specialist managing paid acquisition strategies for ShareGate growth and experimentation with performance marketing channels.
Emergency Response Manager – Stormwater
Stormwater Response Manager handling stormwater-related requests and coordinating resources for national clients. Seeking qualified candidates with project management experience in environmental consulting.