AI SDET – Software Development Engineer in Test
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI SDET testing AI agents for safety, compliance, and accuracy at CI&T, an enterprise AI and technology transformation company. Managing evaluation harnesses, regulatory quality gates, shadow validation, and adversarial automated testing.
Responsibilities:
- Own and manage the Langfuse evaluation harness, including creation of "Golden Paths" and adversarial test scenarios
- Validate regulatory quality gates (SR 26-2, NYDFS Reg 187)
- Run shadow mode validation to compare AI performance against human benchmarks
- Test for data quality degradation, population segmentation errors, and human-in-the-loop (HITL) checkpoint reliability
- Verify that the Designer’s output standards, including explanation blocks and signal types, are maintained in all evaluation scenarios
- Find edge cases through skeptical and adversarial testing
- Collaborate with engineers to fix logic failures
Requirements:
- Advanced proficiency in Python 3.11+, Pytest, and automated testing frameworks
- Direct experience with AI evaluation tools such as Langfuse, RAGAS, or similar
- Experience with LLM-as-a-judge patterns
- Knowledge of model risk management and regulatory compliance testing
- Experience with shadow mode deployment strategies and statistical validation
- Skeptical and adversarial mindset to find edge cases
- Uncompromising attention to detail regarding regulatory evidence
- Collaborative approach to working with engineers to fix logic failures
Benefits:
- Health and dental insurance
- Meal and food allowance
- Childcare assistance
- Extended paternity leave
- Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;
- Profit Sharing and Results Participation (PLR);
- Life insurance
- Continuous learning platform (CI&T University);
- Discount club
- Free online platform dedicated to physical, mental, and overall well-being
- Pregnancy and responsible parenting course
- Partnerships with online learning platforms
- Language learning platform
Similar Jobs
Senior General Test Engineer
Senior QA engineer automating UI, API, and performance testing for a global mining company through ELEKS. Developing test scripts, documentation, metrics, and quality assurance processes.
Senior Software Engineer – Test
Senior SDET improving quality for Replicant’s voice AI customer-service platform. Building AI-driven testing, automation frameworks, analytics, and CI/CD quality gates for scalable releases.
Senior Test Automation Engineer, Python
Senior QA engineer testing Python automation, low-code applications, and AI agents for Avenga. Defining strategies, validating outputs, tracking defects, and owning release sign-off.
Test Automation Engineer
Test Automation Engineer building UI, API, and integration testing frameworks for GDIT. Improving software quality for U.S. government, defense, and intelligence missions.
Lead QA Automation Engineer
Lead QA automation for Perch, a digital mortgage platform helping Canadians build wealth through real estate. Setting quality standards, managing QA engineers, and maintaining automated regression coverage.
Automation Test Engineer
Remote Test Automation Engineer building UI, API, and integration test frameworks for a U.S. Government client. Improving reliability, coverage, and defect prevention across modern application stacks.
QA Engineer – Automation, Functional
QA Engineer automating and validating clinical-grade cardiac monitoring software at RhythmScience. Building Playwright coverage, testing web/mobile apps, and improving release quality.
QA Test Engineer
QA Test Engineer for ALTEN Mexico, an engineering and IT services company, testing web applications, APIs, and databases. Designing test plans, automating tests, tracking defects, and validating releases.
Automation Test Engineer
Automation Test Engineer developing cloud-based automated testing for GDIT’s U.S. government case-management modernization. Building QA frameworks, scripts, and CI/CD integrations for enterprise applications.
Middle Test Engineer
Middle Test Engineer testing a PDF reader with document-signing capabilities for ELEKS in Ukraine. Executing tests, managing test artifacts, and reporting progress to stakeholders.
Senior QA Automation Engineer, Playwright/TypeScript
Senior QA Automation Engineer building Playwright/TypeScript coverage for a US telehealth platform. Integrating automated testing across healthcare workflows, APIs, integrations, and AWS CI/CD.
Senior Test Engineer
Senior Test Engineer defining testing strategies and automating quality assessments for software serving European institutions. Remote within the EU, collaborating with development and client IT teams.
QA Automation Analyst – Testing Funcional y Automatización
QA Automation Analyst in a consulting firm providing custom technological solutions. Designing testing plans and collaborating with development teams to ensure application quality.
Senior Hardware Test Development Engineer, Python
Hardware Test Development Engineer ensuring quality and reliability of hardware products for Jabil. Design, develop, and maintain test systems and software in a collaborative team environment.
Senior Test Automation Engineer – ConnectWise
Senior Test Automation Engineer enhancing test automation frameworks for ConnectWise. Focused on migrating to new platforms with CI/CD integration and team collaboration.
Test Development Engineer Intern
Test Development Engineer Intern at Bybit participating in full product testing lifecycle. Collaborating cross-functionally and ensuring product quality through various testing methods.
Senior Technical Account Manager – Automation Test Engineer
Technical Account Manager facilitating technical onboarding and customer success in visual AI test automation. Collaborating with engineering teams and driving technical strategies for customer engagement.
Software Engineer, Test
Software Engineer responsible for automated test coverage for ServiceTitan's Dispatch Engineering team. Building tests with Playwright for a high-impact, AI-forward squad.
Test Engineer – Federal Health
Test Engineer supporting federal healthcare modernization initiatives at rockITdata. Collaborating across teams to deliver reliable and high-performing healthcare technology solutions.
QA Automation Engineer
QA Automation Engineer for IRIUM, a company focusing on tech solutions, performing test automation remotely. Requires Selenium and Appium experience, and knowledge in Java or Python.