AI Software Development Engineer in Test – SDET
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
AI SDET testing LLMs, agents, RAG pipelines, and automation frameworks at Newfold Digital, a global web technology company. Building reliable, secure AI quality solutions.
Responsibilities:
- Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents
- Develop AI evaluation frameworks for hallucination detection, groundedness, toxicity and bias, response quality, context relevance, prompt regression, determinism, consistency, safety, and guardrails
- Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics
- Create synthetic and benchmark datasets for AI testing scenarios
- Test multi-agent orchestration workflows and agent memory/context retention
- Validate MCP integrations and tool interoperability
- Work on Context Engineering, Prompt Engineering, and Prompt Optimisation strategies
- Evaluate latency, token usage, model cost optimisation, throughput, and scalability
- Develop and maintain scalable UI/API automation frameworks using Python and testing tools
- Automate testing for REST APIs, GraphQL APIs, event-driven systems, streaming systems, webhooks, microservices, and MFE architectures
- Build reusable automation libraries, utilities, and AI-assisted testing workflows
- Integrate automation into CI/CD pipelines and quality gates
- Validate RAG architectures, vector databases, embeddings, retrieval accuracy, re-ranking, and vector search relevance
- Build or validate AI agents and workflows involving tool calling, function calling, agent memory, planning/execution, multimodal systems, and structured outputs
- Maintain quality workflows integrated with CI/CD platforms
- Work with Docker and Kubernetes environments
- Perform cloud-based testing and deployments across AWS, Azure, OCI, and GCP
- Validate distributed systems reliability and resiliency
- Monitor AI system behaviour using observability, logging, and tracing tools
- Contribute to shift-left and quality engineering best practices
- Collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams
Requirements:
- Strong programming experience in Python
- Strong understanding of Software Testing & Quality Engineering principles
- Hands-on experience with AI/LLM testing, Prompt Engineering, Context Engineering, AI Evals, RAG systems, and AI Agents
- Experience with CI/CD pipelines, Docker & Kubernetes, cloud platforms (AWS/Azure/OCI/GCP), API testing & automation, microservices testing, and UI automation
- Experience with SQL and data validation
- Strong debugging, analytical, and troubleshooting skills
- Experience working in Agile/Scrum environments
- Experience with Robot Framework preferred
- Experience with Playwright preferred
- Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, and MCP preferred
- Hands-on experience with Cursor, GitHub Copilot, Claude, Gemini, and OpenAI APIs preferred
- Experience with vector databases and embeddings preferred
- Exposure to MLOps workflows and ML lifecycle testing preferred
- Knowledge of performance testing and security testing preferred
- Experience validating AI governance/compliance requirements preferred
- Experience with contract testing, chaos testing, or resilience engineering preferred
- Familiarity with accessibility testing and cross-browser/device testing preferred
- Exposure to synthetic monitoring and production observability preferred
- Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred
- Equivalent hands-on experience and demonstrable projects highly valued
Benefits:
- Work-from-home, hybrid, or work-from-the-office environment scenarios
- OSDE 310 Health Care
- Monthly Salary Reviews
- Tuition Reimbursement
- Internet Reimbursement
- In-company English Lessons
- LinkedIn Learning
- Referral Program Bonus
- F5 extra days off
- Annual Bonus
- Opportunity to work on global projects with a multicultural team
- Learning and development opportunities
- 20+ affinity groups for networking and connection
- Mentoring and career growth opportunities
Similar Jobs
Senior Software Development Engineer in Test
Senior test engineer automating Akamai Cloud IP addressing platform quality. Building frameworks and CI/CD pipelines with Python, REST API testing, and cloud technologies.
Senior QA Automation Engineer
Senior QA Automation Engineer owning automated testing for a Go, Kafka and Google Cloud LiveOps platform. Ensuring resilience, performance and quality across a global video-game commerce company.
Test Engineer
Engenheiro de Testes liderando QA, automação e estratégias BDD/TDD. Gerenciamento de projetos técnicos e equipes distribuídas em múltiplos projetos.
SDET
SDET diseñando automatización para plataformas SaaS, aplicaciones web y APIs. Integrando agentes de IA y herramientas LLM en pruebas CI/CD para Dresden Partners.
Senior Test Engineer – Demo
Senior Test Engineer designing manual and automated testing strategies for Inabia’s telecommunications, technology, and healthcare solutions. Ensuring software quality through CI/CD, regression, integration, and performance testing.
Staff Software Engineer in Test – Mobile
Staff Software Engineer in Test shaping mobile and backend quality at InPost, Europe’s out-of-home parcel delivery platform. Defining reliability strategy, release gates, and AI governance across engineering teams.
Senior QA Engineer – Automation
Senior Automation QA Engineer building reliable Java automation frameworks for SOFTSWISS’s online casino platform. Improving UI, API, distributed-system testing and CI quality.
QA Automation Engineer – Banking
QA Automation Engineer automating banking software tests for Qualysoft, an enterprise IT consulting and digital transformation company. Building API, integration, database, UI, and CI/CD test coverage.
Software Development Engineer in Test – Agentic Automation
SDET building automated tests, evals, and performance frameworks for CrowdStrike’s AI-native cybersecurity platform. Improving reliability of distributed cloud services and agentic workflows.
Senior AEM QA Automation Engineer
Senior AEM QA Automation Engineer designing Playwright and BrowserStack test automation. Supporting Nortal’s enterprise digital transformation through AEM on-premises-to-cloud migration.
Lead AEM QA Automation Engineer
Lead QA automation for Nortal’s AEM on-premises-to-cloud migration. Nortal delivers digital transformation, data, and AI solutions for global enterprises and public-sector organizations.
Senior QA Automation Engineer
Senior QA Automation Engineer building automated API, web, and mobile test frameworks for CodeRoad’s technology delivery clients. Integrating QA into CI/CD pipelines and mentoring automation engineers.
Senior Software Development Engineer in Test – Digital Banking
Senior SDET automating web, mobile, API, and integration testing for Aequilibrium’s Canadian credit union banking modernization projects. Validating secure, reliable member banking experiences.
Senior Engineer, Test Engineering
Senior Test Engineer sustaining ATE, wafer probing, and final testing for SiTime’s MEMS precision-timing semiconductor products. Supporting production qualification, debugging, and process improvements.
Automated Test Engineer, Manager
Automated Test Engineering Manager leading Playwright-based testing for ICF’s federal cloud applications. Driving CI/CD quality gates, data validation, and release readiness.
SDET 3
Senior SDET testing Bidgely’s AI-powered utility energy products, dashboards, APIs, and GenAI agents. Building automation and evaluation frameworks to ensure release quality.
Senior QA Automation Engineer
Senior QA Automation Engineer maintaining and evolving automated test suites for Nagarro's digital product engineering services. Designing frameworks and AI-driven testing workflows across complex software integrations.
Automation Architect
Automation Architect leading Python, AI, and agentic security automation at Lilly, a life-sciences company developing life-changing medicines. Managing engineers and shaping cloud and application security tooling.
Senior Software Development Engineer in Test, QA - AI Systems
Senior SDET building evaluation infrastructure and testing autonomous AI agents. Nexxa applies AI systems to heavy industries, manufacturing, infrastructure, logistics, and legacy environments.
Senior QA Automation Engineer, AI-Focused
Senior QA Automation Engineer driving automated web, API, and backend testing for WorkMotion’s global workforce platform. Applying AI tools to improve QA efficiency and release quality.