Senior QA Engineer – AI-Native Platform
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior QA Engineer building test infrastructure for AI-Native platform at a global OSINT company. Focused on testing non-deterministic AI agent outputs and data isolation.
Responsibilities:
- Build and maintain AI evaluation framework
- Test RAG pipeline quality and retrieval relevance
- Design testing approaches for non-deterministic LLM outputs
- Build multi-tenant isolation test suite, including AI layer isolation
- Write cross-service integration and contract tests
- Test Temporal workflows, event-driven flows, and YAML DSL validation
- Build and maintain automated regression suites
Requirements:
- 3+ years in QA automation / SDET
- Hands-on experience testing AI/ML systems
- Strong Python (pytest, async testing)
- Experience building test infrastructure from scratch
- Experience testing distributed or event-driven systems
- Experience testing multi-tenant applications
- Autonomous work style
- English – B1 level or higher
- RAG / vector search quality evaluation (Nice To Haves)
- LangGraph, LangChain, or similar agent frameworks (Nice To Haves)
- Go test automation (Nice To Haves)
- BentoML / MLflow or equivalent MLOps tooling (Nice To Haves)
- Security testing: prompt injection, context leakage (Nice To Haves)
- Performance / load testing for ML inference endpoints (Nice To Haves)
- Data pipeline testing (Delta Lake, Great Expectations, Dagster) (Nice To Haves)
- Russian – advanced level or higher (Nice To Haves)
Benefits:
- Remote-first setup: work from anywhere in the world (except Russia and Belarus)
- Work on a greenfield, technically challenging product (distributed system at scale)
- Direct collaboration with senior technical stakeholders (engineering leadership, product)
- Ownership and impact: influence QA strategy, tooling, and quality gates from the early stage of the product
- Fast iteration environment: work closely with engineering to improve reliability and release confidence
Similar Jobs
Doctor QA Specialist
Doctor QA Specialist ensuring quality patient communication for telehealth brands. Focused on quality assurance, compliance, and communication excellence across provider messaging.
Property Claims Quality Assurance Intern
Claims Quality Intern at California Fair Plan conducting research and vendor selection for claims review software. Collaborating with departments to enhance efficiency and accuracy in claims auditing.
Property Claims Quality Assurance Supervisor
Claims Quality Assurance Supervisor leading QA teams for property claims accuracy and compliance. Overseeing communications with the Department of Insurance and supporting Claims Examiners.
QA Analyst
QA Analyst ensuring software quality for innovative banking solutions at Stefanini. Managing testing processes and supervising external providers to uphold standards and documentation.
Quality Assurance Analyst – AI Data Annotation
QA Analyst responsible for quality auditing of AI/ML data annotation at Expedock. Collaborating with teams to improve data accuracy and provide feedback to annotators.
QA Engineer – Manual and Automation
Senior Quality Engineer responsible for designing automated test coverage for integration workflows. This role requires strong experience in Playwright and significant QA involvement in the healthcare sector.
Junior QA Engineer
Junior QA Engineer supporting testing of mobile applications for Virtusan. Handling functional, regression, and usability testing for iOS and Android devices.
Senior Supplier Quality Engineer
Supplier Quality Engineer responsible for overseeing supplier performance and compliance. Analyzing processes, ensuring quality standards, and collaborating with internal and external stakeholders.
QA Engineer
Manual QA Engineer ensuring product stability for a fintech SaaS company. Collaborating on product testing across frontend and backend systems to enhance quality.
Quality Engineering Technical Lead
Technical Lead in Quality Engineering, overseeing test strategy and automation in a global game services provider. Collaborate with teams to ensure high-quality standards across platforms.
Pentester
Pentester simulating attacks to identify vulnerabilities across networks and cloud infrastructure at Cyber Defence. Collaborating with teams to enhance client resilience and compliance.
Web QA Engineer
Web QA Engineer ensuring quality of a web-based photo editing product combining frontend and backend technologies. Validate UI and APIs with a focus on automation and efficiency.
Mobile Tester, iOS
Tester de aplicaciones móviles en Capgemini, desarrollando y ejecutando planes de prueba para sistemas Web/Digital. Validando integraciones y realizando pruebas transaccionales en plataformas iOS y Android.
Senior Quality Assurance Engineer
Senior QA Engineer responsible for QA processes and automation in IT projects for a fast-growing company. Collaborating with technical teams to improve product quality in various applications.
Quality Assurance Analyst – Co-op
QA Analyst Co-op for Real Estate Webmasters focused on software testing and product quality. Collaborating with a product team to ensure high standards in web applications.
Global Quality Assurance – Work-Study
Quality Assurance Intern for UPSA, participating in Global Quality activities and audits. Supporting compliance and documentation processes in the pharmaceutical field.
Quality Assurance Analyst
QA Analyst supporting Case Management Modernization Program at GDIT. Ensuring quality assurance for cloud-based data solutions across US federal courts with compliance testing.
Senior Quality Engineer
Sr. Quality Engineer I responsible for developing test automation frameworks at TrueML, a fintech company enhancing customer experiences for distressed borrowers.
Automation Tester
Automation Tester developing automated test scripts at Mind Computing. Ensuring software quality and functionality through analysis, execution, and collaboration.
Functional Tester
Functional Tester ensuring software quality and defect-free applications. Collaborating closely with developers and QA analysts, optimizing testing processes with latest tools and techniques.