ML Engineer, Retrieval – Grounded Generation
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Quality engineer building automated testing and release gates for DEFCON AI’s AI-enabled decision-support systems. Measuring model accuracy, performance, and release evidence.
Responsibilities:
- Own the system test plan for functional, integration, performance, and load testing
- Translate product acceptance criteria into executable tests
- Review acceptance criteria before development and identify ambiguity or unverifiable requirements
- Hold engineering, platform, and partner teams accountable to acceptance criteria and release evidence
- Collaborate daily with engineering while independently reporting test results
- Plan and coordinate customer user acceptance testing and track findings to resolution
- Build and maintain automated test suites integrated into CI/CD as release gates
- Establish automated coverage standards and report against them
- Build performance and load testing for expected user loads
- Maintain regression coverage across releases
- Build labeled evaluation sets with subject-matter experts
- Design and run sampled audits of automated decisions
- Measure system misses, correct outcomes, and user overrides
- Document measures, data, and release evidence
- Produce evidence independently of model builders
- Maintain version inventories across builds, models, configurations, and thresholds
- Produce test reports, evaluation records, and rollback criteria for each release
- Verify test environments meaningfully match production
- Hold the release gate so no release ships without required evidence
Requirements:
- 5+ years in software quality engineering, test automation, or software engineering with substantial dedicated test ownership
- Clear point of view on how testing should be done and judgment about worthwhile practices
- Strong coding ability in a test-automation language and framework, such as Python with pytest or JavaScript/TypeScript with Playwright, Cypress, or Selenium
- Ability to read and reason about application code under test
- Routine use of AI-assisted development, with judgment about verification of AI output
- Experience building automated test suites and integrating them into CI/CD as release gates
- Experience owning a test plan through review and acceptance
- Experience testing against requirements rather than implementation, working with product owners or end users
- Experience with performance and load testing
- Working statistical literacy, including sampling design and measurement confidence
- Experience measuring what a system misses, not only what it gets right
- Willingness to report unfavorable results independently
- Ability to gate work owned by people outside your reporting line
- Clear, audit-ready written documentation
- Willingness to hold and explain a release gate
- US Citizenship required
- Active US Secret clearance required from the start, with favorable investigation and CAC eligibility
- Willingness to travel up to 25% to customer sites, DEFCON AI HQ, and vendor facilities
- Preferred: active Top Secret clearance; government DevSecOps, RMF, or ATO experience; formal accreditation/audit evidence; contract/API testing; test data management; test infrastructure; AI-assisted testing; JMeter, k6, Locust, or comparable tools; ML/model-output evaluation including groundedness and citation checking; model cards or NIST AI RMF; Section 508 and WCAG verification
Benefits:
- Competitive salary, bonus, and equity package
- 100% employer-paid comprehensive health insurance including medical, dental, and vision for you and your family
- Unlimited PTO, with manager/management approval
- Flexible work environment where you manage your work day
- 14 weeks of fully paid parental leave
- Career track opportunity with potential for rapid advancement with strong performance
- Opportunities for professional development and continued learning
- Optional 401K, FSA, and equity incentives
- Mental health benefits through Tara Mind
- Cost-effective GLP-1 solutions through Crux
Similar Jobs
Senior Machine Learning Engineer
Senior ML Engineer building recommendation, ranking, notification, and LLM systems for Coinbase’s crypto and financial markets platform. Owning production ML initiatives that improve customer engagement and trading activity.
Machine Learning Engineer – Risk Scoring, Model Monitoring
Machine Learning Engineer building explainable risk-scoring and monitoring models for Chiri’s healthcare and pharmacy-finance platform. Developing reliable gradient-boosted pipelines with drift detection and controlled deployment.
Software Engineer 2, Machine Learning
Machine learning engineer developing statistical, generative AI, and LLM-powered agent solutions. Deploying scalable ML systems and agent pipelines for an education-focused startup.
Machine Learning Operations Engineer – MLOps
MLOps Engineer deploying agentic AI pipelines and multi-cloud infrastructure. Supporting CodeRoad’s technology execution projects for high-growth U.S.-based companies.
Senior Machine Learning Engineer – LLMs, Agentic Workflows
Senior Machine Learning Engineer designing autonomous agents and multi-agent workflows for Factored. Building production AI systems that help U.S. companies scale teams with LATAM talent.
Senior Machine Learning Engineer – RAGs
Senior Machine Learning Engineer developing production RAG models and GenAI products for Factored’s clients. Building AI infrastructure and deploying LLM solutions with LATAM-based talent.
Senior Machine Learning Engineer – Recommender Systems
Senior Machine Learning Engineer designing large-scale recommender systems for Factored, which helps U.S. companies build AI, ML, and data teams with LATAM talent. Owning pipelines from experimentation to production.
Senior MLOps Engineer
Senior MLOps Engineer building scalable ML infrastructure and deployment pipelines. Factored connects U.S. companies with top LATAM AI, ML, and data talent.
Senior ML Ops Engineer
Senior Engineer deploying NVIDIA AI infrastructure, distributed workloads, and MLOps solutions for NVIDIA cloud partners. Supporting production AI deployments across DGX Cloud, NCP, and major CSP environments.
Senior AI/ML Engineer
Senior AI/ML Engineer building reliable LLM agents, evaluation systems, and APIs for Recare’s digital healthcare coordination platform. Protecting health data while guiding production AI strategy.
AI/ML Engineer II
AI/ML Engineer II using Python for TEKsystems’ cloud, data, and customer experience solutions. Supporting enterprise transformation through technology, strategy, design, execution, and operations.
Software Engineering PMTS – ML Engineer
ML Engineer building real-time telemetry pipelines and feature platforms for Salesforce’s AI CRM. Designing streaming systems that power personalization and real-time inference.
Machine Learning Scientist – Wildfire
Machine Learning Wildfire Scientist owning fire behavior modeling for EmberWorks Sensing's ORBIT airborne thermal sensing platform. Building validated production models from wildfire observations and geospatial data.
Machine Learning Wildfire Scientist
Machine Learning Wildfire Scientist developing fire behavior models for EmberWorks' ORBIT airborne thermal sensing system. Building production pipelines and validating wildfire intelligence for active-fire operations.
Senior/Lead Machine Learning Engineer
Machine Learning Engineer building LLM-heavy data intelligence pipelines for Mimica’s AI process intelligence platform. Improving evaluation, anonymization, efficiency, and production ML systems.
Machine Learning Expert
Machine Learning Expert designing difficult inference, evaluation, and systems tasks for Mercor’s frontier AI lab projects. Contributing research ideas to train and enhance advanced AI systems.
Machine Learning Expert
Machine Learning Expert designing challenging inference, evaluation, and systems tasks for Mercor’s leading AI lab partners. Contributing research ideas to train and enhance frontier AI models.
Machine Learning Expert
Machine Learning Expert designing difficult inference, evaluation, and systems tasks for Mercor’s frontier AI research projects. Contributing ideas that challenge advanced AI models.
Senior ML Engineer
Machine Learning Engineer building autonomous agents, voice AI, and probabilistic models for AssessFirst’s HR Tech recruitment platform. Deploying reliable, fair systems with Product, Science, and MLOps teams.
ML Engineer – Agents, LLMs
ML Engineer improving post-training, evaluation, and tool-using agents for KIEFER’s sovereign AI infrastructure and enterprise products. Enhancing long-context reliability and production inference.