Software Development Engineer in Test – Agentic Automation
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
SDET building automated tests, evals, and performance frameworks for CrowdStrike’s AI-native cybersecurity platform. Improving reliability of distributed cloud services and agentic workflows.
Responsibilities:
- Design, develop and maintain automated test suites and frameworks for distributed, cloud-based systems
- Maintain and extend agentic test generation and analysis tooling built into major testing frameworks
- Build AI-powered workflows and agent skill files to automate manual test toil, including test data setup, failure triage and release validation
- Maintain eval suites, golden datasets and scoring pipelines for AI agents, scoring correctness, tool use, trajectory, faithfulness, safety, latency and cost
- Enforce CI eval gates so changes to prompts, models, tools or orchestration are measured before release
- Feed production failures back into evaluation sets
- Detect and mitigate unreliable AI output, including hallucinated assertions, flaky or redundant tests, incorrect failure triage and judge bias
- Design, run and analyze performance, load and stress tests to find bottlenecks and validate scalability across distributed services
- Validate integration of platform systems, services, data and applications, as well as customer-facing features end-to-end
- Review product, system and agent designs for testability, evaluability, flaws and improvements
- Share knowledge and contribute to the team's quality and evaluation standards
Requirements:
- Degree in Computer Science or equivalent experience (OOP, data structures, algorithms, distributed systems)
- 5+ years in software quality, including 3+ years of hands-on SDET or test automation development
- Strong programming skills in at least one of Java, Node.js or Python, with Python preferred for eval and agent work
- Proven experience designing, building and maintaining test frameworks integrated with CI/CD pipelines (e.g., Jenkins, GitLab CI/CD)
- Experience testing web services and APIs (REST, gRPC, JSON)
- Hands-on experience with performance and load testing using tools such as Locust or k6, including designing realistic workloads and interpreting results (latency percentiles, throughput, error rates)
- A skeptical, verification-first mindset toward AI output, with a clear sense of when to use deterministic assertions and when to use probabilistic evaluation
- Proven experience utilizing AI technologies to enhance decision-making, streamline workflows and processes, improve efficiency and drive business outcomes
- Expected to maintain legal entitlement to work in Canada
- Subject to background checks, including criminal record, credit, and/or reference checks
- Bonus: End-to-end eval-driven development experience, including release gating, trajectory and tool-use evals, and calibrating LLM-as-judge against human review
- Bonus: Prompt engineering and context or tool design for agents
- Bonus: Integrating performance tests into CI/CD and using observability tools (e.g., Grafana, Prometheus, CloudWatch)
- Bonus: Background in cybersecurity, threat intelligence, malware analysis or sandboxing/detonation environments
Benefits:
- Market leader in compensation and equity awards
- Comprehensive physical and mental wellness programs
- Competitive vacation and holidays for recharge
- Paid parental and adoption leaves
- Professional development opportunities for all employees regardless of level or role
- Employee Networks, geographic neighborhood groups, and volunteer opportunities to build connections
- Vibrant office culture with world class amenities
- Variable/incentive compensation
- Equity
- Equal pay for equal work
Similar Jobs
Senior QA Engineer – Automation
Senior Automation QA Engineer building reliable Java automation frameworks for SOFTSWISS’s online casino platform. Improving UI, API, distributed-system testing and CI quality.
QA Automation Engineer – Banking
QA Automation Engineer automating banking software tests for Qualysoft, an enterprise IT consulting and digital transformation company. Building API, integration, database, UI, and CI/CD test coverage.
Senior AEM QA Automation Engineer
Senior AEM QA Automation Engineer designing Playwright and BrowserStack test automation. Supporting Nortal’s enterprise digital transformation through AEM on-premises-to-cloud migration.
Lead AEM QA Automation Engineer
Lead QA automation for Nortal’s AEM on-premises-to-cloud migration. Nortal delivers digital transformation, data, and AI solutions for global enterprises and public-sector organizations.
Senior QA Automation Engineer
Senior QA Automation Engineer building automated API, web, and mobile test frameworks for CodeRoad’s technology delivery clients. Integrating QA into CI/CD pipelines and mentoring automation engineers.
Senior Software Development Engineer in Test – Digital Banking
Senior SDET automating web, mobile, API, and integration testing for Aequilibrium’s Canadian credit union banking modernization projects. Validating secure, reliable member banking experiences.
Senior Engineer, Test Engineering
Senior Test Engineer sustaining ATE, wafer probing, and final testing for SiTime’s MEMS precision-timing semiconductor products. Supporting production qualification, debugging, and process improvements.
Automated Test Engineer, Manager
Automated Test Engineering Manager leading Playwright-based testing for ICF’s federal cloud applications. Driving CI/CD quality gates, data validation, and release readiness.
SDET 3
Senior SDET testing Bidgely’s AI-powered utility energy products, dashboards, APIs, and GenAI agents. Building automation and evaluation frameworks to ensure release quality.
Senior QA Automation Engineer
Senior QA Automation Engineer maintaining and evolving automated test suites for Nagarro's digital product engineering services. Designing frameworks and AI-driven testing workflows across complex software integrations.
Automation Architect
Automation Architect leading Python, AI, and agentic security automation at Lilly, a life-sciences company developing life-changing medicines. Managing engineers and shaping cloud and application security tooling.
Senior Software Development Engineer in Test, QA - AI Systems
Senior SDET building evaluation infrastructure and testing autonomous AI agents. Nexxa applies AI systems to heavy industries, manufacturing, infrastructure, logistics, and legacy environments.
Senior QA Automation Engineer, AI-Focused
Senior QA Automation Engineer driving automated web, API, and backend testing for WorkMotion’s global workforce platform. Applying AI tools to improve QA efficiency and release quality.
QA Engineer – Automation
QA Engineer automating testing frameworks and suites for TransUnion’s software and data services. Debugging products, identifying defects, and supporting reliable system integration.
Software Test Automation Engineer
Software Test Automation Engineer building scalable automated tests for web, API, and backend systems. Defining CI/CD quality strategies and partnering on DevSecOps testing practices.
QA Automation Engineer, Temporary Position
QA Automation Engineer improving CTERA’s cloud storage product quality through Java API automation, Playwright UI testing, and QA infrastructure. Maintaining Jenkins suites and the QA Cloud Agent.
Senior QA/Automation Test Engineer
Senior QA/Automation Test Engineer leading automation, testing strategy, and DevSecOps quality for Guidehouse’s enterprise software platforms. Driving reliable releases and mentoring QA engineers.
Lead Automation Architect
Lead Automation Architect designing secure Ansible and Jenkins automation for Peraton’s Federal Financial infrastructure. Driving scalable solutions across air-gapped and AWS GovCloud environments.
Senior Software Development Engineer in Test
Senior SDET automating testing, deployment, and validation for Akamai's Kubernetes-based cloud platform. Building reliable test frameworks for distributed cloud and edge services.
Senior Test Engineer – T24, Mortgage/Lending
Senior Test Engineer testing Temenos T24 core banking applications. Validating mortgage, lending, API, workflow, and integration quality in Agile DevOps environments.