Senior LLM Systems Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Senior LLM Systems Engineer improving accuracy and resilience of LLM-driven oracle automation systems. Collaborating on evaluations, tooling, and debugging for production operations.
Responsibilities:
- We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack.
- Focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes.
- Build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions.
- Improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time.
- Design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs.
- Build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures.
- Help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction.
Requirements:
- 3+ years of professional software engineering experience in Python, TypeScript, or similar production languages.
- Hands-on experience building production systems that use LLMs, agents, retrieval, structured outputs, or model-powered workflows.
- Experience designing evaluations, test datasets, regression checks, quality metrics, or manual review loops for AI systems.
- Strong debugging ability across APIs, databases, queues, logs, model outputs, and external data sources.
- Practical understanding of prompt engineering, tool calling, structured output validation, retrieval, and common LLM failure modes.
- Ability to reason carefully about correctness in uncertain or adversarial environments.
- High agency, strong ownership, and clear written communication.
Benefits:
- Pay packages include competitive salaries & meaningful long term equity participation
- Salaries for this role range from $100-200k (USD)
- Will pay in stablecoins or fiat
- Philosophies for a culture that show we care: Take vacation when you need it, family care, training and development (just to name a few)
- 100% remote, which means we encourage you to create the work environment that you thrive in
- At least two team wide offsites a year
Similar Jobs
Systems Architect, Java, Springboot
Systems Architect responsible for overall system architecture of global B2B applications at CI&T. Engaging with clients and defining architectural standards across teams.
Business Systems Analyst, Acumatica ERP
Business Systems Analyst managing CRM and ERP systems for Alimentiv. Focusing on Acumatica and Salesforce to enhance Finance and Business Development operations.
Systems Analyst – Project-focused
Systems Analyst focusing on projects to improve IT solutions and operational efficiency at PUCRS. Collaborating with business analysts and development teams to achieve strategic objectives.
BSA – Business Systems Analyst Specialist
Specialist BSA driving integration between SAP and fiscal solutions in a leading fintech. Focus on compliance and operational efficiency in financial technology environments.
Systems Analyst III
Systems Analyst at The Standard Insurance focusing on cloud-based solutions and team collaboration. Involving incident response and application lifecycle coordination in a remote environment.
OutSystems Engineer
Senior / Lead OutSystems Engineer focusing on intuitive digital quote journeys for HNW customers. Engaging in hands-on application development and systems integration within a strategic insurance initiative.
Senior System Engineer – Cloud Native
Senior System Engineer at Adfinis AG, leading cloud-native projects and ensuring quality for clients in a remote-first environment.
Operations Systems Analyst
Operations Systems Analyst improving digital workspace processes at Outsource Access. Spearheading evaluations, engineering logic solutions, and creating detailed playbooks.
Systems Analyst, Level 2
Systems Analyst providing technical assistance and managing escalated support tickets for a Managed Service Provider. Collaborating with a talented team to enhance core infrastructure and support clients remotely.
Systems Integration Analyst – Junior
Junior Systems Integration Analyst at Stefanini involved in API and system integrations. Responsible for client interaction and problem resolution in a collaborative environment.
Senior Associate, Energy Systems Analyst
Senior Associate, Energy Systems Analyst at Converge Strategies, LLC providing technical analysis and recommendations for energy resilience projects. Engaging stakeholders and developing solutions for clean energy technologies.
Associate IT Systems Engineer
Associate IT Systems Engineer monitoring, configuring, and troubleshooting technology solutions at Florida Blue. Collaborating on infrastructure improvements and maintaining system performance and security.
Senior Systems Engineer
Develop architecture and troubleshoot server infrastructure for healthcare systems. Responsible for data center operations and technical leadership in cloud service and server management.
Senior Systems Engineer, Internal Support
Senior Engineer supporting internal systems at Atlas Technica, working with Azure AVD, Endpoint infrastructure, and more. Collaborate with the infrastructure team for excellent career development.
Systems Analyst, Mid-level
Backend Developer at C&A developing and maintaining scalable applications and APIs. Collaborating with teams to ensure system performance and reliability.
Senior Systems Engineer
System Engineer, Sr. on the IST/System Engineering Team generating monitoring and observability recommendations. Collaborating to enhance reliability and improve technology services provided to veterans.
Systems Engineer
Systems Engineer focusing on supporting system operations and lifecycle management. Collaborating with teams to implement technical solutions in a dynamic environment.
Analista de Sistemas Mainframe
Analyze and develop projects of mainframe systems in innovative technology. Collaborate on agile methodologies within a dynamic work environment.
Mainframe Systems Analyst, Mid-level (Pl)
Analista de Sistemas Mainframe Pl na Minsait, envolvendo análise, desenvolvimento e testes de sistemas. Trabalhando com linguagens Mainframe e metodologias ágeis.
Presales, Systems Engineer
Pre-Sales Systems Engineer providing technical sales support for complex service provider networks. Responsible for managing the technical close and ensuring customer satisfaction during the sales process.