LLM Evaluation Engineer
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Develop evaluation systems for AI behavior in enterprise environments using LLMs and real-time data. Collaborate on guardrails and enforcement strategies for AI compliance.
Responsibilities:
- Build the evaluation layer in the ThirdLaw platform for LLM prompts and responses
- Design and tune guardrails, classifiers, and semantic judgment systems in real-time
- Implement evaluation strategies with semantic similarity, foundation model scoring, and rule-based systems
- Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking)
- Prototype, tune, and productize small language models for classification, labeling, or scoring
- Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage
- Build tools to observe, debug, and improve evaluator performance across data distributions
- Define abstractions for reusable evaluation components that can scale across use cases
Requirements:
- 7+ years of experience in ML systems or AI engineering roles
- At least 1–2 years working directly with LLMs, NLP pipelines, or semantic search
- Deep understanding of foundation models (e.g. OpenAI, Claude, Mistral, Llama) and APIs
- Hands-on experience with vector search (e.g. FAISS, Qdrant, Weaviate) and embeddings pipelines
- Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules
- Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow
- Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production
Benefits:
- Generous benefits
- Market cash compensation
- Above-market equity
- Well-designed benefits
Similar Jobs
Power Systems Study Engineer – III
Power Systems Study Engineer III at DLB Associates working on power system studies for mission critical projects. Collaborating within the Engineering Team to optimize performance.
Cloud Operation Engineer
Cloud Operation Engineer responsible for designing, implementing, and maintaining server infrastructure at CSG Systems, Inc. Ensuring stability and security for enterprise systems in both on-premise and cloud environments.
Senior I&C Engineer – Energy
Senior I&C Engineer Energy designing and maintaining systems for nuclear facilities. Delivering technical documentation and ensuring compliance with nuclear safety regulations.
DFT Engineer
Senior DFT Engineer leading end-to-end DFT strategy for complex SoCs at Niobium Microsystems. Overseeing architecture, integration, and validation processes within the team.
External Lighting Engineer
External Lighting/Street Lighting Engineer at Fixatex performing installations and maintenance of street and external lighting across London. Ensuring compliance with safety standards and delivering quality service in social housing and local authority sectors.
Thermal Hydraulics Engineer 3
Thermal Hydraulics Engineer developing thermal systems analysis for NuScale Power’s microreactor technology. Collaborating on modeling, analysis, and software selection within a dynamic team.
Interconnection Engineer
Interconnection Engineer managing engineering items for clean energy solutions at WECTEC Staffing Services. Collaborating with corporate teams and Transmission Owners to ensure interconnection agreements are met.
Sandbox Agentic Engineer
AI Agent Engineer developing LLM-based AI solutions for cybersecurity at CrowdStrike. Leading AI agent development and optimizing performance with modern frameworks and technologies.
Critical Power Field Technician, UPS Engineer
Critical Power Technician maintaining critical power systems in Nashville, TN. Role involves technical support, customer service, and mentoring crew members in UPS systems.
Forward Deployed Engineer
Forward Deployed Engineer at Truelogic Software integrating AI solutions with clients. Collaborating on impactful projects to enhance digital transformations in businesses.
Chief Engineer – Container Vessel
Chief Engineer leading Engine room operations for Columbia Shipmanagement's fleet, ensuring equipment functionality and safety. Requires strong experience with specific engine types and good communication skills.
Technical Engineer
Technical Engineer for the Wendy's engineering team focusing on mobile development and problem solving.
Staff IT/MSP Engineer
Staff IT Engineer responsible for technical health of MSP's client base, designing deployment standards. Bridging business goals with technical execution, reporting to CTO or Director of Engineering.
Customer Delivery Engineer
Customer Delivery Engineer at Tiger Data implementing PostgreSQL projects and optimizing customer experiences with AI tools during migrations and production rollouts.
Senior Identity Access Management Engineer
Senior IAM Engineer improving Identity and Access Management processes at U.S. FinTech. Responsible for automation, documentation and overseeing AWS IAM solutions for security compliance.
Senior Software V&V Engineer
Senior Software V&V Engineer ensuring thorough verification and validation of software in cancer diagnostics. Collaborate with cross-functional teams for bioinformatics pipeline testing and regulatory compliance.
Project Engineer
Project Engineer working on maintenance and development of automation systems. Involves offshore operations and all stages of automation lifecycle support in Rio de Janeiro.
Project Engineer
Project Engineer maintaining automation systems for customer facilities including offshore operations. Engaging in all stages of systems lifecycle from requirements gathering to startup.
Systems Monitoring Engineer
Systems & Monitoring Engineer architecting monitoring stack for Hedera’s DLT ecosystem. Blending NOC practices with Web3 challenges to ensure uptime and compliance.
IT Data Center Engineer, KVM, FortiGate
IT Data Center Engineer managing critical infrastructure for a U.S.-based mobile operator. Focusing on network management, system optimization, and incident resolution in a remote setting.