PhD Research Fellow – LLMs, Agents, Multi-Agent Systems, AI Reliability
Posted 6hrs ago
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Pesquisador bolsista desenvolvendo pipelines, métricas e evals para agentes LLM e multiagentes. Projeto de inovação do Instituto Itaú de Ciência, Tecnologia e Inovação com Inova Talentos.
Responsibilities:
- Map agent execution flows and identify key points for log collection and monitoring
- Develop Python pipelines for collecting, storing, and processing execution logs
- Define and implement metrics to evaluate agent performance, quality, and consistency
- Build automated evaluation routines (evals) for different scenarios and use cases
- Develop analyses and reports on agent behavior, identifying failures and opportunities for improvement
- Support the implementation of dashboards or monitoring tools
- Document the architecture, metrics, and results obtained throughout the project
- Work on an innovation project at the Itaú Institute of Science, Technology and Innovation in partnership with Inova Talentos
Requirements:
- Program: TECH 4
- PhD
- Degree in Data Science, Statistics, Computer Engineering, Computer Science, or a related field
- Python programming
- Ability to organize and reproduce a software project
- Experience with LLMs beyond casual use
- Knowledge of multi-agent systems and how to orchestrate them
- Ability to extract, validate, and compare information in documents
- Ability to define and interpret evaluation metrics
- Experience instrumenting systems with logs, traces, and metrics
- Ability to explain failures from a technical perspective
- Ability to read articles and research papers, turn ideas into experiments, and clearly write and present results
- Availability to work 40 hours per week
- Previous applied research experience with AI, ML, or LLMs
- Ability to build and evaluate end-to-end observable systems
- Ability to transform an operational workflow into an agent architecture
- Clear scientific writing and strong verbal communication skills
- Autonomy and experimental maturity
Benefits:
- Fellowship stipend: BRL 11,000.00
- Work arrangement: Remote
- Commitment: 40 hours per week
- Duration: 12 months
















