Research Scientist, Benchmarks & Evals
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Research Scientist building benchmarks and evaluations for Kled’s opt-in human data network. Measuring how its data improves frontier AI image, video, and document models.
Responsibilities:
- Design and ship public benchmarks for image, video and document models
- Build private test sets from real, opt-in data that no model has trained on
- Run data-value experiments by training models with and without Kled data and measuring changes
- Design human evaluation studies and automatic metrics
- Publish leaderboards and reports trusted by researchers at frontier labs
- Identify model shortcomings and advise what data to collect next
- Work directly with founders and lab buyers to turn results into data deals
Requirements:
- 3+ years ML research or research engineering experience
- A benchmark or evaluation you built that other people actually used
- Strong hands-on experience training and fine-tuning image or video models (PyTorch)
- Deep understanding of experimental design and statistics (controls, ablations, human studies)
- Experience running experiments end to end, from GPUs to final report
- Clear writing for both researchers and buyers
- Publications at NeurIPS, ICML, ICLR or CVPR (datasets and benchmarks especially) (bonus)
- Experience on an evaluation or data team at an AI lab (bonus)
- Experience selling or licensing data to AI labs (bonus)
- Experience with data valuation or data attribution research (bonus)
- Python / PyTorch
- Open-weight image and video models
- Cloud GPUs
- PostgreSQL (Supabase)
- S3 storage
Benefits:
- $350,000 – $750,000 equity
- 0.25% – 1% equity
Similar Jobs
Senior Principal Scientist – US Regulatory Innovation
Senior Principal Scientist shaping US regulatory strategy for Dechra’s veterinary drugs and biologics. Guiding FDA/USDA submissions, product approvals, and regulatory innovation.
Senior Principal Scientist, Regulatory Liaison
Senior Principal Scientist leading global regulatory strategies for Merck’s immunology drug and biologic programs. Guiding submissions, health-authority interactions, labeling, and product approvals worldwide.
PhD Research Fellow – Computing, Engineering, Applied Mathematics
Bolsista doutor pesquisando GraphRAG, RAG e agentes de IA para o projeto de inovação do Instituto Itaú. Desenvolvendo protótipos semânticos e mecanismos de avaliação e segurança.
PhD Research Fellow, Computer Science, Engineering, Applied Mathematics
Bolsista doutor pesquisando GraphRAG, RAG e agentes inteligentes. Desenvolvendo protótipos e mecanismos de avaliação para projeto de inovação do Instituto Itaú de Ciência, Tecnologia e Inovação.
Research Fellow – Workforce Intelligence
Research Fellow analyzing construction labor markets, compensation, and workforce data. Supporting workforce intelligence reports, dashboards, methodologies, and operational planning systems.
Senior Researcher
Senior Researcher leading mixed-methods research and mentoring teams at Ad Hoc, a technology company building digital services for government agencies. Influencing research strategy, stakeholder decisions, and product outcomes.
Master’s Research Fellow – Computational Memory, Cognitive CRM, Intelligent Agents
Bolsista mestre pesquisando memória computacional, RAG temporal e agentes inteligentes no Instituto Itaú de Ciência, Tecnologia e Inovação. Desenvolvendo protótipos e métricas para IA generativa fundamentada.
Research Assistant – Student Worker
Student Research Assistant supporting National University research on women of color Marriage and Family Therapists. Assisting with qualitative data collection, coding, documentation, and scholarly materials.
Senior Research Scientist – Architectures Research
Senior Research Scientist developing efficient model architectures for Nebius’s AI cloud platform. Advancing attention, memory, reasoning, and inference methods through frontier applied research.
Research Assistant – Senior, Early Childhood
Research Assistant supporting Child Trends’ early childhood social science research projects. Collecting, analyzing, and organizing data for research, evaluation, and policy projects.
Senior Research Scientist, Executive Engagement & Insights
Senior Research Scientist leading applied research for Syndio’s pay governance and compensation intelligence platform. Combining proprietary pay data with labor-market data to inform executives, products, and customers.
LLM Research Scientist – Pre-training, Computer Vision, Adversarial Robustness
LLM research scientist training and improving vision and language models for Mercor’s AI lab and enterprise partners. Focusing on adversarial robustness, computer vision, generative modeling, and multilingual pre-training.
LLM Research Scientist – Pre-training, Computer Vision, Adversarial Robustness
LLM research scientist training and improving frontier AI models for vision, language, and adversarial robustness. Conducting empirical research and optimizing models under data, compute, size, and latency constraints.
Research Scientist – First Author
Research Scientist validating AI-generated reproductions, simulations, and results for an AI-for-science benchmark. Contributing to a public paper and evaluation set as lead or first author.
Research Scientist – First Author
Research Fellow validating AI-generated scientific research for a specialized AI lab. Creating benchmark tasks, evaluation standards, and a public first-author research paper.
Research Scientist – First Author
Research Fellow validating AI-generated scientific research for a specialized AI lab. Creating benchmark tasks, evaluation standards, and a public first-author research paper.
Research Assistant – AI Development
Research Assistant developing AI conversational systems and prototypes for Johns Hopkins education research. Managing study data, APIs, user-activity tracking, and iterative feature testing.
Research Scientist Intern
Research Scientist Intern advancing Pluralis’s decentralized large-model training research. Publishing foundational Protocol Learning papers toward frontier-scale models.
Master's/Ph.D. Research Fellow – Credit Risk Modeling, Python, Statistics
Pesquisador bolsista desenvolvendo modelos de rating e provisão de crédito para o Itaú. Aplicação de Python, estatística e análise de dados em projeto de inovação.
Cybersecurity Research Assistant
Cybersecurity Research Assistant securing Kubernetes and HPC infrastructure for Axle’s NIH rare-disease research. Building controls, monitoring, automation, and AI-assisted security tooling.