Tech Lead Data Scientist, AI Evaluation – Monitoring
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Tech Lead Data Scientist guiding AI evaluation and monitoring at Geisinger. Leading technical teams and shaping methodologies for AI systems in healthcare.
Responsibilities:
- Principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems
- Set technical direction for AI evaluation across a growing portfolio
- Provide technical leadership to a team of data analysts
- Partner with AI program teams to improve validation and monitoring methods
- Design validation studies and monitoring plans for AI systems
- Instrument production monitoring and define measurable metrics
- Mentor analysts on methodology and technical rigor
Requirements:
- 6+ years in data science, statistics, ML engineering, or applied quantitative research
- Strong foundation in experimental design and causal inference
- Hands-on experience designing and running model evaluation studies in real production settings
- Experience evaluating LLM or generative AI systems
- Proven ability to translate ambiguous failure modes into concrete evaluation designs
- Strong fluency in Python and SQL
- Experience with fairness and equity evaluation for ML systems
- MS or PhD in a quantitative field preferred
Benefits:
- Healthcare benefits from day one
- Vision insurance
- Dental insurance
- Domestic partner benefits
- Collaboration and cooperation atmosphere
Similar Jobs
Principal, Analytics and Data Science – Pricing Forecasting
Principal position leading data science and analytics within CVS Health's underwriting transformation program for pricing forecasting and financial decision-making. Collaborating across teams to drive actionable insights based on advanced analytics.
Senior Data Scientist
Senior Data Scientist at Montreal improving AI in education. Focused on reducing student dropout and preventing financial fraud.
Senior Data Scientist
Senior Data Scientist at Switchfly responsible for innovative data science and ML applications. Driving business results through analytic innovation and supporting large-scale adoption of data science.
Senior Data Scientist
Data Scientist responsible for creating a Medical Decision-Support Engine using AI and medical data. Leveraging LLMs and Knowledge Graphs to enhance mental health diagnostics.
Senior Data Scientist
Data Scientist developing insights and models for a fast-growing healthcare marketplace. Collaborating with teams to improve data-driven decision-making and marketplace efficiency.
Data Lead
Data lead responsible for structuring and scaling engineering teams in Cappta, a Brazilian fintech specialized in payment solutions. Focus on decision-making architecture and team development.
Senior AI Data Scientist
Senior Data Scientist leading AI projects to optimize patient care and operational efficiency in a healthcare organization. Also mentors junior data scientists for a data-driven culture.
Data Scientist – Innovation
Data Scientist developing scalable procedures for data extraction and analysis. Enhancing Comscore products with insights from diverse data sources and collaborating with teams.
Senior Data Scientist
Senior Data Scientist working with a skilled data science team at Cotiviti. Analyzing healthcare data and delivering value through advanced data science models and collaboration.
Data Scientist
Data Scientist developing machine learning solutions for fraud prevention and AML compliance. Collaborating with clients to address evolving fraud challenges in a remote-first organization.
Data Scientist, Finance
Data Scientist enhancing Finance through data analytics for Whatnot's fast-growing marketplace. Building dashboards and actionable KPIs while collaborating on data engineering.
Senior Data Scientist – Fraud Data Infrastructure, Automation
Senior Data Scientist at Socure transforming raw datasets into actionable insights to prevent fraud. Leading innovative approaches such as agentic AI and data automation for improved decision systems.
Data Scientist / ML Engineer
Data Scientist / ML Engineer developing scalable machine learning systems for Antarctica Capital. Collaborating with teams to optimize models and infrastructure in a remote setting.
Data Scientist
Data Scientist at WMT Digital solving dynamic pricing problems using advanced mathematics and Python. Focused on pipeline development, data analysis, and collaboration with stakeholders.
Senior Data Scientist
Senior Data Scientist collaborating with AI Technical Product Manager and ML Engineers at Blanc Labs. Analyzing data to drive actionable insights and measurable business value.
Senior Data Scientist
Senior Data Scientist at Blanc Labs shaping AI-driven product decisions through data analysis. Collaborating with cross-functional teams to drive measurable business value from AI initiatives.
Senior Data Scientist – Flex
Leading development of data science aspects for Akamai's web security products in a multidisciplinary research team. Responsible for innovation, model development, and insights on web security attacks.
Manager, Data Science
Manager, Data Science role at Whatnot leading team to drive analytical strategy and solve marketplace challenges. Focus on growth, ecosystem health, and developing data-driven solutions.
ERP Functional Data Lead Analyst
Functional Data Lead Analyst managing ERP implementation projects at aerospace and defense company. Coordinating large scale conversion activities for efficient project execution and stakeholder collaboration.
Senior Master Data Manager, Media
Senior Master Data Manager Media managing data integrity for Mediaplus Germany. Collaborating with clients, purchasing, and accounting teams to maintain and process centralized data.