Ziwei Yang, Ph.D.
03 · Discover Biology · AI Agents for Scientific Discovery

Biomedical AI Copilots

Making Large Language Models Reliable Data Science Programming Copilots for Biomedical Research

This study measures how reliably large language models write data analysis code for biomedical research, finds overall accuracy below 40% on a new benchmark of 293 tasks, and shows that planning the analysis before coding raises it to 74%.

LLMsBiomedical Data Science Zifeng Wang, Benjamin Danek, Ziwei Yang, Zheng Chen, Jimeng Sun · Nature Biomedical Engineering, 2026
Benchmark construction: analyses such as participant baselines, Kaplan-Meier curves, statistical tests, OncoPrints, ridge plots and volcano plots are extracted from medical publications found through PubMed and ClinicalTrials.gov; multimodal patient data (omics, clinical, clinical sample) are collected from TCGA and cBioPortal; the result is a coding benchmark with coding tasks in Python and R, analysis results, test cases and an execution sandbox.
How the benchmark was built. Analyses are extracted from published medical studies, the patient data behind them are collected from sources such as TCGA and cBioPortal, and each analysis becomes a coding task with reference results, test cases and an execution sandbox. Figure adapted from Wang et al., Nature Biomedical Engineering 2026 (preprint version, arXiv:2410.21591v2, CC BY 4.0).

Overview

Large language models can turn a plain request into analysis code and a plot in seconds. In biomedical research, though, a plausible-looking but wrong analysis can spread incorrect findings. How often is the code actually right?

To find out, the study built BioDSBench: 293 coding tasks in Python and R, derived from 39 published studies across seven research areas from biomarkers to pan-cancer analysis, each run on real patient data and checked with test cases. Eight proprietary and eight open-source LLMs reached an overall accuracy below 40%.

The most effective fix was to plan first. An agent that drafts an analysis plan, checks it against the actual data and refines it before writing code reached 74%. The same idea drives a platform where researchers develop analysis plans together with an LLM and run them in one environment.

Approach

Benchmark the models, find what helps, then build it into a tool.

  1. 01

    Benchmark

    Turn published analyses into coding tasks with reference answers and test cases, and score each model by whether its code passes them on the first attempt.

  2. 02

    Find what helps

    Compare adaptation strategies, including few-shot examples, retrieval, automatic prompting, self-reflection, code agents and plan-first prompting, across easy, medium and hard tasks.

  3. 03

    Build the platform

    Put plan-first co-development into a platform where users make requests, review generated code and run it on selected datasets, then test it with medical researchers.

Key Results

Headline numbers

<40%
overall accuracy of 16 LLMs
under various prompting strategies
74%
accuracy of the plan-first agent
plans and refines before coding
>80%
of analysis code completed
user study: 5 researchers, 3 studies

All three numbers are from the published abstract. Accuracy is the share of tasks whose generated code passes the test cases.

Benchmark

First-attempt success (Pass@1) by task difficulty Plan-first agentOther strategies
StrategyEasyMediumHard
Few-shot 60%32%13%
AutoPrompt 50%34%3%
RAG 60%37%10%
ManualPrompt 63%45%13%
CodeAgent 50%13%13%
PlanPrompt 63%58%48%
DSWizard (plan-first agent) 90%76%55%
Share of tasks solved on the first attempt for seven adaptation strategies; the bar under each value shows its size. DSWizard is the paper's agent that refines an analysis plan against the data before generating code (prompt in the Supplementary Information). Data from the published Source Data for Fig. 3g; redrawn.

Resources

Paper

Making Large Language Models Reliable Data Science Programming Copilots for Biomedical Research

Nature Biomedical Engineering 2026 · an earlier version is on arXiv

Code

RyanWangZf/BioDSBench

Code for the methods and experiments

Data

zifeng-ai/BioDSBench

Tasks with reference answers and test cases

Platform

Biomedical data science platform

Web app for planning and running analyses with an LLM; access on request