Empowering clinical trial design with agentic intelligence and real-world data

Overview

EmulatRx is a modular multi-agent framework designed to support efficient, automated, and expert-aligned CTD. It comprises five specialized agents, i.e., Supervisor, Trialist, Informatician, Clinician, and Statistician, each powered by LLMs and equipped with distinct domain-specific capabilities (Fig. 1a). These agents collaborate through structured conversations to perform key steps in the trial design workflow, including protocol specification, data extraction, covariate selection, statistical modeling, and interpretation. The architecture not only includes a core sequential pipeline, i.e., Supervisor → Trialist → Informatician → Clinician → Statistician → Supervisor (as shown in Fig. 1c), which reflects the natural progression from initial trial specification to analysis and reporting, but also extends beyond this traditional sequential execution by supporting dynamic interactions among agents. For instance, the Informatician can consult the Clinician when facing data sparsity or missing covariates, prompting iterative refinements in eligibility criteria or variable/outcome substitutions. This flexible agent communication strategy allows the system to adapt to appropriately accommodate the challenges in RWD. Note that EmulatRx does not require manual specification of the workflow, allowing agents to dynamically respond to one another’s outputs and adapt to new constraints or evolving objectives. For instance, if the Informatician identifies high levels of missingness for a key variable or detects poor covariate balance, the system can autonomously initiate a feedback loop with the Clinician to assess alternative variable definitions or biomedically appropriate surrogates. To ensure control and reproducibility, EmulatRx acts not as an unstructured chatbot but as a structured system built on LangGraph, which enforces a strict Graph-Based Control Flow. In this context, we define an agent as a modular unit combining a role-specific LLM reasoning engine, a persistent memory state, and a set of executable tools. Agents operate as distinct nodes within a predefined graph, and transitions (handoffs) between them act as directed edges governed by explicit logic. All conversation history and data transformations are stored in a centralized, serializable state object, allowing the complete progress of tasks to be inspected and reproduced (e.g., verifying exactly why the control transitioned from the Informatician to the Clinician). Furthermore, to mitigate the stochasticity of LLMs, the framework implements an LLM Response Cache and enforces fixed random seeds for all downstream statistical tools, ensuring that experimental results can be identically replicated provided the cache is preserved. In addition, EmulatRx integrates tool-augmented reasoning capabilities of each agent (Fig. 1b). A clinical trial knowledge was built from clinical trial registries on clinicaltrial.gov to help the Trialist identify relevant protocols; the RAG module empowers the Clinician to ground its decisions in biomedical literature by performing semantic searches over large corpora such as PubMed; and the Trial Simulator provides the Statistician with access to statistical and machine learning libraries for confounder adjustment, outcome analysis, and treatment effect estimation. These tools are invoked automatically within the agent workflow to convert natural language insights into executable code, structured queries, and interpretable analytics. Beyond these core utilities, EmulatRx supports advanced reasoning functions (Fig. 1d), including knowledge grounding from literature to improve factual accuracy, RLHF to align agent outputs with expert preferences, eligibility criteria (EC) optimization using Shapley-based attribution methods to quantify the influence of inclusion rules on outcomes, and subgroup analyses to uncover heterogeneous treatment effects that may be masked in the aggregate population. These capabilities are modular yet synergistic, enabling EmulatRx to continuously refine trial protocols with both methodological robustness and clinical interpretability.

The final output of this multi-agent system is a comprehensive trial design report (Fig. 1e), which synthesizes contributions from all agents into a unified document. This report includes standardized sections, such as abstract, introduction, methods, protocol specifications, results, and discussion, and is enriched with protocol tables, statistical summaries, and visualizations (e.g., hazard ratios with confidence intervals, covariate balance diagnostics). By automating the generation of such high-quality outputs, EmulatRx not only accelerates the design cycle but also ensures transparency and reproducibility. Together, these components reflect EmulatRx’s capacity to transform the traditionally manual, expert-driven process of CTD into an efficient, intelligent, and collaborative workflow.

We evaluated EmulatRx’s performance by assessing the specialized capabilities of each agent using MIMIC-IV²⁴ database and INSIGHT clinical research network²⁵. We selected a diverse evaluation set of clinical trials covering various diseases, interventions, and trial phases. We curated 20 clinical trials in total, including 10 trials from MIMIC-IV focusing on acute conditions (e.g., septic shock, acute heart failure, acute pulmonary edema, and acute kidney injury) and 10 trials from INSIGHT covering chronic diseases (e.g., Alzheimer’s disease and Parkinson’s disease) with long-term follow-up and longitudinal eligibility constraints. This design enables evaluation across distinct care settings, temporal structures, and phenotype definitions. Our evaluation spanned five core dimensions: (1) clinical trial query based on the knowledge graph, (2) entity extraction and trial parsing for each trial, (3) SQL query generation by the Informatician, (4) causal inference and outcome analysis by the Statistician, and (5) clinical reasoning and recommendation quality by the Clinician. Across these tasks, we benchmarked several LLMs, including GPT-4o²⁶ and three locally deployed LLMs Phi-4²⁷, DeepSeek-R1:14b²⁸ (hereafter referred to as DeepSeek-R1), and Gemma-3:12b²⁹ (hereafter referred to as Gemma 3), to evaluate their impact on agent performance.

Evaluations on trialist

For the Trialist, evaluations concentrate on (1) the query accuracy of the clinical trials based on the knowledge graph; (2) entity parsing for the trial information.

We collected 1363 clinical trials from clinicaltrial.gov, covering the diseases of septic shock, acute kidney injury, acute heart failure, and acute pulmonary edema in the ICU. We evaluated the accuracy of querying relevant clinical trials with a use case as the effect of hydrocortisone in septic shock patients. We designed the following example, ranging from easy to complicated, depending on the involvement of criteria:

Case query 1: Retrieve all septic shock clinical trials that target “hydrocortisone” as the intervention.

Case query 2: Retrieve all septic shock clinical trials that target “hydrocortisone” as the intervention but exclude patients with Glucose-6 phosphate dehydrogenase (G-6PD) deficiency from participating.

Case query 3: Retrieve all septic shock clinical trials that target “hydrocortisone” as the intervention but exclude patients with platelet counts of less than 30000 per cubic millimeter from participating.

We followed existing work³⁰ to employ a hybrid approach combining keyword search with manual review to identify relevant clinical trials to serve as the ground truth for each case query (details provided in Supplementary Note 1). We then compared the manually curated results with those retrieved by the automated queries with the following methods: (1) query on our trial knowledge graph; (2) query with the API of the clinicaltrial.gov; (3) query directly with the GPT-4o. For the ClinicalTrials.gov API baseline, the queries were manually constructed by a human by identifying relevant keywords from each clinical question and issuing those keywords to the ClinicalTrials.gov API. To ensure transparency and reproducibility, the exact query keywords and corresponding API calls used for each case are explicitly listed in Supplementary Note 1. The GPT-4o baseline was included as a naïve LLM-based retrieval reference to illustrate the limitations of directly using a general-purpose LLM for trial identification without external grounding. Specifically, the same clinical questions were posed to GPT-4o in a standard chat interface, and the trial identifiers returned by the model were collected and evaluated manually.

The results are presented in Table 1. With the support of the clinical trial knowledge graph, Trialist accurately identifies all relevant clinical trials across all query conditions. In contrast, directly using the ClinicalTrials.gov API can retrieve all relevant trials only when the query is limited to the target disease and treatment. Its performance declines when the query includes additional eligibility criteria, due to the lack of preprocessing and structured representation of free-text eligibility information in the original ClinicalTrials.gov database. Directly querying LLMs yields high precision, but misses many eligible clinical trials, leading to lower recall.

Besides querying all the relevant trials, parsing the eligibility criteria in each trial is also essential in the following steps. For quantitative evaluation, we used the above 20 clinical trials covering both acute and chronic diseases. Eligibility criteria of the 20 selected trials were manually annotated by two biomedical informatics experts to create a gold standard, and inter-annotator agreement is measured using Cohen’s Kappa³¹, targeting a score of ≥0.7. The evaluation was conducted at the level of knowledge graph nodes corresponding to clinical concepts parsed from trial protocols. A parsed concept node was deemed correct only when all associated attributes, including the concept name, semantic category (e.g., condition or observation), temporal qualifier, and value, exactly matched the corresponding human annotation. Metrics for entity parsing include: (1) Precision: The proportion of correctly identified concepts among all extracted concepts. (2) Recall: The proportion of correctly identified concepts out of all concepts in the gold standard. (3) F1-score: The harmonic mean of Precision and Recall.

Results are presented in Fig. 2a. A total of 266 concepts were annotated across the 20 selected clinical trials. GPT-4o achieved the best performance in identifying these concepts, with a recall of 98.9% and precision of 96.7%. The high recall indicates that most of the ground-truth concepts (263 out of 266) can be correctly recognized. In comparison, the highest performance among the other models was achieved by Gemma, with a recall of 88.0% and a precision of 91.8%. Notably, GPT-4o significantly outperformed other models in handling components that involve multiple concepts or omitted scopes, especially when enhanced with our prompt augmentation strategy. For example, in the criterion “Allergy to vitamin C, hydrocortisone, or thiamine,” GPT-4o accurately extracted the concepts “allergy to vitamin C,” “allergy to hydrocortisone,” and “allergy to thiamine,” while other models returned fragmented or incomplete extractions such as “allergy,” “vitamin C,” “hydrocortisone,” and “thiamine.” In another case, for the criterion “patients <18 years,” GPT-4o correctly inferred the omitted concept “age,” which other models failed to recognize.

Evaluations on informatician

For the Informatician, evaluations focus on the SQL generation following Criteria2Query3.0³², using all clinical trials parsed by Trialist. Each generated SQL query is reviewed to identify potential errors manually. To ensure query correctness, the generated SQL statements were executed on the database, and their output was compared with expected cohort retrieval results.

Errors identified in the SQL queries were classified into seven categories (Supplementary Table 1), broadly falling into two types: semantic errors and structural errors³². Semantic errors included logic errors, where relational, temporal, or numerical expressions were misinterpreted (e.g., incorrect handling of age restrictions), concept omissions, where extracted clinical concepts were not incorporated into the query, and incorrect concept mapping, where the linkage between extracted criteria and database standard terminologies was inaccurate. Structural errors encompassed function misuse, integrity constraint violations, schema reference errors, and syntax errors, which affected the overall execution and validity of the queries.

The results are summarized in Fig. 2b, where accuracy is defined as the proportion of trials in which no error of a given type occurred. We observed that GPT-4o achieved the highest accuracy across all error categories, indicating superior reliability in translating trial specifications into executable SQL queries. In particular, GPT-4o demonstrated strong performance in preserving logical consistency and schema validity, while maintaining high accuracy across both semantic and structural error types. Phi-4 showed moderate accuracy, whereas DeepSeek-R1 and Gemma 3 consistently exhibited lower accuracy, especially for error types related to concept grounding and query syntax. These results indicate that GPT-4o is the most reliable model for SQL generation within the Informatician agent.

The evaluation results using GPT-4o further revealed a significant association between total error count and eligibility criteria complexity (Table 2). We adopted a heuristic definition from prior literature³³ to classify eligibility criteria as simple or complex and quantified the overall complexity of one trial using a complexity score, defined as the proportion of complex eligibility criteria. We then examined the relationship between total error count and eligibility criteria complexity, observing a statistically significant positive correlation (Spearman’s ρ = 0.45, p = 0.043 < 0.05)³⁴. This result indicates that higher eligibility criteria complexity is associated with increased error count. This trend suggests that query accuracy degrades as eligibility criteria become more complex, reflecting the growing difficulty of accurately parsing and translating intricate clinical conditions into executable SQL queries. Overall, the Informatician agent demonstrates strong SQL generation capabilities. These findings are consistent with prior observations reported in Criteria2Query 3.0³².

For the emulated trial cohort of NCT03872011 with the complexity score of 0.64³³, EmulatRx identified 33 additional eligible patients through natural language processing of clinical notes that were not discoverable via structured electronic health record data alone (Supplementary Fig. 1), highlighting the added sensitivity afforded by unstructured data analysis.

Evaluations on statistician

For the Statistician, we primarily evaluate the causal inference methods. Covariate balance is measured using standardized mean differences (SMD)³⁵, aiming for values below 0.1. The accuracy of survival estimates is compared against published literature, while consistency with known trial results is assessed. Generated reports are also reviewed for clarity and alignment with research objectives. We also performed several evaluations of the various components of the Statistician using synthetic datasets.

For the first evaluation, we created a synthetic dataset with 1,000 subjects and 10 covariates sampled from a standard normal distribution. Treatment assignment was imbalanced using a logistic function based on Sequential Organ Failure Assessment (SOFA) and age. SOFA is a tool used to assess organ dysfunction in critically ill patients in the ICU³⁶. Survival times were simulated under an exponential proportional hazards model with a baseline hazard of 0.1 and treatment effects corresponding to ground truth HRs of 0.5, 1, 2, and 3. Random censoring was incorporated using an exponential distribution, and the ground truth average treatment effect (ATE) was defined as the risk difference at a 10-time unit horizon. Four different large language model (LLM) bases (GPT‑4o, Phi-4, DeepSeek-R1, and Gemma 3) were used to drive agent decision-making and model selection (Table 3).

Across all LLMs, propensity score matching was consistently chosen as the balancing method by the Statistician. For a ground truth HR of 0.5, the Statistician estimated a HR of 0.6345 (95% CI: 0.5448–0.7389) and an ATE of –0.0997, compared with a ground truth ATE of –0.1759. When the true effect was null (HR = 1), the estimated HR was 1.0383 (95% CI: 0.8958–1.2034) with an ATE of 0.0244 (ground truth ATE = 0). For a moderate effect (HR = 2), the estimated HR was 1.7524 (95% CI: 1.5152–2.0267) and the ATE was 0.1461 (ground truth ATE = 0.1633). Finally, for HR = 3, the Statistician produced an estimated HR of 2.8074 (95% CI: 2.4330–3.2394) with an ATE of 0.2225, versus a ground truth ATE of 0.2832. These estimates were robust across all LLMs and outcome model selections, demonstrating that the Statistician can reliably recapitulate the ground truth parameters in trial emulation. It is important to note that the HRs and confidence intervals in Table 3 are identical across different LLMs. This is because of the Statistician agent’s tool-use architecture. The agent’s role is to select the optimal statistical method based on data characteristics, while the actual computation is done by deterministic statistical libraries. In this evaluation, all four LLMs identified the same optimal strategy for the synthetic dataset. As such, they used the exact same execution tools, leading to identical numerical results.

In the second evaluation, we generated a synthetic survival dataset with 1000 subjects containing two covariates (SOFA and age) and a binary treatment. In the overall (unstratified) analysis, the treatment effect was engineered such that the average HR was null (i.e., no significant treatment effect). However, the treatment effect was designed to interact with the SOFA score such that, when subjects were grouped by a clinically meaningful cutoff, significant subgroup effects would emerge. Specifically, for treated subjects, the log hazard effect was set to +1 when the SOFA score was below a threshold (e.g., <8.0) and –1 when the score was above that threshold; a constant was subtracted to force the marginal (unstratified) HR to be 1. We evaluated the ability of our multi-agent framework (incorporating Statistician and Clinician agents) to (1) detect the absence of an overall treatment effect and (2) uncover significant subgroup effects by stratifying on SOFA. Three different large language model (LLM) bases (GPT‑4o, Phi-4, and DeepSeek-R1) were deployed to drive agent decisions regarding subgroup creation and subsequent survival analysis using Cox proportional hazards models.

Table 4 shows the results. Unadjusted Cox modeling of the full dataset revealed no significant treatment effect. However, after stratification by SOFA, subgroup analyses produced statistically significant differences. For analyses driven by GPT‑4o and Phi-4, subjects with “SOFA < 8.0” had an estimated HR of 1.3849 [95% CI: 1.2101, 1.5850] while those with “SOFA ≥ 8.0” had an estimated HR of 0.7280 [95% CI: 0.5921, 0.8950]. In contrast, the DeepSeek-R1 and Gemma 3 driven analysis identified a slightly different cutoff (SOFA < 7.0 vs. SOFA ≥ 7.0) yielding HRs of 1.3443 [95% CI: 1.1760, 1.5366] and 0.8286 [95% CI: 0.6853, 1.0019], respectively. These findings demonstrate that while the overall analysis recapitulated the engineered null effect, appropriate stratification by the effect modifier SOFA uncovered significant heterogeneity in treatment response. The consistency across different LLM bases supports the robustness of our multi-agent evaluation framework.

We evaluated our EC optimization algorithm by creating synthetic datasets with known ground-truth importances for each EC and then measuring how closely the Monte Carlo-based Shapley estimator recovered these importances under varying numbers of ECs. For each run, we drew a specified number of ECs (from 2 to 20), sampled true importances uniformly from a predefined range, and combined these with a baseline hazard-ratio (HR) value. We enumerated every possible subset of the ECs and generated noisy HRs by adding Gaussian noise (standard deviation = 0.5) to each subset’s baseline plus its summed importances. Our algorithm then performed randomized permutations of the ECs to compute incremental differences in HR from the empty to the full set of rules, up to a maximum of 1000 iterations or until the standard error of the mean (SEM) reached a small tolerance (1e-3). We compared each criterion’s estimated Shapley value against its ground-truth importance using mean absolute error (MAE). Supplementary Table 2 shows the MAE for varying number of ECs.

Supplementary Table 2 shows that, even in the presence of moderate noise, the algorithm converges effectively and typically produces low MAE values, indicating good agreement between the estimated Shapley values and the known true importances.

To demonstrate the utility of adaptive sample size calculation, we present a showcase based on trial NCT00475852, which investigated the effects of nesiritide in patients with acute heart failure. In the real-world cohort assembled by the Informatician agent from the MIMIC-IV database, a total of 6971 patients met the eligibility criteria aligned with the original trial protocol. However, to assess the feasibility of prospective trial planning and explore whether fewer participants would suffice to achieve adequate statistical power, the Statistician agent was prompted to initiate sample size estimation. Using the Schoenfeld formula for the Cox proportional hazards model, the agent incorporated key characteristics from the observed data, including treatment allocation ratio, baseline event rate, censoring proportion, and average follow-up time. The target specifications for the calculation were a Type I error rate (α) of 0.05, power (1–β) of 0.80, and an expected HR based on prior trial literature. The automated estimation yielded a required sample size of 3,107 patients, largely lower than the original cohort size of 6,971. This reduction reflects the relatively high event rate and long follow-up observed in the RWD, which collectively increased the statistical efficiency of the emulated trial design. This result highlights the value of data-driven sample size planning within EmulatRx. By dynamically adjusting for empirical characteristics of the study population, the system can help optimize resource allocation while preserving inferential power. Such functionality is especially valuable in real-world CTD scenarios with limited resources.

Evaluations on clinician

The Clinician agent is evaluated for its ability to integrate medical expertise into the workflow. A questionnaire based on a 5-point Likert scale³⁷ was provided to three independent clinical domain experts to assess the relevance, clarity, and accuracy of the Clinician’s recommendations³². Additionally, the agent’s efficiency in refining trial designs and resolving discrepancies is measured through qualitative feedback from domain experts.

Figure 2c presents the evaluation of Clinician-generated responses across five key dimensions: readability, correctness, coherence, creativity, and usefulness, with detailed subcategory scores provided in Table 5. Overall, GPT-4o outperformed all other models, achieving the highest average score of 4.88. Phi-4 and DeepSeek-R1 followed with strong overall performance (4.78 and 4.71, respectively), while Gemma 3 lagged with a significantly lower average score of 4.40. Readability: GPT-4o achieved near-perfect consistency in readability, scoring 4.99–5.00 across all aspects, including writing clarity, logical structure, and appropriate terminology. DeepSeek-R1 followed with solid scores around 4.86–4.88, showing well-organized responses. Phi-4 scored slightly lower (4.75–4.82), suggesting occasional inconsistencies in formatting or flow. In contrast, Gemma 3 underperformed in this dimension, with scores ranging from 4.54 to 4.58, indicating relative challenges in producing clearly formatted and accessible clinical content.

Correctness. GPT-4o again led in this category, scoring 4.99 for factual accuracy and 4.99–5.00 in logical reasoning and domain-specific usage, reflecting both precision and reasoning strength. Phi-4 was close behind (4.88–4.89), demonstrating high reliability. DeepSeek-R1 showed slightly lower performance (4.71–4.73), pointing to occasional gaps in factual or logical precision³⁸. Gemma 3 trailed with scores between 4.42 and 4.48, underscoring its relative limitation in maintaining strict clinical accuracy.

Coherence. GPT-4o performed strongly again (4.99–5.00), maintaining logical consistency across individual and multiple responses. Phi-4 showed robust coherence (4.86–4.93), effectively avoiding contradictions. DeepSeek-R1 also produced coherent outputs (4.83–4.84), while Gemma 3’s scores (4.47–4.53) pointed to comparatively greater structural fragility and potential internal inconsistencies. Creativity: All four models showed room for growth in this aspect. GPT‑4o displayed relatively higher creativity (3.93-3.95), especially in offering multiple perspectives. Usefulness: GPT-4o stood out as the most practically useful model, scoring 4.97–4.99 for application fit and modifiability. Phi-4 was highly competitive in this category (4.90–4.94), indicating strong utility for downstream tasks. DeepSeek-R1 scored slightly lower (4.74–4.77), while Gemma 3 scored the lowest (4.50–4.52), indicating its output may require more refinement for practical application.

Finally, we conducted an ablation study to validate the contribution of the RAG module. The results demonstrated that RAG is essential for knowledge grounding; while the non-RAG baseline correctly reported statistical outputs, the RAG-enabled agent produced significantly richer summaries that contextualized findings against existing biomedical literature and consistently included granular metrics such as Average Treatment Effects. In summary, GPT‑4o delivered the most accurate, relevant, and clinically useful responses, making it the clear leader among all evaluated LLMs. DeepSeek‑R1 and Phi‑4 demonstrated decent usability with minor flaws, while Gemma 3 consistently underperformed across nearly all dimensions, indicating that its clinical reasoning and formatting still require significant improvement for integration in expert workflows.

Showcases

To demonstrate how EmulatRx operates on diverse clinical questions, we present three separate disease-based examples using GPT-4o, each reflecting distinct trial emulation scenarios (Fig. 3). These examples illustrate how the Supervisor, Trialist, Informatician, Clinician, and Statistician agents coordinate to parse protocols, handle eligibility criteria, select covariates, balance confounders, and produce final analyses.

Case 1: Impact of Nesiritide in Acute Heart Failure Patients. In this first demonstration, EmulatRx emulated a clinical trial (NCT00475852) assessing nesiritide’s impact on heart failure outcomes. The Supervisor initiated the workflow by requesting a detailed target trial protocol. The Trialist agent retrieved and parsed the inclusion criteria, specifying patients hospitalized for acute decompensated heart failure (ADHF) or diagnosed within 48 h, while strictly excluding those with high-risk hypotension (systolic BP < 90 mmHg), acute coronary syndromes, or specific structural heart defects.

The workflow then moved to a collaborative design phase between the Clinician and Informatician agents. The Clinician agent, leveraging its biomedical knowledge base, proposed a high-dimensional feature space comprising 89 distinct covariates, including granular metrics such as A-aDO2, methemoglobin, and abs_eosinophils alongside standard vital signs and organ function scores (e.g., SOFA). The Informatician agent translated these definitions into OMOP-compliant SQL queries, successfully instantiating a large-scale cohort of 13,942 patients.

During the statistical analysis design, the Statistician agent evaluated the dataset’s properties and selected IPTW over PSM to maximize data utilization given the large sample size. In the covariate balancing phase, the agent detected persistent imbalances in abs_eosinophils. Through an autonomous feedback loop, the Statistician consulted the Clinician, who advised dropping the eosinophil count as it was not a primary confounder for hemodynamic outcomes in this context, allowing the model to achieve satisfactory balance (SMD < 0.1).

The outcome analysis was conducted using a Doubly Robust Estimation framework, combining the IPTW weights with a Cox Proportional Hazards model. The analysis yielded a HR of 0.5913 (95% CI: 0.46–0.76), indicating a statistically significant reduction in the composite risk of rehospitalization and all-cause mortality for the nesiritide-treated group. This result, which aligns with the directional findings of the original ASCEND-HF trial but with a stronger effect size likely due to the specific real-world subpopulation captured, was synthesized into a final report by the Supervisor. The report highlighted a 2.8% absolute risk reduction, concluding that nesiritide offers a quantifiable therapeutic benefit in this observational cohort. The final report is in Supplementary Note 2.

Case 2: Effect of Renal Replacement Therapy for Severe Acute Kidney Injury. In the second case study, EmulatRx was tasked with emulating a study (NCT06091982) to determine the impact of Renal Replacement Therapy (RRT) on mortality in patients with severe Acute Kidney Injury (AKI). The Supervisor directed the Trialist agent to parse the protocol, which specified inclusion criteria for adults (age >= 18 years) with confirmed severe AKI, while excluding those with prior chronic dialysis history or incomplete baseline data.

The Informatician agent, collaborating with the Clinician to map clinical concepts to the MIMIC-IV schema, successfully identified a robust cohort of 11,124 patients. The Clinician agent ensured the feature space was comprehensive, selecting 88 covariates that included renal function markers, electrolyte panels, and hemodynamic indicators.

Given the substantial sample size, the Statistician agent autonomously selected IPTW as the optimal balancing strategy to preserve the full statistical power of the cohort, rather than pruning samples via matching. The agent implemented a Doubly Robust Estimation method, adjusting for residual confounding in the outcome model. In contrast to earlier iterations that reported null findings, this high-fidelity emulation revealed a HR of 0.6443 (p = 0.0008), suggesting a significant survival benefit associated with RRT in this critically ill population. The final report synthesized these findings, noting that the rigorous adjustment for confounding provided by the IPTW approach likely unmasked a treatment effect that was previously obscured in smaller, less representative subsamples. The final report is in Supplementary Note 3.

Case 3: Effect of Hydrocortisone in Septic Shock Patients. In the final example, EmulatRx emulated a trial (NCT04134403) evaluating hydrocortisone therapy in patients with septic shock. The Supervisor requested a protocol targeting patients with persistent vasopressor dependence and lactate levels >2.0 mmol/L. The Trialist agent parsed the eligibility criteria to exclude patients with confounding conditions such as major hemorrhage, burns, or prolonged vasopressor use prior to randomization.

Working in tandem, the Clinician and Informatician agents curated a dataset of 2306 patients, a substantial increase over the 1153 patients identified in the previous run. The Clinician agent specified 87 covariates, including granular inflammatory markers and organ function scores, ensuring a dense feature representation for causal inference.

The Statistician agent employed IPTW coupled with Doubly Robust Estimation. This emulation detected a Hazard Ratio of 1.2983 (95% CI: 1.07–1.58; p = 0.0082), indicating a statistically significant increased risk of 28-day mortality in the hydrocortisone-treated group within this specific MIMIC-IV subpopulation. These findings are consistent with previous analyses on the MIMIC-IV data, as it relates to the usage of steroids on sepsis patients⁸. The final report flagged this finding as a critical safety signal, prompting the Clinician agent to recommend extreme caution and further subgroup stratification in future prospective designs. This case underscores EmulatRx’s capacity to identify potential harms in real-world settings that may differ from controlled trial environments due to population heterogeneity. The final report is in Supplementary Note 4.

We further examined the interaction dynamics to assess whether our framework indeed operates as a collaborative multi-agent framework. Supplementary Fig. 2 summarizes the observed agent-to-agent interaction transitions aggregated across all trials and LLMs. The patterns illustrate rich inter-agent interactions, including conversations between the Informatician and Clinician during cohort construction, between the Statistician and Clinician during covariate and model refinement, and coordination mediated by the Supervisor. These observations indicate that our framework does not function as a simple linear pipeline, but rather as a dynamically interacting system in which agents exchange intermediate results and feedback to iteratively refine trial design and analysis.

To quantitatively evaluate the efficiency of the framework, we measured the end-to-end execution time of the full trial design pipeline. Supplementary Fig. 3 reports the wall-clock runtime across executions with all trials among different LLM backends. We observed that GPT-4o achieves the shortest median runtime (5.75 ± 1.52 min). Phi-4 and Gemma-3 exhibit moderate runtimes (20.95 ± 7.94 min, 25.87 ± 4.38 minutes, respectively), while DeepSeek-R1 shows higher variance and longer runtimes (31.36 ± 14.51 minutes), largely attributable to its extended chain-of-thought reasoning and local inference latency. In contrast, manual execution of comparable workflows requiring domain experts to translate trial protocols, write and debug SQL queries, engineer covariates, conduct causal analyses, and iteratively refine design choices typically spans multiple days to weeks^39,40,41,42. These results provide empirical evidence that EmulatRx not only facilitates but substantially accelerates the clinical trial design process.