The language model made many natural processing (NLP) tasks seem easy. Tools like ChatGpt can produce surprisingly good responses, and even veteran experts wonder if some of the work will be handed over to the algorithm sooner than later. But just as impressive as these models, it still stumbles over tasks that require accurate, domain-specific extraction.
Motivation: Why build a pico extractor?
This idea arises during conversations with students and is trying to calculate future trends in Parkinson's disease treatment and potential costs awaiting insurance if they graduate from International Health Management and their current exams have been transformed into a successful product. The first step was classic and tedious. Isolate the PICO elements – descriptions of population, interventions, comparisons, and results are from the execution of trial statements published on ClinicalTrials.gov. This PICO framework is often used in evidence-based medicine to build clinical trial data. She wasn't a coder or an NLP specialist, so she did this completely by hand and worked on a spreadsheet. Even in the LLM era, it has become clear that there is a real demand for simple and reliable tools for biomedical information extraction.
Step 1: Understand your data and set goals
Like all data projects, the first order for a business is setting clear goals and identifying who will use the results. Here, the aim was to extract PICO elements for downstream predictive analysis or meta-study. Audience: Those interested in systematically analyzing clinical trial data, such as researchers, clinicians, or data scientists. With this range in mind, I started by exporting from ClinicalTrials.gov in JSON format. Initial field extraction and data cleaning provided some structured information (Table 1) especially for intervention, whereas other important fields were still disorderly redundant due to downstream automated analysis. This is where NLP shines. This allows important details to be distilled from unstructured texts such as eligibility criteria and tested drugs. Named Entity Recognition (NER) enables automatic discovery and classification of key entities. For example, identify population groups described in the Eligibility section, or identify outcome measures within the study overview. Therefore, this project naturally moved from basic preprocessing to implementation of domain-adapted NER models.
Step 2: Benchmarking an existing model
My next step was to investigate ready-made NER models, especially those trained in biomedical literature and available via Huggingface, a central repository of trans models. Of the 19 candidates, only Bioelectra-Pico (110 million parameters) [1] I worked directly to extract Pico elements, but the other elements are trained in NER tasks, but not particularly in Pico recognition. The manually annotated test set of Bioelectra's Test 20 in my own “Gold Standard” set showed that it was far from ideal performance, with particularly weaknesses in the “comparator” element. This was probably because comparators are rarely explained in trial summaries, forcing a return to a practical rule-based approach and searching directly for intervention texts for standard comparator keywords such as “placebo” and “normal care.”
Step 3: Fine tuning with domain-specific data
To further improve performance, we moved on to fine-tuning thanks to Bids-Xu-Lab's annotated PICO dataset, which includes Alzheimer's unique samples. [2]. Three models were selected for the experiment to balance the need for high accuracy with efficiency and scalability. Biobert-V1.1110 million parameters [3]a powerful track record of biomedical NLP tasks, served as a major model. We also included two small derived models to optimize speed and memory usage. Compactbiobertwith 65 million parameters, it is a distilled version of Biobert-V1.1. and Biomobilebertwith only 25 million parameters, there was an additional compressed variant, and after compression, received additional continuous learning [4]. I tweaked all three models using a Google Colab GPU. This allows for efficient training.
Step 4: Evaluation and Insights
The results summarized in Table 2 reveal clear trends. All variants work strongly in extracting sample populations, with Biomobilebert leading at F1 = 0.91. The extraction of the results was close to the upper limits of all models. However, interventions have been found to be more difficult to extract. The recall was very high (0.83–0.87), precision delay (0.54–0.61), and the model is frequently tagged with free text. This is because, in many cases, the study description covers the keywords of drugs or “intervention-like” but does not necessarily focus on the main planned intervention.
In a thorough examination, this highlights the complexity of biomedical NER. Interventions sometimes appeared as short fragmented strings such as “whole use”, “week”, “top”, “tissue”. Similarly, looking at the population gave us rather calm examples such as “with percentage” and “state”, indicating the need for additional cleanup and pipeline optimization. At the same time, the model can extract impressively detailed population descriptors, such as “qualified population descriptors for eligible adults with a diagnosis of cognitively-free or potentially Alzheimer's disease, frontotemporal dementia, or dementia with Lewy bodies.” Although such long strings may be correct, the descriptions of participants in each trial are very specific and often require some form of abstraction or standardization, which tend to be too verbose for practical summaries.
This highlights the classic challenges of biomedical NLP. Context issues, and domain-specific texts, often resist purely common extraction methods. For comparator elements, a rule-based approach (matching explicit comparator keywords) works best, reminding us that blended statistical learning between practical and practical heuristics is often the most viable strategy in real-world applications.
One of the main causes of these “naughty” extractions comes from how the exam is explained in a broader context section. Possible improvements to advance include adding post-processing filters that discard short or vague snippets, incorporating domain-specific control vocabulary (as only recognized intervention terms are retained), or applying concepts that link to known ontology. These steps help ensure that the pipeline produces cleaner and standardized output.
Performance Words: For any end-user tool, speed is just as important as accuracy. Biomobilebert's compact size translated into faster inferences and is my favorite model, especially as it worked best for population, comparison, and outcome elements.
Step 5: Enabling the Tool – Deployment
Technical solutions are as valuable as they are accessible. I wrapped the final pipeline into a streamlined app, allowing users to upload ClinicalTrials.gov datasets, switch models, extract PICO elements, and download results. The quick summary plot provides a matching art view of the top interventions and outcomes (see Figure 1). I purposely left a low-performance bioelectra model for users to compare durations of performance to understand the increased efficiency of using smaller architectures. This tool was too late to save time in manual data extraction for my students, but I hope it will benefit others facing similar tasks.
To make deployment easier, I containerized my apps with Docker so that my followers and collaborators can get them up and running quickly. We also invested a lot of effort in Github Repo. [5]provides thorough documentation to promote further contributions and adaptation to new domains.
Lessons learned
This project presents the complete journey of developing a real extraction pipeline, from setting clear goals and benchmarking existing models to fine-tune them with specialized data and deploying user-friendly applications. The models and data were readily available for fine-tuning, but turning them into truly useful tools proved to be more challenging than expected. We highlighted the limitations of a versatile solution by dealing with complex, multi-word biomedical entities that are often only partially recognized. The lack of abstraction in extracted text also has been a hindrance for those seeking to identify global trends. Moving forward, we need a more focused approach and pipeline optimization, rather than relying on a simple Prêt-a-Porter solution.
If you are interested in extending this task or adapting your approach to other biomedical tasks, we recommend exploring the repository. [5] And contribute. Just fork the project Happy coding!
reference
- [1] S. Alrowili and V. Shanker, “Biom-Transformers: Construction of a Large Biomedical Language Model with Bert, Albert, and Electra,” Proceedings of the 20th workshop on biomedical language processingD. Demner-Fushman, KB Cohen, S. Ananiadou, and J. Tsujii, eds. , Online: Association for Computational Linguistics, June 2021, pp. 221–227. doi: 10.18653/v1/2021.bionlp-1.24.
- [2] bids-xu-lab/section_spific_annotation_of_pico. (August 23, 2025). Jupyter notebook. Clinical NLP Lab. Accessed: September 13th, 2025. [Online]. Available: https://github.com/bids-xu-lab/section_spific_annotation_of_pico
- [3] J. Lee et al.“Biobert: A Pre-Trained Biomedical Language Expression Model for Biomedical Text Mining.” BioinformaticsVol. 36, no. 4, pp. 1234–1240, February 2020, doi: 10.1093/bioinformatics/btz682.
- [4] O. Rohanian, M. Nouriborji, S. Kouthaki and Da Clifton, “On the effectiveness of compact biomedical transformers.” BioinformaticsVol. 39, no. 3, p. BTAD103, March 2023, doi: 10.1093/bioinformatics/btad103.
- [5] Eren, ELENJ/Biomed-Extractor. (September 13, 2025). Jupyter notebook. Accessed: September 13th, 2025. [Online]. Available: https://github.com/elenj/biomed-extractor
