When Relevance Is Not Enough: Aspect-Controlled Biomedical Evidence Retrieval
186
Expert-test abstracts
0.4%
Instruction-only switching
38.8%
Adapted switching
Tech Stack
Description
Semantic search can find text about the right clinical trial yet miss the specific evidence a question asks for. This project tests whether an embedding model can change its sentence ranking when the requested evidence changes from Population to Intervention or Outcome.
I developed a within-document retrieval evaluation from EBM-NLP 2.0 and compared generic retrieval, instruction-only retrieval, and a lightweight query-side LoRA adaptation of multilingual E5. This is a preliminary research study; the paper is currently a draft.
- Converted token-level P/I/O annotations into sentence-level retrieval targets, retaining multi-aspect candidates in the search pool.
- Measured both ordinary retrieval and strict aspect-specific switching on an expert-annotated test set.
- Found that query-side adaptation substantially improved two-aspect switching over instructions alone without reducing ordinary retrieval quality.
Project Highlights
Research Questions
A clinical-trial abstract may describe who participated (Population), what treatment was given (Intervention), and what happened (Outcome). All three concern the same study, but a user may need only one. RQ1. Can an instruction alone make a frozen embedding model retrieve the requested evidence type? RQ2. Does training a small query-side LoRA module improve this control without harming ordinary retrieval? RQ3. Does the effect differ for Population, Intervention, and Outcome?
Experimental Pipeline
The same abstract supplies the same candidate sentences for all three requests. Only the requested evidence type changes. The dataset's labels are used to check the rankings, not shown to the model during testing.
| Step | What happens |
|---|---|
| 1. Prepare data | Use real English EBM-NLP 2.0 trial abstracts. Split their text into candidate sentences and transfer token-level P/I/O labels to each candidate; a sentence may have more than one label. |
| 2. Split by document | Use 4,187 documents for training, 474 for development, and 186 expert-annotated documents for the final test. |
| 3. Train | Construct 18,921 training examples from labelled candidates. Each query has a correct sentence and two incorrect sentences. Update only the query-side LoRA module. |
| 4. Select | After each epoch, check retrieval on the development documents and retain the best checkpoint. Do not train on the development or test sets. |
| 5. Test | Ask for P, I, and O separately for each of the 186 test abstracts (558 queries). Rank the same candidates each time and compare the rankings with expert labels. |
Three Approaches
| Approach | What changes? |
|---|---|
| Generic retrieval | No Population/Intervention/Outcome instruction or training. |
| Aspect-instructed | The frozen encoder receives an aspect instruction; no training. |
| Aspect-adapted | A query-side LoRA module is trained to respond to the aspect instruction; candidate embeddings stay fixed. |
Results and Analysis
The expert-test results below are percentages. Top-1 asks whether the first sentence contains the requested aspect. Aspect-Specific Top-1 is stricter: the first sentence must contain only that aspect. Two-Aspect Switching requires both requests in an eligible pair to retrieve different, correct aspect-only sentences.
| Method | Top-1 | Aspect-Specific Top-1 | Two-Aspect Switching |
|---|---|---|---|
| Generic | 82.1 | 2.7 | 0.0 |
| Instruction-only | 84.2 | 4.0 | 0.4 |
| Query-side LoRA | 89.7 ± 0.8 | 61.3 ± 1.2 | 38.8 ± 3.1 |
Instructions slightly improved ordinary retrieval but almost never made the model switch to the correct aspect-only evidence. This is why a good Top-1 score does not by itself demonstrate control.
With query-side LoRA, Two-Aspect Switching rose from 0.4% to 38.8% on average across three seeds, and ordinary Top-1 also rose from 84.2% to 89.7%. The switching score uses 264 eligible aspect pairs from 128 test documents, not all 558 queries.
Control improved for all three aspects, but Intervention remained harder: its Aspect-Specific Top-1 was 50.7%, compared with 67.8% for Population and 65.6% for Outcome. The study does not yet establish why this difference occurs.
Scope and Next Questions
This is a controlled proof of concept, not an open-corpus clinical search system. It uses one English biomedical dataset, fixed aspect instructions, and one base encoder. Further work needs to test natural questions, other languages and datasets, different embedding models, and larger document collections.