Through project CARE (Clinically Aligned Radiology Expertise), we investigate how AI assistants can be integrated into radiology workflows in a reliable and clinically meaningful manner. CARE develops foundation and multimodal models for radiology across 2D and 3D imaging, with an emphasis on clinical fidelity, calibration, and interpretability — aligning model behavior with the way radiologists reason and validating it through collaborations with hospital partners.
The program currently comprises two sub-projects: CARE-X, addressing chest X-ray (2D) interpretation, and MedCompose, addressing compositional representation learning for 3D imaging such as CT and MRI.
CARE-X
CARE-X is a chest X-ray vision-language model designed to produce outputs a clinician can act on, rather than outputs that merely read well. The chest X-ray is the most common medical image worldwide, and its interpretation involves detecting findings, localizing them, performing measurements, and composing a structured report. While current vision-language models perform the report-writing task competently, a well-formed report can still be clinically incorrect. CARE-X targets three specific failure modes of generative radiology models.
1. Calibration and thresholdable predictions
A purely generative model emits diagnoses as free text, providing no probability estimate and no mechanism to adjust the sensitivity–specificity trade-off across clinical settings (for example, screening versus confirmatory use). CARE-X attaches lightweight discriminative heads that share the generative model’s underlying representation and produce a calibrated, thresholdable score from a single forward pass.
This provides two inference modes from one model:
- Generative mode — autoregressive decoding for free-text reports, VQA answers, and text-based bounding boxes.
- Auxiliary-head mode — discriminative probability scores and detection bounding boxes with associated confidence scores.
The generative and discriminative objectives are complementary: adding the discriminative signal sharpened the shared feature representation and improved generation quality. Clinicians can adjust decision thresholds directly from the model’s outputs.
2. Reward-aligned optimization
Supervised training with cross-entropy optimizes token likelihood, which is a weak proxy for clinical fidelity; a report can achieve high likelihood while omitting a finding or introducing a hallucinated pathology. CARE-X therefore applies reinforcement learning to optimize clinical metrics directly.
For report generation, three complementary rewards are combined so that each compensates for the others’ blind spots:
- Entity-level factual correctness.
- Penalization of omissions and hallucinations of clinically significant findings.
- Semantic faithfulness beyond surface wording.
This reward-driven formulation is extended across all three tasks — report generation, visual question answering, and localization — with rewards tailored to what is clinically relevant in each.
3. Tool-augmented measurement
A class of diagnoses is defined by measurement against an established threshold rather than by pattern recognition; for example, cardiomegaly is determined by a ratio and is fundamentally a geometric computation. Requiring a neural network to estimate such quantities perceptually is inappropriate. Instead, we need to enable the model to issue tool calls: retaining full visual access to the image, it emits a structured tool call to detect the relevant landmarks, compute the ratio, and compare against evidence-based criteria, then synthesizes the diagnosis from the deterministic result.
Hidden Causality in Radiology Reports
Traditional radiology reports document a patient’s symptoms, imaging findings, and final diagnosis, but they rarely make explicit the causal relationships and reasoning that lead to that diagnosis. This omission limits both the interpretability of the report and its value for clinical education. As part of the NTCIR-18 Hidden-RAD Challenge (opens in new tab) (Hidden Causality Inclusion in Radiology Reports), we investigate how AI models can recover this hidden causality — generating a causality-exploration section that reflects the diagnostic reasoning a radiologist implicitly performs.
The work also situates domain-specialized small models against a broad set of general-domain and reasoning-focused baselines, examining how effectively causal reasoning can be integrated with a radiology report or image. Beyond the challenge itself, the recovered causal explanations point toward richer, more transparent automated radiology reporting workflows, where a generated report conveys not only what was observed but why a particular diagnosis follows.
MedCompose: Compositional Representation Learning for 3D Medical Imaging
MedCompose addresses a foundational problem in representation learning for high-dimensional medical imaging. Modern 3D encoders can process large volumes such as CT and MRI scans and learn dense representations, but these representations tend to be entangled: alongside clinically meaningful features, they encode scanner artifacts, acquisition noise, and patient-specific variation. Such models may perform well during training yet fail to generalize across scanners, hospitals, and imaging protocols because they depend on unstable details rather than semantic structure. The project asks how models can be constrained to represent meaningful clinical concepts rather than noise, and how their reasoning can be made interpretable and aligned with clinical practice.
Clinical motivation: compositional reasoning
Radiological characterization is compositional. A standard description such as “calcified, well-circumscribed nodule measuring 5 mm in the lateral segment of the right middle lobe” decomposes into distinct concept axes:
- Textura — internal appearance, density, and composition (“calcified”).
- Forma — shape and the interface between the lesion boundary and surrounding tissue (“well-circumscribed”).
- Scala — spatial scale and physical extent (“5 mm”).
- Locus — anatomical location (“right middle lobe”).
A finding is thus a structured combination of distinct visual concepts. MedCompose-CT operationalizes this observation through a three-stage architecture.
Architecture: Decompose, Discretize, Resynthesize
Decompose. After the 3D encoder extracts visual information, the network branches to separate the feature streams into the four concept axes — Textura, Forma, Scala, and Locus — rather than retaining a single mixed representation.
Discretize. Each concept stream passes through a vector-quantized bottleneck, or concept codebook, constraining the model to express observations using a finite vocabulary of recurring visual patterns learned across many scans rather than an unconstrained continuous vector per patient. This encourages the learning of stable, reusable prototypes to which new findings are assigned. The discretization serves two purposes: it improves interpretability by structuring the representation around recognizable concepts, and it provides semantic denoising by discouraging the encoding of scanner-specific noise, acquisition artifacts, and irrelevant patient variation in favor of recurring structure that generalizes across datasets.
Resynthesize. Because clinical meaning arises from the interaction of attributes rather than from isolated traits — the significance of a small calcified nodule depends on its location — the quantized concept representations are passed to an attention-based resynthesis module that models the interaction of texture, shape, scale, and location within the 3D scene before producing the final segmentation or report.
Properties
- Interpretability — predictions are traceable to specific concept axes, allowing inspection of whether the model reasoned from shape or density and whether that reasoning aligns with clinical logic.
- Robustness — because the discrete bottlenecks prioritize recurring structure, the model is more resilient to scanner variation, acquisition noise, and protocol differences without explicit domain adaptation.
- Compositionality — reusable concept parts allow previously unseen findings to be represented as new combinations of familiar building blocks, supporting generalization.
The approach is intended to generalize beyond 3D medical imaging to any domain describable through structured, reusable semantic concepts, as a step toward more interpretable and faithful visual reasoning.