Zero-Shot Visual-Language Foundation Models in Pathology: A New Paradigm for AI-Driven Diagnosis

Abstract

This article explores the transformative potential of visual-language foundation models (VLFMs) for zero-shot classification in computational pathology. Tailored for researchers, scientists, and drug development professionals, it provides a comprehensive analysis of how models like CONCH and TITAN are overcoming the critical challenge of label scarcity by learning from image-text pairs. The scope ranges from foundational concepts and core architectures to advanced fine-tuning methodologies, performance optimization strategies, and rigorous benchmarking against traditional deep learning and human experts. By synthesizing the latest research, this review serves as a strategic guide for developing and deploying robust, generalizable AI tools that can accelerate diagnostic workflows, enhance personalized medicine, and fuel drug discovery.

Understanding Visual-Language Foundation Models and Their Role in Pathology

The Problem of Label Scarcity in Computational Pathology

The development of robust artificial intelligence (AI) models for computational pathology has been fundamentally constrained by the pervasive challenge of label scarcity. The acquisition of high-quality, expert-annotated data for whole-slide images (WSIs) is labor-intensive, time-consuming, and cost-prohibitive, making it difficult to scale across the thousands of possible diagnoses and rare diseases encountered in pathology practice [1]. This scarcity severely limits the development of task-specific supervised learning models, particularly for rare diseases or complex tasks requiring specialized expertise [1] [2].

Visual-language foundation models represent a paradigm shift in addressing these limitations. By leveraging task-agnostic pretraining on diverse sources of histopathology images paired with biomedical text, these models learn rich, aligned representations of visual and linguistic concepts in pathology [1] [3]. The resulting models can be applied to a wide array of downstream tasks—including classification, segmentation, and retrieval—in a zero-shot manner, requiring minimal or no further labeled data for effective deployment [1]. This approach mirrors how human pathologists teach and reason about histopathologic entities using both visual cues and descriptive language [1].

Quantitative Performance of Zero-Shot Models

Evaluation across multiple benchmarks demonstrates that visual-language foundation models achieve state-of-the-art performance on diverse pathology tasks without task-specific training. The table below summarizes the zero-shot classification performance of CONCH, a leading visual-language foundation model, across multiple tissue and disease types.

Table 1: Zero-shot classification performance of CONCH across diverse diagnostic tasks

Task Description	Dataset	Disease/Cancer Type	Primary Metric	Performance
Slide-level Subtyping	TCGA NSCLC	Non-small cell lung cancer	Balanced Accuracy	90.7% [1]
Slide-level Subtyping	TCGA RCC	Renal cell carcinoma	Balanced Accuracy	90.2% [1]
Slide-level Subtyping	TCGA BRCA	Invasive breast carcinoma	Balanced Accuracy	91.3% [1]
ROI-level Classification	CRC100k	Colorectal cancer	Balanced Accuracy	79.1% [1]
ROI-level Classification	WSSS4LUAD	Lung adenocarcinoma	Balanced Accuracy	71.9% [1]
Gleason Pattern Classification	SICAP	Prostate cancer	Quadratic Weighted κ	0.690 [1]

Comparative studies reveal that CONCH substantially outperforms concurrent visual-language models such as PLIP and BiomedCLIP across these tasks, often by large margins (e.g., >10% accuracy on TCGA NSCLC and RCC subtyping, and >35% on TCGA BRCA subtyping) [1]. This performance establishes a strong baseline for clinical applications, especially when training labels are scarce.

Beyond classification, these models enable cross-modal retrieval, allowing pathologists to search for similar image cases using textual descriptions or generate descriptive captions for histopathology images, thereby enhancing diagnostic workflows and educational applications [1].

Experimental Protocols for Zero-Shot Evaluation

Protocol 1: Zero-Shot Tile and WSI Classification

This protocol details the methodology for applying a pre-trained visual-language foundation model to classify tissue regions or entire whole-slide images without task-specific training [1].

Table 2: Key reagents and computational tools for zero-shot classification

Item	Specification/Version	Function/Purpose
Pre-trained VLM Weights	CONCH (Hugging Face)	Provides foundational image and text encoders for zero-shot inference [3].
Whole-Slide Image Data	SICAP, TCGA, CRC100k	Forms the input image data for evaluation; should represent target diseases [1].
Text Prompt Templates	Custom-defined ensemble	Converts class names into multiple textual descriptions to enhance prediction robustness [1].
Computational Framework	PyTorch/TensorFlow	Enables model loading, inference, and similarity score calculation.

Procedure:

• Task Definition and Prompt Engineering: Define the classification task and corresponding class names (e.g., "invasive ductal carcinoma," "normal colon mucosa"). Create an ensemble of text prompts for each class to account for variations in terminology. For example: "This is a histology image of {classname}," "A micrograph showing {classname}," etc [1] [4].
• Text Feature Embedding: Use the pre-trained model's text encoder to generate feature embeddings for all text prompts in the ensemble. This projects the textual concepts into the shared visual-text representation space [1].
• Image Preprocessing and Tiling: For whole-slide images, segment the WSI into smaller, manageable tiles at a specified magnification (e.g., 20×). Apply standard normalization to each tile [1] [5].
• Image Feature Embedding: Process each image tile through the pre-trained model's image encoder to obtain visual feature embeddings [1].
• Similarity Calculation and Aggregation: Compute the cosine similarity between each image tile embedding and all text prompt embeddings. For WSI-level prediction, aggregate tile-level similarity scores (e.g., by averaging or using attention mechanisms) to produce a slide-level classification [1] [5].
• Visualization: Generate heatmaps overlaying the WSI, where each tile's color intensity represents its similarity score to the predicted class text prompt, providing visual interpretability for model predictions [1].

Protocol 2: Systematic Prompt Engineering for Diagnostic Pathology

This protocol, derived from recent investigational studies, provides a structured framework for optimizing text prompts to maximize zero-shot diagnostic accuracy [4] [6].

Procedure:

• Dimensional Analysis: Systematically vary prompts along four key dimensions:
  • Domain Specificity (DS): Incorporate low, medium, or high levels of domain-specific terminology [6].
  • Anatomical Precision (AP): Specify anatomical references with low, medium, or high precision (e.g., from "this tissue" to "colonic mucosa") [6].
  • Instructional Framing (IF): Frame the prompt using different styles: Expert (positioning model as expert), Minimal (basic instruction), or Task-oriented [6].
  • Output Constraints (OC): Explicitly define the output format and length, or use implicit, flexible constraints [6].
• Template Design and Testing: Create multiple prompt templates that combine different levels of these dimensions. For example, a high-DS, high-AP, Expert-framed prompt with Explicit output constraints [6].
• Model Inference and Evaluation: Execute zero-shot inference using each prompt template across the target dataset.
• Performance Analysis: Evaluate model performance (e.g., accuracy, F1-score) for each prompt variant. Identify the optimal combination of dimensions for the specific diagnostic task [4].

Recent findings indicate that precise anatomical references and moderate-to-high domain specificity significantly enhance performance, with the CONCH model showing particular sensitivity to these dimensions [4] [6].

Workflow Visualization of Zero-Shot Classification

The following diagram illustrates the integrated workflow for zero-shot classification in computational pathology, combining the protocols for model inference and prompt engineering.

Diagram 1: Zero-shot classification workflow for computational pathology, integrating visual and textual processing pipelines.

Essential Research Reagent Solutions

The successful implementation of zero-shot classification in computational pathology relies on several key computational and data resources, as previously referenced in the experimental protocols.

Table 3: Essential research reagents and resources for zero-shot pathology

Category	Resource	Function in Research
Foundation Models	CONCH [1] [3]	Pre-trained visual-language foundation model for pathology; enables zero-shot transfer to various tasks without fine-tuning.
Foundation Models	PLIP [1]	Pathology-language image pretraining model; serves as a baseline for comparative performance studies.
Foundation Models	BiomedCLIP [1]	Biomedical-specific CLIP model; provides domain-adapted visual-language representations.
Computational Frameworks	MI-Zero [1] [5]	Framework combining visual-language models with multiple instance learning for WSI-level zero-shot classification.
Data Resources	TCGA (The Cancer Genome Atlas) [1] [2]	Provides extensive, well-characterized WSI datasets across multiple cancer types for model evaluation.
Data Resources	Grand Challenge Datasets [2]	Source of public pathology image datasets (e.g., PANDA, CAMELYON) for benchmarking and validation.

What Are Visual-Language Foundation Models? From CLIP to Pathology-Specific Architectures

Visual-Language Foundation Models (VLFMs) represent a transformative class of artificial intelligence systems pre-trained on vast datasets containing both images and associated textual information. Unlike traditional computer vision models trained for specific tasks, foundation models learn generalized representations that can be adapted to numerous downstream applications with minimal task-specific modification [7]. These models fundamentally change how AI systems understand and process visual information by creating a shared embedding space where both images and text can be compared and related through a common representation [8]. This capability is particularly valuable in medical domains like pathology, where diagnostic reasoning inherently combines visual pattern recognition with conceptual knowledge typically expressed in natural language.

The core innovation of VLFMs lies in their ability to perform cross-modal alignment—learning relationships between visual patterns in images and semantic concepts in text. During pre-training, these models process millions of image-text pairs, adjusting their parameters so that embeddings for a histopathology image showing, for instance, invasive ductal carcinoma become positioned close to the text embedding for "invasive ductal carcinoma" in the shared feature space [1]. This training paradigm enables remarkable capabilities including zero-shot classification, where models can recognize categories they were never explicitly trained to identify, by simply comparing image features with text descriptions of the target classes [9].

Evolution of Architectural Paradigms

Foundational Architectures: CLIP and Beyond

The development of VLFMs began with architectures like CLIP (Contrastive Language-Image Pre-training), which established the dual-encoder paradigm that remains influential today [9]. CLIP employs separate image and text encoders trained jointly using a contrastive learning objective that maximizes agreement between matching image-text pairs while minimizing agreement for non-matching pairs [8]. This architecture enables efficient mapping of both visual and textual inputs into a shared embedding space where semantic similarity can be measured using simple cosine distance [9].

Subsequent models like ALIGN scaled this approach using noisier but larger datasets, while CoCa (Contrastive Captioners) incorporated both contrastive and generative objectives, adding a text decoder to enable caption generation capabilities [1]. The transformer architecture has been particularly instrumental in advancing VLFMs, with its attention mechanism allowing models to focus on relevant regions of both images and text, capturing long-range dependencies essential for understanding complex histopathological images [7].

Pathology-Specific Adaptations

While general-purpose VLFMs demonstrated promising capabilities, their application to pathology required significant domain adaptation to address the unique challenges of computational pathology, including gigapixel whole-slide images (WSIs), subtle morphological features, and specialized domain knowledge [8]. This led to the development of pathology-specific VLFMs including:

• CONCH (CONtrastive learning from Captions for Histopathology): Trained on 1.17 million histopathology image-caption pairs, CONCH incorporates both contrastive alignment and captioning objectives, enabling state-of-the-art performance on classification, segmentation, and retrieval tasks [1].
• TITAN (Transformer-based pathology Image and Text Alignment Network): A multimodal whole-slide foundation model pretrained on 335,645 whole-slide images with corresponding pathology reports and synthetic captions. TITAN can extract general-purpose slide representations and generate pathology reports without requiring fine-tuning [10].
• PLIP (Pathology Language Image Pre-training): A community-built VLFM developed using medical Twitter data that serves as an open-source resource for the computational pathology community [8].
• QuiltNet & Quilt-LLaVA: QuiltNet adapts the CLIP framework for pathology using 1M histopathology image-text pairs, while Quilt-LLaVA extends this approach by integrating a large language model for enhanced conversational capabilities [9].

Table 1: Comparison of Pathology-Specific Visual-Language Foundation Models

Model	Architecture	Training Data	Key Capabilities	Parameters
CONCH	CoCa-based	1.17M image-caption pairs	Classification, segmentation, captioning, retrieval	~200M [9]
TITAN	Vision Transformer	335,645 WSIs + reports	Slide representation, report generation, rare cancer retrieval	Not specified [10]
PLIP	CLIP-based	Medical Twitter data	Zero-shot classification, image-text retrieval	~150M [8]
QuiltNet	CLIP-based	1M image-text pairs	Contrastive learning, feature alignment	~150M [9]
Quilt-LLaVA	LLaVA-based	107K Q&A pairs	Visual question answering, conversational AI	~7B [9]

Quantitative Performance Comparison

Evaluation of VLFMs in pathology applications demonstrates their growing capabilities across diverse tasks. On slide-level classification benchmarks, CONCH achieved zero-shot accuracies of 90.7% for NSCLC subtyping and 90.2% for RCC subtyping, outperforming other models by significant margins [1]. On the more challenging task of invasive breast carcinoma (BRCA) subtyping, CONCH achieved 91.3% accuracy while other models performed at near-random levels [1].

The TITAN model has shown particular strength in low-data regimes and rare disease applications, demonstrating robust performance in cancer prognosis and rare cancer retrieval without requiring fine-tuning [10]. In comprehensive evaluations across 14 diverse benchmarks, CONCH consistently outperformed other VLFMs including PLIP, BiomedCLIP, and general-purpose models like OpenAICLIP, establishing new state-of-the-art performance across classification, segmentation, captioning, and retrieval tasks [1].

Table 2: Zero-Shot Classification Performance of CONCH Across Pathology Tasks [1]

Task	Dataset	Evaluation Metric	CONCH Performance	Next Best Model
NSCLC Subtyping	TCGA NSCLC	Balanced Accuracy	90.7%	78.7% (PLIP)
RCC Subtyping	TCGA RCC	Balanced Accuracy	90.2%	80.4% (PLIP)
BRCA Subtyping	TCGA BRCA	Balanced Accuracy	91.3%	55.3% (BiomedCLIP)
Gleason Grading	SICAP	Quadratic Kappa	0.690	0.550 (BiomedCLIP)
Tissue Classification	CRC100K	Balanced Accuracy	79.1%	67.4% (PLIP)

Experimental Protocols for Zero-Shot Classification in Pathology

Protocol 1: Whole-Slide Image Classification Using TITAN

Purpose: To perform slide-level classification of whole-slide images without task-specific fine-tuning.

Materials and Reagents:

• TITAN model (pretrained weights)
• Whole-slide images (WSIs) in standard formats (SVS, NDPI, etc.)
• Computational environment with GPU acceleration (≥16GB VRAM recommended)
• Python libraries: PyTorch, OpenSlide, Whole-Slide Data Loader

Procedure:

• WSI Preprocessing: Load WSIs and extract representative regions of interest (ROIs) at 20× magnification. Convert each ROI to a feature grid using a pretrained patch encoder [10].
• Feature Grid Construction: Spatially arrange patch features in a 2D grid replicating their positions in the original tissue. Apply random cropping to generate views of 16×16 features covering regions of 8,192×8,192 pixels [10].
• Embedding Extraction: Process feature grids through TITAN's vision transformer encoder using attention with linear bias (ALiBi) for long-context extrapolation. Extract slide-level embeddings from the [CLS] token or via pooling operations [10].
• Text Prompt Preparation: Create descriptive text prompts for each target class (e.g., "histopathology image of invasive ductal carcinoma"). Multiple prompt variations can be ensemble for improved performance [9].
• Similarity Calculation: Encode text prompts using TITAN's text encoder. Compute cosine similarity between slide embeddings and all text prompt embeddings.
• Classification: Assign the class label corresponding to the highest similarity score. Generate confidence scores based on similarity values.

Troubleshooting Tips:

• For large WSIs exceeding memory constraints, implement sliding window inference with aggregation.
• If performance is suboptimal, refine text prompts to include more specific pathological terminology.
• Ensure consistent magnification levels during feature extraction to maintain morphological context.

Protocol 2: Fine-Grained Patch Alignment for Brain Tumor Subclassification

Purpose: To enhance zero-shot classification of subtle brain tumor subtypes using patch-level feature alignment.

Materials and Reagents:

• Pretrained VLFMs (CONCH or similar)
• Whole-slide images of brain tumor specimens
• Computational environment with GPU acceleration
• Python libraries: PyTorch, Transformers, OpenSlide

Procedure:

• Patch Sampling: Extract patches of 512×512 pixels at 20× magnification from diagnostically relevant regions of WSIs, ensuring coverage of diverse morphological patterns [11].
• Local Feature Refinement: Process patches through VLFM image encoder. Model spatial relationships between patches using self-attention mechanisms to enhance feature discriminability [11].
• Fine-Grained Text Generation: Leverage large language models (e.g., GPT-4) to generate detailed, pathology-aware descriptions for each tumor subtype. Include specific morphological features (e.g., "nuclear atypia," "serpentine necrosis," "microvascular proliferation") [11].
• Patch-Text Alignment: Compute similarity between refined patch features and text embeddings of class descriptions. Apply transformer-based cross-attention to model fine-grained correspondences [11].
• Uncertainty-Aware Aggregation: Aggregate patch-level predictions using uncertainty-weighted voting, prioritizing patches with higher prediction confidence [11].
• Slide-Level Classification: Generate final slide-level prediction based on aggregated patch scores. Visualize attention maps to highlight diagnostically relevant regions.

Validation:

• Compare predictions with ground truth annotations from certified pathologists.
• Perform qualitative analysis of attention maps to ensure alignment with known pathological features.
• Evaluate robustness through cross-validation across multiple institutions and scanner types.

Research Reagent Solutions

Table 3: Essential Computational Tools for VLFM Research in Pathology

Tool Name	Type	Function	Availability
CONCH	Vision-Language Model	Feature extraction, zero-shot classification, retrieval	Public [1]
TITAN	Whole-Slide Foundation Model	Slide-level representation, report generation	Upon request [10]
PLIP	Pathology Language-Image Model	Open-source visual-language pretraining	Public [8]
Quilt-LLaVA	Large Multimodal Model	Visual question answering, conversational AI	Public [9]
FM²	Model Fusion Framework	Disentangled consensus-divergence representation	Public [8]
FG-PAN	Fine-Grained Alignment	Patch-text alignment for subtle subtypes	Public [11]

Workflow Visualization

Diagram 1: Workflow of Visual-Language Foundation Models in Pathology. This diagram illustrates the complete pipeline from multimodal data input through feature extraction, cross-modal alignment, and diverse zero-shot applications in computational pathology.

Diagram 2: Architectural Paradigms of Pathology VLMs. This diagram compares three predominant architectural frameworks for visual-language foundation models in pathology, highlighting their distinctive components and learning objectives.

Future Directions and Challenges

The development of VLFMs for pathology faces several important challenges that represent active research directions. Computational efficiency remains a significant constraint, as processing gigapixel whole-slide images requires substantial memory and processing resources [10]. Current approaches address this through hierarchical processing and feature distillation, but more efficient architectures are needed for clinical deployment. Interpretability and reliability are particularly crucial in medical applications, where model decisions must be explainable and trustworthy [7]. Techniques like attention visualization and uncertainty quantification are being integrated into modern VLFMs to address these concerns.

Future research directions include the development of specialized foundation models for rare diseases, where limited training data creates particular challenges for general-purpose models [10]. The integration of multimodal patient data beyond images and text—including genomic profiles, clinical variables, and longitudinal outcomes—represents another promising frontier [7]. Finally, federated learning approaches are being explored to enable model training across multiple institutions while preserving data privacy, which is essential for advancing model generalizability while complying with healthcare regulations [7].

As VLFMs continue to evolve, they hold the potential to fundamentally transform pathology practice, not by replacing pathologists but by augmenting their capabilities through powerful tools for pattern recognition, data integration, and knowledge retrieval. The progression from general-purpose models like CLIP to specialized pathology architectures represents an important step toward clinically relevant AI systems that understand both the visual language of histopathology and the semantic language of human diagnosis.

Contrastive learning is a self-supervised machine learning paradigm that trains models to distinguish between similar (positive) and dissimilar (negative) data pairs. In computational pathology, this framework is powerfully applied to create a unified embedding space where histopathology images and textual descriptions can be directly compared. The core objective is to maximize agreement between matching image-text pairs while minimizing agreement between non-matching pairs. This enables models to learn rich, transferable representations of histopathological tissue morphology without relying on extensive manual annotations, thereby directly facilitating zero-shot classification capabilities where models can recognize and categorize pathological concepts without task-specific training data.

Visual-language foundation models like CONCH (Contrastive learning from Captions for Histopathology) exemplify this approach, having been pretrained on over 1.17 million histopathology image-caption pairs to create a joint embedding space where semantically similar images and texts are positioned close together regardless of whether they were explicitly seen during training [12]. This architecture enables powerful applications including cross-modal retrieval, where a text query can find relevant images or vice versa, and zero-shot classification, where natural language descriptions of diseases can be used to categorize unseen histopathology images without additional training.

Key Technical Principles and Architectures

Core Architectural Frameworks

Visual-language foundation models in pathology typically employ dual-encoder architectures consisting of separate image and text encoders trained with contrastive objectives:

• Image Encoder: A Vision Transformer (ViT) processes tissue regions-of-interest (ROIs) or whole-slide images, converting visual patterns into embedding vectors. Models like CONCH use a ViT-B architecture, while others like UNI employ ViT-H [13].
• Text Encoder: A transformer-based language model processes descriptive text, such as pathology reports or synthetic captions, converting natural language into the same embedding space.
• Contrastive Objective: The training maximizes cosine similarity between corresponding image-text pairs while minimizing similarity for non-corresponding pairs. For a batch of N image-text pairs, the contrastive loss function treats the N matched pairs as positives and the N²-N mismatched pairs as negatives.

Multimodal Alignment Strategies

Advanced models employ sophisticated alignment techniques to bridge visual and linguistic domains:

• Cross-modal Attention: Models like TITAN (Transformer-based pathology Image and Text Alignment Network) use attention mechanisms to align image patches with relevant text tokens, enabling fine-grained correspondence between morphological features and descriptive concepts [10] [14].
• Knowledge Distillation: Some frameworks leverage pretrained patch encoders like CONCHv1.5 to extract visual features before multimodal alignment, creating hierarchical representations [10].
• Synthetic Data Augmentation: TITAN incorporates 423,122 synthetic captions generated from a multimodal generative AI copilot to enhance training diversity and robustness [10] [14].

Table 1: Contrastive Learning Architectures in Computational Pathology

Model	Architecture	Training Data	Embedding Dimension	Parameters
CONCH	ViT-B + Text Encoder	1.17M image-text pairs	512	~200M [13] [12]
TITAN	ViT + Transformer	335,645 WSIs + 423K synthetic captions	768	Not specified [10]
Quilt-Net	ViT-B/32 + GPT-2	1M image-text samples	512	~150M [9]
Quilt-LLaVA	Visual encoder + LLM	~107K QA pairs	Not specified	~7B [9]

Quantitative Benchmarking of Model Performance

Zero-Shot Classification Capabilities

Comprehensive evaluations demonstrate the practical efficacy of contrastively trained visual-language models across diverse pathological tasks. In systematic assessments of diagnostic accuracy across 3,507 digestive system whole-slide images, the CONCH model achieved highest accuracy when provided with precise anatomical references, with performance consistently degrading when anatomical precision was reduced [9]. This highlights the critical importance of prompt design and anatomical context in leveraging these models for zero-shot histopathological image analysis.

The TITAN model demonstrates exceptional versatility in zero-shot settings, outperforming both region-of-interest and slide foundation models across multiple machine learning settings including linear probing, few-shot and zero-shot classification, rare cancer retrieval, cross-modal retrieval and pathology report generation [10] [14]. This generalizability is particularly valuable for resource-limited clinical scenarios involving rare diseases with limited training data.

Cross-Modal Retrieval Performance

The joint embedding space enables powerful retrieval capabilities between visual and textual domains:

• Image-to-Text Retrieval: Given a histopathology image, models can retrieve relevant pathological descriptions or generate comprehensive reports.
• Text-to-Image Retrieval: Natural language queries describing morphological findings can retrieve visually similar tissue regions from large databases.

Table 2: Performance Benchmarks of Visual-Language Models in Pathology

Model	Zero-shot Accuracy (%)	Top-1 Retrieval Rate	Training Paradigm	Report Generation Quality
CONCH	15-20% gain over non-VLMs [13]	State-of-the-art [12]	Vision-Language Contrastive	Not specified
TITAN	Outperforms ROI/slide models [10]	Enables rare cancer retrieval [10]	Multimodal SSL + V-L Alignment	Generates clinically relevant reports [10]
Quilt-Net	Varies with prompt design [9]	Not specified	CLIP-based fine-tuning	Not applicable
PLIP	Competitive on subtype classification [13]	Effective on public datasets [13]	Vision-Language Contrastive	Not applicable

Experimental Protocols for Zero-Shot Classification

Protocol: Zero-Shot Diagnostic Classification Using Pre-trained VLMs

Purpose: To evaluate the zero-shot classification performance of visual-language foundation models on histopathology whole-slide images without task-specific fine-tuning.

Materials:

• Pre-trained visual-language model (CONCH, TITAN, or PLIP)
• Whole-slide images (WSIs) for evaluation
• Textual prompts describing target diseases/conditions
• Computational resources (GPU recommended)
• Python environment with necessary libraries (PyTorch, Transformers, OpenSlide)

Procedure:

• Image Preprocessing:
  • Load WSIs and extract tissue regions using Otsu's algorithm for foreground detection [13].
  • For whole-slide models like TITAN, divide WSIs into non-overlapping patches (e.g., 512×512 pixels at 20× magnification) [10].
  • Extract feature embeddings using the model's visual encoder.

• Prompt Engineering:

  • Design text prompts incorporating domain-specific terminology, anatomical precision, and instructional framing [9].
  • Example prompts: "H&E stain of colon tissue showing invasive adenocarcinoma with desmoplastic reaction" or "Lymph node tissue with metastatic carcinoma deposits".
  • Generate text embeddings using the model's text encoder.

• Similarity Calculation:

  • Compute cosine similarity between image embeddings and all candidate text prompt embeddings in the joint space.
  • For patch-level features, aggregate similarities across all patches in a WSI using attention pooling or max pooling.

• Classification:

  • Assign the class label corresponding to the text prompt with highest similarity score.
  • Set confidence thresholds based on similarity scores for reliable predictions.

Validation:

• Compare predictions with ground truth diagnoses from pathologists.
• Calculate standard metrics (accuracy, F1-score, AUC-ROC) across different tissue types and disease conditions.
• Perform qualitative analysis of attention maps to verify diagnostically relevant regions [9].

Protocol: Cross-Modal Retrieval for Rare Cancer Identification

Purpose: To retrieve histologically similar cases from a database using text descriptions for rare cancer subtyping.

Materials:

• Database of whole-slide images with diagnostic annotations
• Pre-trained multimodal foundation model (TITAN recommended for slide-level retrieval)
• Clinical descriptions of rare cancer phenotypes

Procedure:

• Database Indexing:
  • Precompute and index visual embeddings for all WSIs in the database using the model's image encoder.
  • Store embeddings in a search-efficient data structure (e.g., FAISS index).

• Query Processing:

  • Encode textual description of rare cancer using the model's text encoder.
  • Alternatively, use a representative image patch as query if textual description is insufficient.

• Similarity Search:

  • Perform k-nearest neighbor search in the joint embedding space.
  • Retrieve top-k most similar cases based on cosine similarity.

• Validation:

  • Assess retrieval precision using expert-annotated ground truth.
  • Evaluate clinical utility through pathologist concordance studies [9].

Implementation Workflows and Signaling Pathways

Contrastive Learning Workflow for Pathology Images and Text

Zero-Shot Classification Pipeline for Pathology

Research Reagent Solutions for Implementation

Table 3: Essential Research Reagents for Visual-Language Pathology Research

Resource Category	Specific Examples	Function/Application	Availability
Pre-trained Models	CONCH, TITAN, PLIP, Quilt-Net	Provide foundation for zero-shot classification and retrieval	Publicly available on GitHub, Hugging Face [12]
Pathology Datasets	TCGA, Quilt-1M, OpenPath, PMC-Patients-DD	Benchmark model performance and train custom encoders	Public access with restrictions [9] [13]
Annotation Tools	QuPath, ImageJ, ASAP	Manual annotation for validation studies	Open source
Computational Frameworks	PyTorch, Transformers, FAISS	Model implementation, training, and efficient similarity search	Open source
Visualization Libraries	matplotlib, Plotly, SlideMap	Embedding visualization and result interpretation	Open source

The diagnosis of diseases from tissue samples, or histopathology, is undergoing a revolutionary transformation through artificial intelligence. Foundation models, pre-trained on vast datasets, are enabling new capabilities in computational pathology. Among these, visual-language models such as CONCH (CONtrastive learning from Captions for Histopathology) and TITAN (Transformer-based pathology Image and Text Alignment Network) represent a paradigm shift. By learning from both histopathology images and their corresponding textual descriptions, these models achieve remarkable zero-shot classification capabilities—performing diagnostic tasks without task-specific training data. This application note details the technical specifications, performance benchmarks, and experimental protocols for leveraging CONCH and TITAN in pathology research and drug development.

CONCH and TITAN are visual-language foundation models specifically designed for computational pathology, but they employ distinct architectural approaches and training methodologies.

CONCH is a vision-language foundation model pre-trained on 1.17 million histopathology image-caption pairs, the largest such dataset at its time of development [1] [15]. Its architecture is based on the CoCa (Contrastive Captioner) framework, integrating an image encoder, a text encoder, and a multimodal fusion decoder [1]. The model is trained using a combination of contrastive alignment objectives that align image and text modalities in a shared representation space, and a captioning objective that learns to generate captions corresponding to images [1]. This dual approach enables CONCH to perform both image-text retrieval and classification tasks effectively.

TITAN represents a more recent advancement as a multimodal whole-slide foundation model pretrained on 335,645 whole-slide images [10]. Its pretraining strategy consists of three stages: (1) vision-only unimodal pretraining on region-of-interest (ROI) crops, (2) cross-modal alignment of generated morphological descriptions at the ROI-level using 423,122 synthetic captions, and (3) cross-modal alignment at the whole-slide image level with corresponding pathology reports [10]. A cornerstone of TITAN's architecture is its use of a Vision Transformer (ViT) that operates on pre-extracted patch features from whole-slide images, employing attention with linear bias (ALiBi) for long-context extrapolation [10].

Table 1: Technical Specifications of CONCH and TITAN

Specification	CONCH	TITAN
Model Type	Visual-language encoders	Multimodal whole-slide Vision Transformer
Primary Innovation	Contrastive learning from captions	Hierarchical whole-slide encoding with synthetic data
Training Data	1.17M image-caption pairs [1]	335,645 WSIs + 423K synthetic captions + 183K reports [10]
Vision Encoder	ViT-B/16 (90M params) [15]	Vision Transformer (ViT)
Text Encoder	L12-E768-H12 (110M params) [15]	Integrated transformer architecture
Multimodal Alignment	Image-text contrastive + captioning loss [1]	Vision-language alignment at ROI and WSI levels [10]
Key Capabilities	Classification, segmentation, retrieval, captioning	Slide representation, report generation, rare cancer retrieval

Performance Benchmarks and Comparative Analysis

Both CONCH and TITAN have demonstrated state-of-the-art performance across diverse pathology tasks, particularly in zero-shot settings where no task-specific training is required.

Zero-Shot Classification Performance

In comprehensive evaluations, CONCH has shown superior zero-shot classification capabilities compared to other visual-language foundation models. On slide-level cancer subtyping tasks, CONCH achieved remarkable accuracy: 90.7% on non-small cell lung cancer (NSCLC) subtyping, 90.2% on renal cell carcinoma (RCC) subtyping, and 91.3% on invasive breast carcinoma (BRCA) subtyping [1]. This represents a performance improvement of 9.8-35% over other models like PLIP and BiomedCLIP across these tasks [1]. On the more challenging lung adenocarcinoma (LUAD) pattern classification task, CONCH achieved a Cohen's κ of 0.200, outperforming other models [1].

TITAN has demonstrated exceptional performance in resource-limited clinical scenarios, including rare disease retrieval and cancer prognosis [10]. In evaluations across diverse clinical tasks, TITAN outperformed both region-of-interest and slide foundation models across multiple machine learning settings, including linear probing, few-shot, and zero-shot classification [10]. The model's pretraining with synthetic fine-grained morphological descriptions has shown particular utility for rare cancer retrieval tasks [10].

Performance on Non-Neoplastic Pathology

Recent benchmarking studies have evaluated how these pathology foundation models generalize beyond cancer to non-neoplastic diseases. In placental pathology benchmarks—doubly out-of-distribution for these models as placental data was largely absent from their training—pathology foundation models still outperformed general-purpose models [16]. However, the performance gap between pathology and non-pathology models diminished in tasks related to inflammation, suggesting areas for future improvement [16].

Table 2: Zero-Shot Classification Performance Across Pathology Tasks

Task/Dataset	Model	Performance Metric	Score	Comparative Advantage
TCGA NSCLC Subtyping	CONCH	Balanced Accuracy	90.7% [1]	+12.0% over next best (PLIP)
TCGA RCC Subtyping	CONCH	Balanced Accuracy	90.2% [1]	+9.8% over next best (PLIP)
TCGA BRCA Subtyping	CONCH	Balanced Accuracy	91.3% [1]	~35% over other models
SICAP (Gleason Patterns)	CONCH	Quadratic κ	0.690 [1]	+0.140 over BiomedCLIP
Rare Cancer Retrieval	TITAN	Retrieval Accuracy	Superior performance [10]	Outperforms other slide foundation models
Placental Gestational Age	Pathology FMs	KNN Regression	Best performance [16]	Outperforms non-pathology models

Experimental Protocols and Workflows

Protocol 1: Zero-Shot Whole-Slide Image Classification

Principle: Leverage pretrained visual-language models to classify entire gigapixel whole-slide images without task-specific training.

Procedure:

• WSI Tiling: Divide the whole-slide image into smaller, manageable patches at 20× magnification (e.g., 256×256 or 512×512 pixels) [1].
• Feature Extraction: For each patch, extract image embeddings using the vision encoder of either CONCH or TITAN.
• Prompt Engineering: Create a set of text prompts describing the candidate diagnostic classes. Employ prompt ensembles with multiple phrasings for each class to enhance robustness [1] [4].
• Text Embedding: Encode all text prompts using the model's text encoder to obtain text embeddings in the shared vision-language space.
• Similarity Calculation: Compute cosine similarity scores between each image patch embedding and all text embeddings.
• Score Aggregation: Use an aggregation strategy (e.g., mean, max, or attention-based pooling) across all patches to generate a slide-level prediction [1] [17].

Protocol 2: Cross-Modal Slide Retrieval

Principle: Retrieve the most relevant whole-slide images based on textual queries, or generate textual descriptions based on slide content.

Procedure:

• Database Building: Extract and store feature embeddings for all whole-slide images in the target database using the model's vision encoder.
• Query Processing: For text-to-slide retrieval, encode the natural language query using the model's text encoder. For slide-to-text retrieval, use the slide's feature embedding.
• Similarity Search: Compute similarity scores (cosine similarity) between the query embedding and all database embeddings.
• Result Ranking: Rank database entries by similarity scores and return the top-k matches.
• Report Generation (TITAN-specific): Leverage TITAN's capability to generate pathology reports from whole-slide images by decoding from the aligned multimodal representation [10].

Protocol 3: Fine-Grained Patch-Text Alignment for Challenging Subtypes

Principle: Enhance zero-shot classification of morphologically similar subtypes through localized patch-text alignment and spatial reasoning.

Procedure:

• Local Feature Refinement: Implement a local feature refinement module (e.g., using local window attention) to enhance patch-level visual features by modeling spatial relationships among representative patches [17].
• Fine-Grained Text Generation: Leverage large language models (LLMs) to generate pathology-aware, fine-grained class descriptions that capture subtle morphological distinctions [17].
• Patch-Text Alignment: Align refined visual features with LLM-generated fine-grained descriptions in the shared embedding space.
• Uncertainty-Aware Aggregation: Aggregate patch-level predictions using uncertainty-aware strategies to generate slide-level diagnoses [17].

The Scientist's Toolkit: Essential Research Reagents

Table 3: Essential Computational Tools for Pathology Visual-Language Research

Tool/Resource	Function	Access Information
CONCH Model Weights	Pre-trained weights for the CONCH model for feature extraction and zero-shot inference	Available via Hugging Face Hub after request and approval [15]
TITAN Framework	Architecture and methodology for whole-slide representation learning	Implementation details in original publication [10]
Whole-Slide Image Datasets	Benchmark datasets for model evaluation (e.g., TCGA, CAMELYON)	Publicly available with appropriate data use agreements
Pathology Language Prompts	Curated text prompts for pathological concepts and diagnoses	Generated using domain knowledge and LLMs [17]
Synthetic Caption Generators	Tools for generating fine-grained morphological descriptions	PathChat and other multimodal generative AI copilots [10]

Implementation Considerations and Best Practices

Prompt Engineering for Optimal Performance

Effective prompt engineering is critical for maximizing zero-shot performance. Studies have demonstrated that prompt engineering significantly impacts model performance, with the CONCH model achieving highest accuracy when provided with precise anatomical references [4]. Key strategies include:

• Anatomical Precision: Incorporate specific anatomical references relevant to the diagnostic task
• Domain Specificity: Use appropriate pathological terminology and nomenclature
• Prompt Ensembling: Employ multiple phrasings for the same concept to enhance robustness
• Instructional Framing: Structure prompts with clear instructional context when appropriate

Computational Infrastructure Requirements

Implementing these models requires substantial computational resources, particularly for whole-slide image analysis:

• Hardware: GPU acceleration (e.g., NVIDIA A100 or equivalent) is essential for efficient inference
• Memory: Large RAM capacity (≥64GB) is recommended for handling gigapixel whole-slide images
• Storage: High-speed SSD storage facilitates efficient loading and processing of large whole-slide image files

Integration with Existing Pathology Workflows

Successful deployment requires thoughtful integration with existing digital pathology infrastructure:

• DICOM Compatibility: Ensure compatibility with standard whole-slide image formats
• LIS Integration: Develop interfaces with Laboratory Information Systems for seamless clinical workflow integration
• Validation Frameworks: Implement rigorous validation protocols against ground truth pathological diagnoses

The development of CONCH and TITAN represents a significant milestone in computational pathology, demonstrating that visual-language foundation models can achieve remarkable zero-shot classification capabilities across diverse pathological diagnoses. These models offer particular promise for rare diseases and low-resource scenarios where annotated training data is scarce.

Future research directions include expansion to non-neoplastic pathologies, integration with multi-omics data, development of interactive agentic systems for slide navigation [18], and advancement of prompt optimization techniques. As these technologies mature, they hold potential to augment pathological diagnosis, enhance diagnostic consistency, and accelerate drug development workflows.

The protocols and guidelines presented in this application note provide researchers with practical methodologies for leveraging these pioneering models in their computational pathology research.

Visual-language foundation models represent a paradigm shift in computational pathology, moving from single-task, supervised models to versatile, general-purpose tools. Their key advantages of generalizability and task agnosticism are primarily enabled through large-scale, self-supervised pretraining on diverse multimodal datasets [10] [19]. These models demonstrate robust performance across unseen tasks without task-specific fine-tuning, making them particularly valuable for research and drug development applications where labeled data is scarce and hypothesis generation is critical [20]. The following application note details the quantitative performance, experimental protocols, and practical implementation of these capabilities, with a specific focus on zero-shot classification scenarios.

Quantitative Performance Benchmarking

Foundation models consistently outperform traditional supervised approaches, especially in low-data regimes and zero-shot settings. The table below summarizes key performance metrics across diverse pathology tasks.

Table 1: Performance Benchmarking of Pathology Foundation Models

Model	Pretraining Data	Task Type	Performance Metric	Result	Significance
TITAN [10]	335,645 WSIs + 423k synthetic captions	Rare Cancer Retrieval	Not specified	Outperforms baselines	Generalizability to rare diseases with limited data
TITAN [10]	335,645 WSIs + pathology reports	Zero-shot Classification	Not specified	Superior to ROI/slide models	Effective without clinical labels or fine-tuning
CONCH [9]	1.17M image-text pairs [21]	Cancer Invasiveness (Zero-shot)	Accuracy	Highest with precise prompts	Demonstrates critical impact of prompt design
Prov-GigaPath [19]	1.3B patches from 171k slides	Pan-Cancer Subtyping	State-of-the-art	25/26 tasks	Generalist capability across cancers and institutions
Virchow [19]	Millions of tissue images	Tumor Detection (9 common & 7 rare cancers)	AUC	0.95	Label efficiency and generalization to rare cancers

Experimental Protocols for Zero-Shot Classification

Protocol: Zero-Shot Slide-Level Classification Using Prompt Ensembling

This protocol enables disease classification without task-specific model retraining by leveraging the model's inherent semantic understanding [9].

I. Research Reagent Solutions Table 2: Essential Components for Zero-Shot Classification

Component	Function	Example / Specification
Visual-Language Model (VLM)	Encodes images and text into a shared embedding space.	CONCH [9] [21] or TITAN [10] pretrained weights.
Whole-Slide Image (WSI)	The input gigapixel digital pathology image.	Formalin-fixed, paraffin-embedded (FFPE) tissue section, H&E stained.
Text Prompt Templates	Define the classification classes in natural language.	"A whole-slide image of [ANATOMY] with [CLASS_LABEL]."
Prompt Engineering Framework	Systematically varies prompt structure to improve robustness.	Modulates domain specificity, anatomical precision, and instructional framing [9].

II. Procedure

• WSI Preprocessing: Segment the gigapixel WSI into smaller, non-overlapping patches (e.g., 512x512 pixels at 20x magnification) [10]. Extract feature embeddings for each patch using the model's vision encoder.
• Prompt Design: For each potential class (e.g., "invasive carcinoma," "high-grade dysplasia," "normal tissue"), generate multiple text prompts. Systematically vary:
  • Anatomical Precision: "colon tissue" vs. "muscular colon wall."
  • Domain Specificity: "showing signs of malignancy" vs. "with adenocarcinoma."
  • Instructional Framing: "Is this image representative of [CLASS]?" vs. "This is a histology image showing [CLASS]." [9]
• Feature Encoding: Process all text prompts through the model's text encoder to obtain a set of text embeddings, ( E{text}^{1}, E{text}^{2}, ..., E_{text}^{N} ), where ( N ) is the number of class-prompt combinations.
• Similarity Calculation: For a given WSI, compute the similarity between the aggregate WSI image embedding and each text embedding. This is typically done using cosine similarity in the shared multimodal embedding space.
• Classification: The final prediction is the class associated with the text prompt achieving the highest similarity score. For robustness, aggregate results across an ensemble of the best-performing prompts [9].

Diagram 1: Zero-Shot Classification Workflow.

Protocol: Cross-Modal Retrieval for Rare Cancer Identification

This protocol identifies histologically similar cases or relevant reports from a database, crucial for rare disease research and drug target identification [10] [19].

I. Research Reagent Solutions Table 3: Essential Components for Cross-Modal Retrieval

Component	Function	Example / Specification
Multimodal Embedding Database	A searchable repository of feature vectors from WSIs and reports.	Vector database (e.g., FAISS) containing TITAN-generated slide and text embeddings [10].
Query (Image or Text)	The input used to search the database.	A WSI of a rare cancer subtype or a free-text morphological description.
Similarity Metric	Algorithm to find the closest matches in the embedding space.	Cosine similarity or Euclidean distance.

II. Procedure

• Database Construction: Process a large repository of WSIs and their corresponding pathology reports using a multimodal foundation model (e.g., TITAN) to generate paired image and text embeddings. Store these embeddings in a indexed vector database [10].
• Query Processing: For a novel WSI (image query) or a text-based morphological description (text query), process it with the same model to generate its representative embedding.
• Nearest Neighbor Search: Execute a search in the vector database to find the pre-computed embeddings with the smallest distance to the query embedding.
• Result Retrieval: Return the top-k most similar WSIs or pathology reports based on the similarity metric. This enables researchers to find rare cases with similar morphological phenotypes or clinical descriptions, aiding in hypothesis generation and cohort building [10] [19].

Implementation and Integration

The generalizability of foundation models translates into practical advantages for the research and drug development pipeline. They function as a single, reusable backbone, drastically reducing the need for annotated data and specialized model development for each new task [19] [20]. For instance, a model like UNI, pretrained on 100 million patches, can be adapted to 34 different clinical tasks, from cancer subtyping to inflammation analysis, outperforming task-specific models [19]. This "pretrain-once, adapt-to-many" approach accelerates iterative research cycles. Furthermore, their task-agnostic nature is key for discovering novel morphological biomarkers; by analyzing tissue at scale, models can identify subtle patterns and correlations with molecular data (e.g., inferring MSI status from H&E images) that are imperceptible to the human eye, thus opening new avenues for drug target discovery and patient stratification [19] [20].

Implementing Zero-Shot Classification: Architectures, Training, and Real-World Applications

Visual-language foundation models represent a transformative advancement in computational pathology by learning to associate histopathological imagery with descriptive clinical text. These models develop a shared semantic space where visual patterns from Whole Slide Images (WSIs) and textual concepts from pathology reports can be directly compared and integrated. This capability is particularly valuable for zero-shot classification, where models can recognize and categorize pathological findings without task-specific training data by leveraging their cross-modal understanding. The architecture of these systems typically comprises three core components: image encoders that process gigapixel WSIs, text encoders that interpret clinical language, and fusion mechanisms that create aligned representations across modalities. Current research demonstrates that models like TITAN (Transformer-based pathology Image and Text Alignment Network) can generate general-purpose slide representations applicable to diverse clinical scenarios including rare disease retrieval and cancer prognosis without requiring fine-tuning or clinical labels [10] [14].

Model Architectures: Components and Functionality

Image Encoders for Whole Slide Images

Image encoders for computational pathology must address the significant challenge of processing gigapixel Whole Slide Images (WSIs) that can exceed 1GB in size while preserving critical diagnostic information at multiple scales. The predominant approach involves a two-stage feature extraction process that first encodes local regions then aggregates these into slide-level representations.

The TITAN model employs a Vision Transformer (ViT) architecture that operates on pre-extracted patch features rather than raw pixels. The input embedding space is constructed by dividing each WSI into non-overlapping patches of 512×512 pixels at 20× magnification, with 768-dimensional features extracted for each patch using specialized histopathology encoders like CONCHv1.5. To handle computational complexity, TITAN creates views of a WSI by randomly cropping the 2D feature grid, sampling region crops of 16×16 features covering 8,192×8,192 pixel regions. For self-supervised pretraining, it samples two random global (14×14) and ten local (6×6) crops from these region crops [10].

MPath-Net utilizes Multiple Instance Learning (MIL) for WSI feature extraction, treating each slide as a "bag" of patch-level instances. This approach leverages attention-based pooling mechanisms like ABMIL, ACMIL, TransMIL, and DSMI to aggregate patch information without requiring localized annotations. These methods identify diagnostically relevant regions and weight their contributions accordingly, enabling slide-level classification from weakly supervised data [22].

For nuclei segmentation in pathology images, advanced encoder architectures incorporate modules like Dense-CA (Dense Channel Attention) within U-Net based encoder-decoder frameworks. This module improves feature extraction in complex backgrounds by adaptively emphasizing relevant cellular features and reducing information loss during downsampling. The Multi-scale Transformer Attention (MSTA) module further enhances boundary segmentation by fusing features between encoder and decoder using Transformer-based feature fusion across different scales [23].

Table 1: Comparative Analysis of Image Encoder Architectures in Pathology

Architecture	Input Processing	Key Components	Output Representation	Computational Considerations
TITAN ViT	512×512 patches → 768D features	Vision Transformer, ALiBi positional encoding, knowledge distillation	General-purpose slide embeddings	Handles long sequences (>10⁴ tokens), random feature cropping
MPath-Net MIL	Patch-level feature extraction	Attention-based pooling, instance-level weighting	Slide-level classification scores	Reduces annotation dependency, memory-efficient
Dense-CA/MSTA	Full resolution patches	Dense Channel Attention, Multi-scale Transformer	Nuclei segmentation masks	Optimized for boundary accuracy, handles density

Text Encoders for Pathology Reports

Text encoders process clinical narratives from pathology reports to extract semantically meaningful representations that can be aligned with visual features. These encoders must handle specialized medical terminology, variable reporting styles, and the implicit relationships between morphological descriptions and diagnostic conclusions.

In the MPath-Net framework, Sentence-BERT (Bidirectional Encoder Representations from Transformers) generates embeddings from pathology report text. This approach leverages transfer learning from biomedical literature to create 512-dimensional text representations that capture clinical semantics. The encoder remains frozen during multimodal training to preserve pretrained contextual representations while enabling effective integration with visual features [22].

TITAN employs a two-stage alignment process that first associates generated morphological descriptions at the region-of-interest (ROI) level, then performs cross-modal alignment at the whole-slide level. The model was fine-tuned using 423,122 synthetic captions generated from PathChat, a multimodal generative AI copilot for pathology, in addition to 182,862 medical reports. This approach enables the model to learn fine-grained correspondences between visual patterns and textual descriptions [10].

Specialized biomedical language models like ClinicalBERT have also been applied to pathology report encoding, demonstrating superior performance on medical concept extraction compared to general-domain models. These models are pretrained on large corpora of clinical text, allowing them to capture nuances of medical documentation, including abbreviations, differential diagnoses, and structured reporting elements [22].

Multimodal Fusion Mechanisms

Multimodal fusion mechanisms integrate visual and textual representations to create a shared semantic space where cross-modal reasoning and retrieval can occur. The design of these fusion components critically impacts model performance on downstream tasks like zero-shot classification and cross-modal retrieval.

TITAN employs vision-language alignment through contrastive learning at both region and slide levels. The model aligns image and text representations by maximizing the similarity between matching pairs while minimizing similarity for non-matching pairs. This approach enables zero-shot capabilities where text prompts can be directly matched with visual patterns without task-specific training. The alignment is performed after the vision-only pretraining stage, allowing the model to first learn robust visual representations before incorporating linguistic information [10] [14].

MPath-Net utilizes feature-level fusion (a form of intermediate fusion) where 512-dimensional image and text embeddings are concatenated and passed through trainable layers that learn cross-modal interactions. This approach allows joint reasoning over both visual and textual signals while maintaining the integrity of modality-specific representations. The framework employs an end-to-end training process where the image classifier and downstream fusion layers are trained jointly, enabling the model to learn synergistic representations across modalities [22].

Advanced fusion strategies also include transformer-based fusion where cross-attention mechanisms allow elements from each modality to attend to relevant parts of the other modality. This approach enables fine-grained alignment between specific visual features and textual concepts, which is particularly valuable for interpretability and localization of diagnostically relevant regions [24].

Experimental Protocols and Benchmarking

Model Training and Implementation

The development of effective visual-language models for pathology requires carefully designed training protocols that address data limitations, computational constraints, and clinical requirements.

TITAN Pretraining Protocol: The TITAN model undergoes a three-stage pretraining process on the Mass-340K dataset comprising 335,645 WSIs and 182,862 medical reports across 20 organ types. Stage 1 involves vision-only unimodal pretraining using the iBOT framework (masked image modeling and knowledge distillation) on region crops. Stage 2 performs cross-modal alignment of generated morphological descriptions at the ROI-level using 423,122 synthetic caption-ROI pairs. Stage 3 conducts cross-modal alignment at the WSI-level using slide-report pairs. The model uses Attention with Linear Biases (ALiBi) for long-context extrapolation, with linear bias based on relative Euclidean distance between features in the 2D feature grid [10].

MPath-Net Training Protocol: Implementation begins with feature extraction where WSIs are processed using a Multiple Instance Learning approach and pathology reports are encoded using Sentence-BERT. The model then concatenates 512-dimensional image and text embeddings, passing them through custom fine-tuning layers for tumor classification. The training employs joint optimization where the image encoder is initialized with self-supervised weights and remains trainable, while the text encoder remains frozen to preserve linguistic representations. The framework was evaluated on TCGA dataset (1,684 cases: 916 kidney, 768 lung) using standard cross-validation protocols [22].

PathPT Few-shot Adaptation: For rare cancer subtyping with limited data, PathPT implements spatially-aware visual aggregation and task-specific prompt tuning. This approach converts WSI-level supervision into fine-grained tile-level guidance by leveraging the zero-shot capabilities of vision-language models. The method preserves localization on cancerous regions and enables cross-modal reasoning through prompts aligned with histopathological semantics, addressing the key limitation of conventional MIL methods which overlook cross-modal knowledge [24].

Table 2: Performance Benchmarks of Multimodal Pathology Models

Model	Training Data	Zero-shot Accuracy	Few-shot Accuracy	Cross-modal Retrieval	Report Generation
TITAN	335,645 WSIs + 423K synthetic captions + 183K reports	Superior to ROI/slide foundations models across settings	Outperforms baselines in low-data regimes	Enables slidereport retrieval	Generates pathological descriptions
MPath-Net	TCGA (1,684 cases)	Not reported	94.65% accuracy, 0.9553 precision, 0.9472 recall, 0.9473 F1-score	Not primary focus	Not supported
PathPT	2,910 WSIs across 56 rare cancer subtypes	Leverages VL foundation models	Substantial gains in subtyping accuracy and region grounding	Preserves cross-modal localization	Not primary focus

Evaluation Metrics and Benchmarks

Rigorous evaluation of visual-language models in pathology requires diverse assessment strategies across multiple clinical tasks and data regimes.

Zero-shot and Few-shot Evaluation: Models are tested on their ability to recognize novel disease categories without task-specific training (zero-shot) or with very limited examples (few-shot). TITAN was evaluated across diverse clinical tasks including cancer subtyping, biomarker prediction, and outcome prognosis, demonstrating superior performance compared to both region-of-interest and slide foundation models. In few-shot settings, the model maintained strong performance with limited training samples, particularly valuable for rare diseases [10] [14].

Cross-modal Retrieval: This evaluation measures the model's ability to retrieve relevant pathology reports given a query WSI (and vice versa). TITAN demonstrated effective cross-modal retrieval capabilities, enabling clinicians to find similar cases based on either visual or textual queries. This functionality supports diagnostic decision-making by identifying clinically comparable cases [10].

Rare Cancer Retrieval: Specialized evaluation was conducted on rare cancer subtyping, where PathPT was benchmarked on eight rare cancer datasets (four adult and four pediatric) spanning 56 subtypes and 2,910 WSIs. The framework consistently delivered superior performance, achieving substantial gains in subtyping accuracy and cancerous region grounding ability compared to conventional MIL frameworks and vision-language models [24].

Architectural Visualizations

TITAN Model Architecture

MPath-Net Multimodal Fusion Framework

Table 3: Essential Research Resources for Pathology Visual-Language Models

Resource Category	Specific Examples	Function in Research	Key Characteristics
WSI Datasets	Mass-340K (335,645 WSIs), TCGA (1,684 kidney/lung cases), Rare Cancer Benchmarks (8 datasets, 56 subtypes)	Model pretraining and evaluation	Multi-organ, diverse stains, scanner variants, rare disease representation
Text Corpora	Pathology reports (183K), Synthetic captions (423K via PathChat)	Cross-modal alignment, report generation	Clinical narratives, fine-grained morphological descriptions
Image Encoders	CONCHv1.5, Self-supervised patch encoders, DenseNet variants	Feature extraction from histology patches	768D feature output, pretrained on histology data
Text Encoders	Sentence-BERT, ClinicalBERT, Specialized biomedical PLMs	Text representation learning	Domain-specific pretraining, clinical concept capture
Fusion Frameworks	TITAN, MPath-Net, PathPT	Multimodal integration, zero-shot transfer	Cross-modal attention, feature concatenation, prompt tuning
Evaluation Suites	Cancer subtyping tasks, Rare cancer retrieval, Cross-modal retrieval	Performance benchmarking	Multiple few-shot settings, diverse cancer types

Visual-language foundation models represent a paradigm shift in computational pathology, enabling zero-shot classification and cross-modal retrieval without extensive task-specific training. The architectural principles embodied in models like TITAN, MPath-Net, and PathPT demonstrate that effective integration of image encoders, text encoders, and fusion mechanisms can yield powerful general-purpose representations applicable across diverse clinical scenarios. These approaches are particularly valuable for addressing the critical challenge of rare disease diagnosis where limited training data constrains conventional deep learning methods.

Future research directions include scaling pretraining with synthetic data, developing more efficient architectures for gigapixel image processing, enhancing interpretability through better alignment between visual and textual concepts, and expanding clinical validation across broader disease spectra. As these models mature, they hold significant potential to reduce diagnostic variability, support pathologists in challenging cases, and ultimately improve patient care through more accurate and accessible computational pathology tools.

The development of robust visual-language foundation models for computational pathology is fundamentally constrained by the scarcity of large-scale, expertly annotated medical imaging datasets. Traditional supervised learning approaches require extensive domain expertise for data labeling and are often limited to specific tasks and diseases, hindering their broad applicability across the diverse landscape of pathology research and drug development [3] [25]. Zero-shot classification, wherein a model can recognize categories it was never explicitly trained on, presents a promising alternative. This capability is critically dependent on pretraining strategies that effectively align visual representations with rich semantic concepts from text. Leveraging large-scale image-caption pairs extracted from biomedical literature and educational resources offers a powerful pathway to build models that encapsulate the vast, nuanced knowledge of the pathology domain without relying on manual, pathology-specific annotations [26] [27]. These strategies mitigate the data bottleneck and produce models with superior generalization capabilities across a wide array of downstream tasks, from histology image classification to image-text retrieval [3].

This document outlines the core application notes and experimental protocols for leveraging these data sources to pretrain visual-language foundation models, with a specific focus on enabling zero-shot classification in pathology research.

The pretraining ecosystem for biomedical vision-language models primarily relies on large-scale datasets curated from scientific literature, particularly the PubMed Central Open Access (PMC-OA) subset. The table below summarizes the primary data sources and their quantitative significance.

Table 1: Key Large-Scale Biomedical Image-Caption Datasets

Dataset Name	Source	Scale (Image-Caption Pairs)	Domain Focus	Key Features
BIOMEDICA [27]	PMC-OA	> 24 Million	Pan-biomedical (Pathology, Radiology, Cell Biology, etc.)	Extracts all figures & captions; rich metadata (MeSH terms, licenses); expert-guided image content annotations.
PMC-15M [27]	PMC-OA	~15 Million	Pan-medical	A large-scale collection from 3 million articles; used for training models like BiomedCLIP.
ROCOv2 [28]	PMC-OA	~116,000	Radiology	A curated dataset of radiology images; serves as a benchmark for tasks like concept detection and caption prediction in ImageCLEFmed.
CONCH Pretraining Data [3] [25]	Diverse Sources	> 1.17 Million	Histopathology	Includes histopathology images, biomedical text, and image-caption pairs from educational and literature sources.

A critical insight from recent work is the move towards domain-agnostic data collection. Rather than pre-filtering data solely for specific domains like radiology, archives like BIOMEDICA advocate for extracting the entirety of available scientific figures. This approach captures the full spectrum of biomedical knowledge, from radiology and pathology to molecular biology and genetics, thereby creating a more comprehensive and powerful knowledge base for foundation models [27]. The resulting models demonstrate improved performance not only in broad benchmarks but also in specialized tasks, as they can leverage interconnected biological concepts.

Foundational Models and Architectures

The dominant architectural paradigm for zero-shot classification is the CLIP (Contrastive Language-Image Pre-training) framework and its derivatives. These models jointly train an image encoder and a text encoder to maximize the similarity between the embeddings of matched image-text pairs while minimizing the similarity for non-matched pairs within a batch [26] [26].

Several specialized models have been developed using the data sources described in Section 2:

• CONCH (Contrastive learning from Captions for Histopathology): A visual-language foundation model specifically designed for computational pathology. CONCH is pretrained on a diverse set of histopathology images and biomedical text, enabling state-of-the-art performance on zero-shot classification, segmentation, captioning, and retrieval across a suite of 14 diverse benchmarks [3] [25].
• BMCA-CLIP: A suite of CLIP-style models continually pretrained on the massive BIOMEDICA dataset. These models achieve state-of-the-art zero-shot performance across 40 tasks spanning pathology, radiology, ophthalmology, and cell biology, demonstrating an average performance improvement of 6.56% while using 10 times less compute than prior efforts [27].
• BiomedCLIP: A domain-adapted CLIP model pretrained on the PMC-15M dataset, which has shown strong zero-shot capabilities on various biomedical image classification tasks [27].

Experimental Protocols for Pretraining and Fine-Tuning

Protocol 1: Large-Scale Contrastive Pretraining from Scratch

This protocol outlines the process for training a CLIP-style model on a large-scale biomedical image-caption corpus, such as BIOMEDICA or PMC-15M.

Objective: To learn aligned visual and textual representations from a massive collection of biomedical image-caption pairs, creating a foundation model for zero-shot transfer.

Research Reagent Solutions:

Table 2: Essential Reagents for Large-Scale Pretraining

Reagent / Resource	Function / Description	Example / Source
Image-Caption Dataset	The core training data. A large-scale collection of figures and captions from biomedical literature.	BIOMEDICA (24M pairs) [27], PMC-15M (15M pairs) [27]
Image Encoder	A neural network that converts an image into a feature vector.	Vision Transformer (ViT) [27], ResNet-50 [29]
Text Encoder	A neural network that converts a text caption into a feature vector.	Transformer-based models (e.g., BioBERT, DeBERTa) [27] [28]
Contrastive Loss Function	The objective function that pulls positive pairs together and pushes negative pairs apart.	InfoNCE (NT-Xent) loss [26] [26]
High-Performance Compute	Clusters of GPUs or TPUs are required for training on large datasets.	NVIDIA A100/A6000, Google Cloud TPU
High-Throughput Data Loader	Software to efficiently stream and preprocess large datasets during training.	WebDataset format for 3x-10x higher I/O rates [27]

Procedure:

• Data Acquisition and Streaming: Download or directly stream the dataset in a high-throughput format like WebDataset to avoid local storage bottlenecks [27].
• Preprocessing:
  • Images: Resize images to a uniform resolution (e.g., 224x224 or 384x384). Apply standard augmentation techniques (e.g., random cropping, horizontal flipping, color jitter).
  • Text: Tokenize captions using the text encoder's tokenizer. A common practice is to prepend a prompt like "A biomedical image of" to the original caption to improve semantic alignment [26].
• Model Initialization: Initialize the image and text encoders. These can be initialized from scratch or from weights pretrained on general-domain data (e.g., OpenAI CLIP, Microsoft CLIP).
• Contrastive Training:
  • For a batch of N image-caption pairs, compute the image embeddings ( I \in \mathbb{R}^{N \times d} ) and text embeddings ( T \in \mathbb{R}^{N \times d} ), where ( d ) is the embedding dimension.
  • Compute the cosine similarity matrix ( S \in \mathbb{R}^{N \times N} ), where ( S{i,j} = Ii \cdot T_j ).
  • Optimize the symmetric InfoNCE loss. The image-to-text loss component is: [ \mathcal{L}{i2t} = -\frac{1}{N} \sum{i=1}^{N} \log \frac{\exp(S{i,i} / \tau)}{\sum{j=1}^{N} \exp(S{i,j} / \tau)} ] where ( \tau ) is a learnable temperature parameter. The text-to-image loss ( \mathcal{L}{t2i} ) is computed similarly, and the total loss is ( \mathcal{L} = (\mathcal{L}{i2t} + \mathcal{L}{t2i})/2 ) [26] [26].
• Validation: Monitor training using a downstream zero-shot classification task on a held-out validation set not seen during training.

Protocol 2: Fine-Tuning for Enhanced Zero-Shot Pathology Classification

This protocol describes a specialized fine-tuning strategy to adapt a pre-trained visual-language model for improved zero-shot pathology classification, addressing unique challenges in medical data.

Objective: To enhance a pre-trained model's zero-shot performance on pathology tasks by incorporating domain-specific adaptations, such as handling multi-labeled image-report pairs and dense medical captions.

Research Reagent Solutions:

Table 3: Essential Reagents for Fine-Tuning

Reagent / Resource	Function / Description	Example / Source
Pre-trained VLM	The base model to be adapted.	CONCH [3], BMCA-CLIP [27], General-domain CLIP
Domain-Specific Data	A smaller, task-relevant dataset for fine-tuning.	MIMIC-CXR [26], ROCOv2 [28]
Loss Relaxation Module	A modified loss function to handle false-negative pairs.	Upper-bound clipping on similarity scores [26]
Text Sampling Strategy	A method to better utilize multi-sentence medical reports.	Random Sentence Sampling [26]

Procedure:

• Data Preparation: Utilize a dataset of medical images and their corresponding textual reports (e.g., chest X-rays and radiology reports).
• Random Sentence Sampling:
  • For each report containing multiple sentences, randomly sub-sample n sentences during training.
  • This allows the model to learn that an image can be matched to multiple positive text descriptions, each focusing on different clinical findings, thereby enriching the semantic representation [26].
• Loss Relaxation for False Negatives:
  • In a batch, two different images may share some pathology labels (e.g., both have "Edema"). Standard contrastive loss treats them as a negative pair, which is suboptimal.
  • Modify the similarity function used in the InfoNCE loss by clipping its upper bound. This reduces the model's penalty for pairs that are not perfectly positive but share some semantic similarity, forcing the model to focus on hard negative pairs [26].
• Fine-Tuning: Train the model using the adapted data sampling and modified loss function for a limited number of epochs to avoid overfitting to the fine-tuning dataset.

Quantitative Benchmarking and Performance

Evaluating the zero-shot capabilities of the resulting models requires a diverse benchmark of biomedical tasks. The following table summarizes the performance of several state-of-the-art models, demonstrating the effectiveness of the described pretraining strategies.

Table 4: Benchmarking Zero-Shot Performance of Biomedical VLMs

Model	Pretraining Data Scale	Key Benchmark Tasks	Reported Performance
CONCH [3] [25]	>1.17M image-text pairs	14 diverse benchmarks (Classification, Segmentation, Retrieval)	State-of-the-art (SOTA) zero-shot performance on histology tasks.
BMCA-CLIP [27]	24M image-caption pairs	40 tasks across Pathology, Radiology, Ophthalmology, etc.	Average 6.56% improvement in zero-shot classification; up to 29.8% and 17.5% in dermatology and ophthalmology.
Method from [1] (Fine-tuned CLIP)	MIMIC-CXR (Fine-tuning)	Zero-shot classification on CheXpert dataset (5 pathologies)	Outperformed board-certified radiologists for the 5 competition pathologies.
BiomedCLIP [27]	~15M image-caption pairs	Various biomedical image classification tasks	Strong zero-shot performance, established a previous state-of-the-art.

Application Notes for Zero-Shot Classification in Pathology

Protocol 3: Executing Zero-Shot Pathology Classification

Objective: To use a pretrained visual-language model to classify pathology images into disease categories without task-specific training.

Procedure:

• Prompt Engineering: For each pathology class c of interest (e.g., "Adenocarcinoma," "Lymphocytic Infiltration"), create a set of positive and negative text prompts. A common and effective template is:
  • Positive Prompt: "{label}" (e.g., "Adenocarcinoma")
  • Negative Prompt: "no {label}" (e.g., "No adenocarcinoma") [26]
• Embedding Calculation:
  • Compute the image embedding ( v = E{img}(I{target}) ) for the target pathology image.
  • Compute the text embeddings for all positive and negative prompts: ( t{+,c} = E{txt}(T{+,c}) ), ( t{-,c} = E{txt}(T{-,c}) ).
• Similarity and Probability Calculation:
  • Calculate the cosine similarity between the image embedding and each prompt's text embedding: [ s{+,c} = \text{cosine}(v, t{+,c}), \quad s{-,c} = \text{cosine}(v, t{-,c}) ]
  • Compute the probability for each pathology c using the softmax function over the two similarities: [ probc = \frac{\exp(s{+,c})}{\exp(s{+,c}) + \exp(s{-,c})} ] This probability represents the model's confidence that the pathology is present in the image [26].

Critical Considerations for Practitioners:

• Data Diversity is Crucial: Models trained on pan-biomedical data (like BMCA-CLIP) consistently outperform those trained on narrow domains, as they can leverage cross-disciplinary knowledge [27].
• Computational Efficiency: Leveraging high-throughput data loaders and efficient model architectures (e.g., ViT) is essential for scaling pretraining to tens of millions of samples [27].
• Beyond Classification: These foundation models are not limited to classification. They can be directly applied to or fine-tuned for a wide range of tasks, including image captioning, visual question answering, and image-text retrieval, providing a versatile toolkit for pathology research and drug development [3] [28].

Zero-shot classification represents a paradigm shift in computational pathology, enabling the diagnosis of digital whole-slide images (WSIs) without task-specific model training or fine-tuning. This capability is powered by visual-language foundation models (VLMs), which learn aligned representations of histopathology images and medical text during large-scale pretraining. By leveraging semantic knowledge embedded in natural language descriptions, these models can recognize and classify novel pathological conditions not explicitly encountered during training. This protocol details the implementation of a zero-shot inference pipeline, from WSI preprocessing to final classification, providing researchers with a framework to overcome the critical bottleneck of data annotation in medical artificial intelligence (AI).

The integration of whole-slide imaging and artificial intelligence is transforming pathological diagnosis and research. Foundation models, pretrained on massive datasets of histopathology images and text, encode a rich understanding of disease morphology that can be generalized to new diagnostic tasks through zero-shot inference. This approach is particularly valuable for rare diseases, where annotated data are scarce, and for accelerating model deployment across diverse clinical scenarios. The pipeline described herein leverages models such as TITAN (Transformer-based pathology Image and Text Alignment Network) and CONCH (CONtrastive learning from Captions for Histopathology), which have demonstrated state-of-the-art performance across multiple pathology benchmarks without requiring fine-tuning.

Foundation Models for Zero-Shot Pathology

Visual-language foundation models bridge the gap between histopathological visual patterns and clinical terminology. These models are pretrained using self-supervised learning on large-scale datasets of WSI regions paired with corresponding pathological descriptions, either from medical reports or synthetic captions.

Table 1: Key Foundation Models for Zero-Shot Pathology

Model Name	Architecture	Pretraining Data Scale	Key Capabilities	Reported Performance
TITAN [10] [14]	Vision Transformer (ViT)	335,645 WSIs; 423,122 synthetic captions; 182,862 reports	Zero-shot classification, rare cancer retrieval, report generation	Outperforms ROI and slide foundation models across linear probing, few-shot, and zero-shot settings
CONCH [12]	Vision-Language Model	1.17M image-caption pairs	Image classification, segmentation, captioning, cross-modal retrieval	State-of-the-art on 14 diverse pathology benchmarks
Prov-GigaPath [30]	LongNet Transformer	1.3B image tiles from 171,189 WSIs	Cancer subtyping, mutation prediction, prognosis	State-of-the-art on 25/26 tasks; 23.5% AUROC improvement on EGFR mutation prediction
ZEUS [31]	VLM-based pipeline	N/A (leverages pretrained VLMs)	Zero-shot tumor segmentation	84.5% Dice Similarity Coefficient on skin tumor dataset

These models learn to project both image patches and text descriptions into a shared embedding space where semantically similar concepts are located proximally. During zero-shot inference, classification is performed by comparing image embeddings against text embeddings of class descriptions, effectively measuring the semantic similarity between visual patterns and pathological concepts.

Zero-Shot Inference Pipeline Methodology

The zero-shot inference pipeline transforms gigapixel WSIs into diagnostic predictions through a multi-stage computational process. The workflow encompasses WSI preprocessing, feature extraction, prompt engineering, and multimodal similarity calculation.

Zero-Shot Inference Pipeline: The complete workflow from whole-slide image input to classification output.

WSI Preprocessing and Feature Extraction

Whole-slide images present unique computational challenges due to their gigapixel resolution (often exceeding 100,000 × 100,000 pixels). Effective processing requires specialized approaches to handle this scale while preserving diagnostically relevant information.

Protocol 3.1.1: WSI Preprocessing and Tiling

• Tissue Segmentation: Apply automated tissue detection algorithms (e.g., Otsu's thresholding, UNet-based segmentation) to identify tissue-containing regions and exclude background areas. This dramatically reduces computational load by focusing analysis on diagnostically relevant regions.
• Multi-resolution Patching: Extract image patches at appropriate magnification levels (typically 20× for cellular detail). For zero-shot segmentation tasks, use overlapping patches (e.g., 448×448 pixels with 75% overlap) to ensure continuous prediction maps [31].
• Patch Feature Extraction: Process each patch through the vision encoder of a pretrained VLM (e.g., CONCH, TITAN) to generate dense feature representations. Models like CONCH extract 768-dimensional feature vectors for each patch, capturing morphological patterns at the local level [12].

Protocol 3.1.2: Handling Large-Scale Context

The TITAN model addresses the challenge of slide-level context through several technical innovations [10]:

• Constructs a 2D feature grid replicating patch positions within tissue
• Uses random cropping of 16×16 feature regions (covering 8,192×8,192 pixels)
• Implements Attention with Linear Biases (ALiBi) for long-context extrapolation
• Samples both global (14×14) and local (6×6) crops for comprehensive representation

Text Prompt Engineering and Embedding Generation

Effective prompt design is critical for zero-shot performance, as it bridges the semantic gap between visual patterns and diagnostic categories.

Protocol 3.2.1: Prompt Ensemble Creation

• Class Name Selection: For each diagnostic class, compile multiple synonymous descriptions. For tumor classification, include both technical terminology ("malignant melanoma") and common descriptions ("aggressive skin cancer with atypical melanocytes").
• Template Formulation: Create multiple textual templates that contextualize class names within histopathological discourse. Examples include:
  • "Whole-slide image showing [CLASSNAME]"
  • "Microscopic view of [CLASSNAME] cells"
  • "Histopathology consistent with [CLASSNAME]"
  • "Tissue morphology indicative of [CLASSNAME]" [31]
• Ensemble Generation: Generate the complete prompt ensemble by substituting each class name into every template, creating N×M possible prompts per class.

Protocol 3.2.2: Text Embedding Calculation

• Encode each prompt through the VLM's text encoder to generate embedding vectors.

• Compute the mean embedding vector across all prompts for each class to create robust class prototypes less sensitive to individual prompt phrasing:

  (wc = \frac{1}{NM} \sum{n=1}^{N} \sum{m=1}^{M} fT(p_{n,m}^c))

  where (wc) is the final embedding for class (c), (fT) is the text encoder, and (p_{n,m}^c) is the prompt created from the (n)-th class name and (m)-th template [31].

Multimodal Alignment and Classification

The core of zero-shot inference lies in measuring the semantic alignment between visual features and textual class descriptions.

Protocol 3.3.1: Similarity Computation

• For each image patch embedding (vj) and each class text embedding (wc), compute the cosine similarity:

  (s{j,c} = \frac{vj \cdot wc}{\|vj\| \|w_c\|})

  This measures the directional alignment between visual and textual representations in the shared embedding space [31].

• For slide-level classification, aggregate patch-level similarities across the entire WSI using attention mechanisms or pooling operations.

Protocol 3.3.2: Prediction and Interpretation

• Assign the diagnostic class with the highest overall similarity to the WSI:

  (\hat{y} = \arg\maxc \left( \frac{1}{|J|} \sum{j \in J} s_{j,c} \right))

  where (J) represents the set of all tissue patches in the WSI.

• Generate interpretability maps by visualizing similarity scores for each patch, highlighting regions that most strongly influenced the classification decision.

Experimental Validation and Performance

Zero-shot approaches have demonstrated competitive performance across diverse pathology tasks, particularly in scenarios with limited annotated data.

Table 2: Quantitative Performance of Zero-Shot Methods

Task	Dataset	Model	Metric	Performance
Skin Tumor Segmentation [31]	AI4SkIN (90 WSIs)	ZEUS (CONCH)	Dice Similarity Coefficient	84.5%
Skin Tumor Segmentation [31]	AI4SkIN (90 WSIs)	ZEUS (KEEP)	Dice Similarity Coefficient	83.7%
Rare Cancer Retrieval [10]	Multiple rare cancers	TITAN	Average Precision	Outperformed slide and ROI foundation models
Mutation Prediction [30]	TCGA (LUAD)	Prov-GigaPath	AUROC Improvement	23.5% improvement on EGFR prediction
Cross-modal Retrieval [10]	Mass-340K	TITAN	Retrieval Accuracy	State-of-the-art performance

The ZEUS framework demonstrates that zero-shot segmentation can achieve Dice scores exceeding 84% on cutaneous spindle cell neoplasms, rivaling supervised approaches while eliminating the need for manual annotations [31]. For rare disease applications, TITAN significantly outperforms both region-of-interest (ROI) and slide foundation models in retrieval tasks, highlighting the value of multimodal pretraining for low-data scenarios [10].

Zero-Shot Classification Workflow: Detailed steps for classifying whole-slide images using vision-language models.

The Scientist's Toolkit: Essential Research Reagents

Implementing zero-shot inference pipelines requires both computational resources and specialized software tools. The following table summarizes key components for establishing this capability in research environments.

Table 3: Research Reagent Solutions for Zero-Shot Pathology

Resource Category	Specific Tools/Models	Function	Access Method
Foundation Models	TITAN [10], CONCH [12], Prov-GigaPath [30]	Provide pre-extracted image and text embeddings for zero-shot inference	GitHub repositories, model zoos
WSI Processing	CLAM [31], PySlyde [32]	Tissue segmentation, patching, and feature extraction	Open-source Python packages
Vision Encoders	CONCH [12], ViT architectures	Encode histology patches into feature representations	Pretrained weights available
Text Encoders	BERT-style models [31], ClinicalBERT	Encode clinical text and prompts into embeddings	HuggingFace Transformers
Similarity Metrics	Cosine similarity, Euclidean distance	Measure alignment between image and text features	Standard Python libraries
Visualization	Matplotlib, Plotly, WholeSlideAnnotation	Generate similarity maps and interpretability visualizations	Open-source Python packages

The zero-shot inference pipeline represents a transformative approach to computational pathology, dramatically reducing the dependency on annotated datasets while maintaining competitive performance. By leveraging visual-language foundation models pretrained on large-scale histopathology datasets, researchers can rapidly deploy diagnostic models for novel diseases and rare conditions. As these models continue to scale in size and training data diversity, their zero-shot capabilities will further narrow the performance gap with supervised approaches, ultimately accelerating the development of AI-powered pathological diagnosis and expanding access to expert-level diagnostic capabilities in resource-limited settings.

The advent of visual-language foundation models is revolutionizing computational pathology by enabling powerful AI tools for cancer subtyping and biomarker prediction. These models, pretrained on massive datasets of histopathology images and corresponding textual reports, learn versatile and transferable feature representations of tissue morphology [10]. This capability is particularly transformative for zero-shot classification, where models can recognize and subtype cancers without task-specific training data [24]. Such an approach directly addresses critical challenges in oncology, including the diagnosis of rare cancers, which comprise 20-25% of all malignancies but often lack large, annotated datasets for traditional supervised learning [24]. By aligning visual patterns with semantic concepts from pathology reports, these models create a shared embedding space where histological features can be interpreted through natural language, enabling subtyping and biomarker prediction based on textual descriptions alone.

The clinical significance of this technology is profound. Molecular and histological subtyping directly influences therapeutic decisions and prognostic assessments [33]. For instance, distinguishing between triple-negative, HER2+, and luminal breast cancers determines eligibility for targeted therapies. Visual-language foundation models can perform this classification in a zero-shot manner by understanding textual descriptions of these subtypes, thereby providing scalable decision support even in resource-limited settings where specialized expertise is scarce [10] [24].

Key Foundation Models and Architectures

Prominent Models in Computational Pathology

Table 1: Key Visual-Language Foundation Models for Pathology

Model Name	Architecture	Pretraining Data	Key Capabilities
TITAN (Transformer-based pathology Image and Text Alignment Network)	Vision Transformer (ViT) with cross-modal alignment [10]	335,645 whole-slide images, 182,862 medical reports, and 423,122 synthetic captions [10]	Whole-slide representation learning, zero-shot classification, cross-modal retrieval, pathology report generation [10]
CONCH (CONtrastive learning from Captions for Histopathology)	Visual-language foundation model [12]	1.17 million histopathology image-caption pairs [12]	Image classification, segmentation, captioning, text-to-image, and image-to-text retrieval [12]
PathPT	Spatially-aware visual aggregation with task-specific prompt tuning [24]	Built upon existing vision-language pathology foundation models [24]	Few-shot and zero-shot rare cancer subtyping, cancerous region grounding, cross-modal reasoning [24]

Model Performance and Applications

Table 2: Performance Overview of Foundation Models in Cancer Subtyping

Model/Application	Cancer Types	Performance Metrics	Key Advantages
TITAN [10]	Pan-cancer evaluation across 20 organs [10]	Outperforms ROI and slide foundation models in linear probing, few-shot, and zero-shot classification [10]	General-purpose slide representations without fine-tuning; effective in resource-limited scenarios [10]
PathPT [24]	8 rare cancer datasets (4 adult, 4 pediatric) spanning 56 subtypes [24]	Substantial gains in subtyping accuracy and cancerous region grounding ability in few-shot settings [24]	Preserves localization on cancerous regions; enables cross-modal reasoning through prompts [24]
BC-predict [33]	Breast cancer (molecular and histological subtyping) [33]	88.79% balanced accuracy for ternary molecular subtyping; 94.23% ensemble accuracy for histological subtyping [33]	Integrates multiple machine learning models for comprehensive breast cancer characterization [33]
VaDTN [34]	SKCM, BRCA, LIHC, LUSC, STAD, PAAD [34]	Significant survival stratification in 4 of 6 cancer types (e.g., SKCM p=7.47×10⁻⁵) [34]	Incorporates tumor-normal distance in latent space for refined subtyping [34]

Experimental Protocols and Workflows

Zero-Shot Classification Protocol for Cancer Subtyping

Principle: Zero-shot classification leverages the semantic alignment between image and text embeddings in visual-language models. The model compares tissue morphology with textual descriptions of cancer subtypes without requiring labeled examples for those specific subtypes [24].

Procedure:

• Text Prompt Engineering: Create descriptive prompts for each cancer subtype using standardized terminology from pathology reports and clinical guidelines (e.g., "Invasive ductal carcinoma with tubule formation, pleomorphic nuclei, and frequent mitoses" for high-grade ductal carcinoma) [10] [24].
• Feature Extraction: Process whole-slide images (WSIs) using the vision encoder of the foundation model to generate slide-level embeddings [10].
• Similarity Calculation: Compute cosine similarity between image embeddings and text prompt embeddings in the shared latent space [10] [12].
• Subtype Assignment: Assign the cancer subtype based on the highest similarity score between the image embedding and textual subtype descriptions [24].

Validation:

• Compare zero-shot predictions with pathologist annotations on a held-out test set [10].
• Calculate standard metrics: accuracy, F1-score, and Cohen's kappa for inter-rater agreement [24].

TITAN Model Implementation for Biomarker Prediction

Workflow Overview: The TITAN model employs a three-stage pretraining approach to develop general-purpose slide representations applicable to biomarker prediction [10].

Implementation Protocol:

• Whole-Slide Image Processing:
  • Divide WSIs into non-overlapping 512×512 pixel patches at 20× magnification [10].
  • Extract 768-dimensional features for each patch using a pretrained patch encoder (e.g., CONCH) [10] [12].
  • Spatially arrange patch features into a 2D feature grid replicating tissue positions [10].

• Model Inference:

  • Sample region crops of 16×16 features (covering 8,192×8,192 pixels) from the WSI feature grid [10].
  • Process through TITAN's Vision Transformer with attention mechanisms to generate slide-level representations [10].
  • For biomarker prediction, use these representations directly or with minimal adaptation for specific prediction tasks [10].

• Biomarker Specific Adaptation:

  • HER2 Status Prediction: Use text prompts describing HER2-positive patterns (e.g., "complete, intense circumferential membrane staining") and HER2-negative patterns [12].
  • Tumor Mutational Burden: Correlate morphological patterns with genomic features through cross-modal retrieval between image embeddings and molecular descriptions [10].

PathPT Framework for Rare Cancer Subtyping

Principle: PathPT enhances few-shot and zero-shot performance for rare cancers by converting WSI-level supervision into fine-grained tile-level guidance and leveraging task-specific prompt tuning [24].

Procedure:

• Spatially-aware Visual Aggregation:
  • Extract tile-level features from WSIs using a vision encoder [24].
  • Aggregate tile features using attention mechanisms that preserve spatial relationships within the tissue [24].

• Task-specific Prompt Tuning:

  • Initialize with descriptive prompts for each rare cancer subtype based on pathological characteristics [24].
  • Optimize prompts using limited labeled examples (few-shot learning) to align visual features with histological semantics [24].

• Cross-modal Reasoning:

  • Leverage the zero-shot capabilities of vision-language models to establish connections between visual patterns and textual descriptions [24].
  • Generate localization maps highlighting regions contributing to subtype classification for improved interpretability [24].

Table 3: Essential Research Reagents and Computational Tools for Zero-Shot Cancer Subtyping

Category	Item/Resource	Specification/Purpose	Example Sources/Formats
Data Resources	Whole Slide Images (WSIs)	Gigapixel digital pathology slides; diverse cancer types and stains [35]	.svs, .ndpi, .mrxs, .tiff formats [35]
	Pathology Reports	Textual descriptions aligned with WSIs for multimodal training [10]	Structured and unstructured clinical text [10]
	Public Datasets	Benchmarking and validation datasets	TCGA, GTEx, CAMELYON16 [35] [34]
Software Tools	WSI Annotation Tools	Precise region and cell-level annotation for validation [35]	QuPath, Digital Slide Archive, Aiforia [35]
	Foundation Models	Pretrained models for feature extraction and zero-shot inference [10] [12]	TITAN, CONCH, PathPT [10] [12] [24]
	Analysis Frameworks	Environments for model development and evaluation	Python, PyTorch, MONAI [35]
Computational Infrastructure	GPU Clusters	Model training and inference on high-resolution WSIs [10]	High-memory GPUs (e.g., NVIDIA A100, H100) [10]
	Storage Systems	Management of large-scale WSI datasets (1-10 GB per slide) [35]	Scalable network-attached storage [35]

Integrated Workflow for Zero-Shot Cancer Subtyping

The following diagram illustrates the complete workflow for zero-shot cancer subtyping using visual-language foundation models, integrating data processing, model inference, and clinical validation:

Validation and Benchmarking Framework

Rigorous validation is essential for clinical translation of zero-shot classification models. The following protocol ensures robust evaluation:

Performance Metrics:

• Classification Accuracy: Standard metrics (accuracy, F1-score, AUC-ROC) for subtype prediction [24] [33].
• Survival Stratification: Kaplan-Meier analysis to assess prognostic relevance of identified subtypes [34].
• Cross-modal Retrieval: Precision and recall for image-to-text and text-to-image retrieval tasks [10] [12].

Benchmarking Protocol:

• Dataset Curation: Compile diverse multi-center cohorts representing different cancer types, staining protocols, and scanner variations [35].
• Comparison Baselines: Evaluate against traditional supervised methods and other foundation models [10] [24].
• Statistical Analysis: Assess significance of performance differences using appropriate statistical tests [34].
• Clinical Correlation: Validate predictions against gold-standard pathological assessments and clinical outcomes [33] [34].

This comprehensive framework for cancer subtyping and biomarker prediction using visual-language foundation models demonstrates the transformative potential of zero-shot classification in computational pathology, enabling robust AI-assisted diagnosis even for rare cancers with limited annotated data.

Experimental Foundation and Key Models

The development of robust rare disease retrieval systems is critically dependent on foundation models that can generalize without disease-specific training data. The following table summarizes the core models enabling these capabilities.

Table 1: Foundation Models for Rare Disease Search and Retrieval

Model Name	Architecture	Core Capability	Training Data Scale	Primary Application in Rare Diseases
TITAN (Transformer-based pathology Image and Text Alignment Network) [10] [14]	Multimodal Vision Transformer (ViT)	Whole-slide image representation & report generation	335,645 WSIs; 182,862 reports; 423,122 synthetic captions	Zero-shot rare cancer retrieval and cross-modal search [10].
PathPT [24]	Vision-Language Model with Spatially-aware Prompt Tuning	Few-shot prompt tuning for rare cancer subtyping	Evaluated on 2,910 WSIs across 56 rare subtypes	Boosts subtyping accuracy and tumor region grounding in few-shot settings [24].
MI-Zero [5]	Visual Language Model + Multi-Instance Learning (MIL)	Zero-shot classification	33,480 image-text pairs	Zero-shot transfer for pathological image classification without labeled data [5].
Knowledge-Guided Multimodal Transformer [36]	Transformer + Graph Neural Network (GNN)	Rare disease diagnosis from EHR, genomics, imaging	MIMIC-IV, ClinVar, CheXpert datasets	Integrates multimodal data and rare disease ontologies (Orphanet) for early diagnosis [36].

Detailed Experimental Protocols

Protocol for TITAN-based Zero-Shot Rare Cancer Retrieval

This protocol outlines the procedure for using the TITAN model to retrieve whole-slide images (WSIs) of rare cancers based on a textual or visual query, without any task-specific fine-tuning [10].

I. Research Reagent Solutions

Table 2: Key Reagents for TITAN-based Retrieval

Item	Function/Description
TITAN Pre-trained Model Weights	Provides the foundational parameters for slide and text encoding. Available from the model's authors [10].
Mass-340K Dataset (or subset)	A large-scale internal dataset of 335,645 WSIs and corresponding reports for pre-training and validation [10].
Target Rare Disease WSI Repository	The database of gigapixel WSIs from which similar cases will be retrieved.
CONCHv1.5 Patch Encoder	Encodes 512x512 pixel patches from a WSI into 768-dimensional feature vectors, forming the input for TITAN [10].
PathChat	A multimodal generative AI copilot used to generate fine-grained synthetic captions for vision-language alignment during pre-training [10].

II. Step-by-Step Methodology

• Input Data Preparation:

  • For each WSI in the retrieval database and for the query slide (if applicable), use the CONCHv1.5 patch encoder to extract feature vectors from non-overlapping 512x512 patches at 20x magnification [10].
  • Arrange these feature vectors into a 2D spatial grid that mirrors the original tissue layout.
  • For a textual query (e.g., "find cases of rare adrenal cortical carcinoma"), use the raw text.

• Feature Encoding:

  • WSI Encoding: Process the 2D feature grid through the TITAN vision encoder to obtain a single, general-purpose slide-level embedding vector for each WSI in the database [10].
  • Text Query Encoding: Process the text query through the TITAN text encoder to obtain a text embedding vector in the same shared latent space [10].

• Similarity Computation & Retrieval:

  • Compute the cosine similarity between the text query embedding and all slide-level embeddings in the database.
  • Rank the WSIs based on the similarity scores.
  • Return the top-K most similar WSIs as the retrieval result.

• Validation and Evaluation:

  • Use metrics such as Recall@K and Mean Average Precision (mAP) to quantify retrieval performance on a test set with known rare disease labels [10].

TITAN Zero-shot Retrieval Workflow

Protocol for PathPT-based Few-Shot Rare Cancer Subtyping

This protocol uses the PathPT framework to adapt a pre-trained vision-language pathology foundation model for accurate subtyping of rare cancers with only a few labeled examples, enhancing both accuracy and interpretability [24].

I. Research Reagent Solutions

Table 3: Key Reagents for PathPT-based Subtyping

Item	Function/Description
Pre-trained VL Foundation Model	A base model (e.g., PLIP, CONCH) providing initial visual and textual representations.
PathPT Framework Code	The novel framework that introduces spatially-aware visual aggregation and task-specific prompt tuning [24].
Few-Shot Rare Cancer Dataset	A small set of labeled WSIs (e.g., 5-20 per subtype) for the target rare cancers for prompt tuning [24].
Task-Specific Prompt Templates	Textual prompts (e.g., "a histology image of [RARE_SUBTYPE]") that are optimized during training [24].

II. Step-by-Step Methodology

• Model Initialization:

  • Start with a pre-trained vision-language (VL) pathology foundation model. PathPT is model-agnostic and can be applied to various VL architectures [24].

• Spatially-aware Visual Aggregation:

  • Unlike conventional Multi-Instance Learning (MIL) that treats patches as independent, PathPT converts WSI-level supervision into tile-level guidance.
  • It leverages the zero-shot capabilities of the VL model to weight the contribution of individual image tiles based on their relevance to the textual prompts, preserving localization information on cancerous regions [24].

• Task-Specific Prompt Tuning:

  • Instead of using fixed, hand-crafted prompts, the continuous vector representations of the prompts are made learnable parameters.
  • These prompt vectors are optimized using the few-shot training data, aligning them with the histopathological semantics of the rare cancer subtypes [24].

• Joint Optimization and Inference:

  • The model is trained end-to-end, jointly optimizing the prompt vectors and the aggregation mechanism to minimize the classification loss on the few-shot dataset.
  • At inference, the tuned model processes a new WSI using the optimized prompts to predict its rare cancer subtype and can highlight diagnostically relevant regions [24].

PathPT Few-shot Subtyping Workflow

Performance and Quantitative Results

Evaluations across diverse clinical tasks demonstrate the superior performance of these foundation models, particularly in data-scarce scenarios relevant to rare diseases.

Table 4: Performance Summary of Foundation Models on Rare Disease Tasks

Model / Task	Evaluation Metric	Performance Result	Benchmark / Baseline Comparison
TITAN: Rare Cancer Retrieval [10]	Recall@K	Outperforms existing models	Superior to both ROI and slide foundation models in zero-shot settings [10].
TITAN: Zero-shot Classification [10]	Accuracy	Outperforms existing models	Effective without fine-tuning or clinical labels [10].
PathPT: Rare Cancer Subtyping (Few-shot) [24]	Subtyping Accuracy	Substantial gains	Consistently superior to 4 state-of-the-art VL models and 4 MIL frameworks under few-shot settings [24].
Knowledge-Guided Multimodal Transformer [36]	Diagnostic Accuracy	Significantly outperforms baselines	Higher accuracy and robustness on MIMIC-IV, ClinVar, and CheXpert datasets [36].
TxGNN: Drug Indication Prediction [37]	Prediction Accuracy	49.2% improvement	Compared to existing methods for predicting drug efficacy for rare diseases [37].
TxGNN: Contraindication Identification [37]	Prediction Accuracy	35.1% improvement	Compared to existing methods [37].

Table 1: Performance Comparison of Vision-Language Models in Computational Pathology

Model Name	Architecture Type	Pretraining Data Scale	Key Capabilities	Report Generation Performance
TITAN [10]	Multimodal Whole-Slide Foundation Model	335,645 WSIs + 182,862 reports + 423,122 synthetic captions	Slide-level representation, zero-shot classification, cross-modal retrieval, pathology report generation	Outperforms slide foundation models in rare cancer retrieval and prognosis; generates clinically relevant reports without fine-tuning.
CONCH [9]	Vision-Language Model (CoCa-inspired)	1.17M histopathology image-caption pairs	Zero-shot diagnostic classification, contextual reasoning	Achieves highest diagnostic accuracy with precise anatomical prompts; foundational for diagnostic text generation.
Quilt-LLaVA [9]	Large Multimodal Model (LLaVA-based)	~107k histopathology Q/A pairs	Visual question answering, generative capabilities	Enables sophisticated image-text interaction for descriptive report sections.
EyeCLIP [38]	Multimodal Visual-Language Model	2.77M ophthalmology images + clinical text	Zero-shot/few-shot classification, cross-modal retrieval	Demonstrates robust zero-shot capabilities for disease classification, providing a model for preliminary findings.

Experimental Protocols for Zero-Shot Preliminary Report Generation

Protocol A: Whole-Slide Image Encoding and Representation Learning

This protocol outlines the foundational training for slide-level representation, as used in developing the TITAN model [10].

• Objective: To learn general-purpose, slide-level feature representations from gigapixel Whole-Slide Images (WSIs) without relying on clinical labels.
• Materials:
  • Hardware: High-performance computing cluster with multiple GPUs (e.g., NVIDIA A100 or H100).
  • Software: PyTorch or TensorFlow deep learning frameworks, OpenSlide or CuImage libraries for WSI handling.
  • Data: A large-scale dataset of WSIs with optional paired pathology reports (e.g., Mass-340K with 335,645 WSIs [10]).
• Methodology:
  • Patch Feature Extraction:
    • Divide each WSI into non-overlapping patches (e.g., 512x512 pixels at 20x magnification).
    • Use a pre-trained histopathology patch encoder (e.g., CONCHv1.5 [10]) to extract a feature vector (e.g., 768-dimensional) for each patch.
    • Spatially arrange these feature vectors into a 2D feature grid that mirrors the original tissue layout.
  • Slide-Level Pretraining:
    • Apply a Vision Transformer (ViT) encoder to the 2D feature grid.
    • Employ self-supervised learning (SSL) objectives like masked image modeling (e.g., iBOT framework [10]) on random crops of the feature grid (e.g., 16x16 features, corresponding to 8,192x8,192 pixel regions).
    • Use Attention with Linear Biases (ALiBi) extended to 2D to manage long-range context and variable WSI sizes.

Protocol B: Multimodal Vision-Language Alignment for Pathology

This protocol describes aligning visual features with textual descriptions to enable text and report generation [10].

• Objective: To align WSI representations with language for zero-shot classification, cross-modal retrieval, and free-text report generation.
• Materials:
  • Model: A pre-trained slide-level encoder from Protocol A (e.g., TITANV [10]).
  • Data:
    • Paired WSIs and pathology reports.
    • Synthetic, fine-grained region-of-interest (ROI) captions generated by a multimodal AI copilot (e.g., 423k captions generated using PathChat [10]).
• Methodology:
  • ROI-Level Alignment:
    • Fine-tune the visual encoder using image-text contrastive learning on pairs of high-resolution ROIs (8k x 8k pixels) and their corresponding synthetic captions. This teaches the model fine-grained morphological concepts.
  • Slide-Level Alignment:
    • Further fine-tune the model using contrastive learning on pairs of entire WSIs and their corresponding clinical pathology reports. This aligns slide-level visual patterns with diagnostic language.

Protocol C: Zero-Shot Inference for Preliminary Report Generation

This protocol details the application of a fully trained model like TITAN to generate preliminary reports for new, unseen WSIs [10].

• Objective: To generate a preliminary pathological description for a novel WSI without task-specific fine-tuning.
• Materials:
  • Model: A fully trained multimodal model (e.g., TITAN after Protocol A and B [10]).
  • Input: A novel, unannotated WSI.
• Methodology:
  • Feature Encoding: Process the novel WSI through the model's visual encoder to obtain a slide-level representation.
  • Cross-Modal Retrieval (Optional): Use the slide embedding to retrieve the K-most similar cases from a database of existing reports, providing reference diagnoses and phrasing [10].
  • Text Generation: Use the model's language decoder to auto-regressively generate a textual report conditioned on the slide-level visual representation. The output may include descriptions of key morphological findings, differential diagnoses, and prognostic indicators.

Workflow Visualization for Preliminary Report Generation

The Scientist's Toolkit: Essential Research Reagents & Materials

Table 2: Key Research Reagents and Computational Tools for Pathology VLM Development

Item Name	Type	Function / Application	Exemplars from Literature
Patch Encoder	Software Model	Extracts foundational feature representations from small image patches. Essential for processing gigapixel WSIs.	CONCH [9] [10], CTransPath [10]
Whole-Slide Image Dataset	Data	Large-scale collection of WSIs, ideally with paired text, used for model pretraining and evaluation.	Mass-340K (335k WSIs) [10], In-house digestive dataset (3.5k WSIs) [9]
Synthetic Caption Generator	Software Model / Tool	Generates fine-grained, descriptive text for image regions to augment training data for vision-language alignment.	PathChat [10]
Vision-Language Alignment Framework	Software Algorithm	Performs contrastive learning to align image and text features in a shared embedding space.	Image-text contrastive loss (e.g., CLIP-style [9] [26] [10])
Slide-Level Encoder (Transformer)	Software Model	Processes sequences of patch features to model long-range dependencies and create a unified slide-level representation.	TITAN (ViT with ALiBi) [10]

Overcoming Challenges: Fine-Tuning, Prompt Engineering, and Data Optimization

Addressing the Multi-Label Nature of Medical Image-Report Pairs

The integration of artificial intelligence (AI) in pathology and medical imaging represents a paradigm shift in diagnostic workflows and drug development research. A significant challenge in this domain is the inherent multi-label nature of medical data, where a single image or whole-slide image (WSI) can contain multiple, co-occurring pathological findings. Traditional AI models, often designed for single-label classification, struggle to capture this complexity. However, the emergence of visual-language foundation models (VLFMs) offers a transformative approach. These models, pretrained on vast datasets of image-text pairs, learn to align visual features with rich semantic descriptions, enabling them to interpret the multi-faceted content of medical images. This document details application notes and experimental protocols for leveraging VLFMs, particularly within a zero-shot learning framework, to address the multi-label challenge in pathology research. By doing so, we can advance capabilities in automated report generation, comprehensive disease subtyping, and efficient toxicity profiling in drug development.

Theoretical Foundation: Multi-Label Learning and Zero-Shot VLFMs

The problem of medical image interpretation is intrinsically a multi-label classification task. A chest X-ray may exhibit cardiomegaly, edema, and a pleural effusion simultaneously [39]. Similarly, a histopathology slide of a drug-treated tissue sample might show multiple distinct pathological findings in both the liver and kidney [40]. Standard classification models require vast amounts of labeled data for each potential label and combination, a requirement that is impractical, especially for rare diseases or novel drug-induced effects.

Visual-language foundation models address this by learning a shared embedding space where images and their textual descriptions are closely aligned. This foundational capability enables zero-shot classification, where the model can recognize concepts not explicitly seen during training by leveraging semantic relationships.

• Mechanism of Zero-Shot Multi-Label Classification: A VLFM, such as TITAN (Transformer-based pathology Image and Text Alignment Network), processes an input image and a set of textual label descriptors (e.g., "presence of granuloma," "hepatocellular necrosis") [10] [14]. The model encodes the image and each text prompt into the shared embedding space. The similarity between the image embedding and each text prompt embedding is computed, and labels whose textual descriptors exceed a similarity threshold are assigned to the image. This allows for the prediction of multiple, non-exclusive labels without task-specific fine-tuning.
• Advantages over Traditional Multi-label Models:
  • Elimination of Labeled Datasets: No need for large, curated datasets for every new set of pathological findings.
  • Inherent Scalability: New labels can be added simply by creating new text prompts, without retraining the model.
  • Cross-Modal Retrieval: Enables searching a database of images using text queries, and vice versa, facilitating rare disease identification [10].

Table 1: Comparison of Model Paradigms for Multi-Label Tasks in Pathology.

Feature	Traditional Multi-Label CNN	VLFM (Zero-Shot)
Data Requirement	Large, fully-labeled datasets for all labels	No labeled data required for inference
Scalability	Adding new labels requires model retraining	New labels added via text prompts
Handling Rare Labels	Poor performance due to data scarcity	Robust through semantic understanding
Primary Output	Probability scores for a fixed set of labels	Similarity scores between image and flexible text descriptors
Interpretability	Often requires separate saliency maps	Inherently more interpretable via text alignment

Application Protocols

Protocol 1: Zero-Shot Multi-Label Classification for Histopathology Slides

This protocol utilizes a pre-trained VLFM to classify multiple pathologies in a whole-slide image (WSI) without any model fine-tuning.

Workflow Overview:

Detailed Methodology:

• Input Preparation: Obtain a digitized H&E-stained WSI. The model is designed to handle the gigapixel scale of WSIs [10].
• Patch Feature Extraction:
  • Divide the WSI into non-overlapping patches of 512 x 512 pixels at 20x magnification.
  • Use a pre-trained patch encoder (e.g., CONCHv1.5) to extract a 768-dimensional feature vector for each patch [10].
  • Spatially arrange these feature vectors into a 2D feature grid that mirrors the original tissue layout.
• Slide-Level Encoding:
  • Process the 2D feature grid through the visual encoder of the TITAN model (e.g., a Vision Transformer with ALiBi positional encoding for long-sequence context) to generate a single, general-purpose slide-level embedding [10].
• Text Prompt Engineering:
  • Curate a comprehensive list of pathological findings relevant to the organ and disease context. For drug toxicity assessment, this could include findings like "hepatocellular hypertrophy," "bile duct hyperplasia," and "tubular necrosis" [40].
  • Convert each finding into a descriptive text prompt. Using class-specific prompts (e.g., "a histopathology image showing [pathology]") often yields better results than single keywords.
• Cross-Modal Similarity and Label Assignment:
  • Encode all text prompts using the model's text encoder.
  • Compute the cosine similarity between the slide-level image embedding and each text prompt embedding.
  • Apply a pre-defined threshold to the similarity scores. All labels whose similarity score exceeds the threshold are assigned to the WSI, resulting in a multi-label prediction.

Protocol 2: Few-Shot Prompt Tuning for Rare Cancer Subtyping

For highly rare cancers with very limited data, a pure zero-shot approach may be insufficient. This protocol uses minimal samples to "teach" the model new concepts via prompt tuning.

Workflow Overview:

Detailed Methodology:

• Support Set Curation: Assemble a small dataset (e.g., 1-5 WSIs per class) of rare cancer subtypes. This constitutes the few-shot training set.
• Model and Prompt Initialization: Select a pre-trained pathology VLFM. Initialize a set of learnable vectors (the "prompt") for each rare cancer subtype [24].
• Prompt Tuning Loop:
  • Keep the core weights of the VLFM frozen to preserve its pre-trained knowledge and prevent overfitting.
  • For each WSI in the support set, process it through the model using the current learnable prompts.
  • The loss function (e.g., cross-entropy) compares the predicted similarities (based on the tuned prompts) to the true labels.
  • Use backpropagation to update only the learnable prompt vectors, making them more effectively represent the rare subtypes within the model's embedding space.
• Inference: Use the tuned prompts with the frozen VLFM to perform multi-label classification on new, unseen WSIs following the similarity calculation and thresholding steps from Protocol 1.

Experimental Validation and Benchmarking

To validate the efficacy of VLFMs for multi-label tasks, rigorous benchmarking against established baselines is essential. The following table and protocol outline this process.

Table 2: Performance Benchmark of VLFMs on Multi-Label Pathology Tasks.

Model / Approach	Dataset	Task	Key Metric	Score	Notes
TITAN (Zero-Shot) [10]	Internal Mass-340K (20 organs)	Slide-level Classification & Report Generation	Outperforms ROI & slide models	Superior performance in linear probing, few-shot, and zero-shot settings.	Pretrained on 335,645 WSIs; enables cross-modal retrieval.
PathPT (Few-Shot) [24]	8 Rare Cancer Datasets (56 subtypes)	Rare Cancer Subtyping	Subtyping Accuracy	Substantial gains in accuracy and region grounding.	Leverages tile-level guidance from VLFMs; superior to conventional MIL.
Qwen2-VL-72B [41]	PathMMU (Pathology VLM Benchmark)	Multiple-Choice VQA	Average Accuracy	63.97%	Top-performing open VLM on pathology-specific understanding.
Att-RethinkNet [40]	Open TG-GATEs	Multi-Label Pathology Prediction (Liver/Kidney)	AUC & Accuracy	Competitive performance vs. state-of-the-art.	Traditional deep learning multi-label model; requires full training.

Protocol 3: Benchmarking Against Multi-Label Baselines

Objective: To quantitatively compare the performance of a zero-shot VLFM against a trained multi-label model on a well-defined task like predicting drug-induced pathological findings.

Detailed Methodology:

• Dataset Selection: Use a public toxicogenomics dataset like Open TG-GATEs, which includes histopathology images and annotated pathological findings for multiple organs (liver and kidney) [40].
• Model Setup:
  • VLFM (Zero-Shot): Apply Protocol 1. The text prompt library should include all pathological findings to be predicted (e.g., "vacuolization," "inflammatory infiltrate").
  • Baseline (Supervised): Train a dedicated multi-label model, such as Att-RethinkNet [40], on the training split of the dataset. This model uses an attention mechanism and a memory structure to capture label correlations.
• Evaluation Metrics: Calculate standard multi-label metrics on a held-out test set:
  • Macro-F1 Score: Averages the F1-score for each label, giving equal weight to all labels. This is crucial for assessing performance on rare findings.
  • AUC-PR (Area Under the Precision-Recall Curve): Particularly informative for imbalanced datasets.
  • Subset Accuracy (Exact Match): Measures the percentage of samples where all labels are correctly predicted.
• Analysis: Compare the results. The key finding to validate is that the zero-shot VLFM can achieve performance comparable to a supervised model without any task-specific training, demonstrating its utility for rapid prototyping and applications with limited labeled data.

The Scientist's Toolkit: Essential Research Reagents

Table 3: Key Resources for Implementing VLFM-based Multi-Label Analysis.

Resource Name	Type	Function / Application	Example / Source
TITAN Model	Foundation Model	General-purpose slide encoding and zero-shot multi-label tasks.	[10] [14]
CONCH / CONCHv1.5	Patch Encoder	Extracts foundational feature representations from image patches.	[10]
Open TG-GATEs	Dataset	Public toxicogenomics data for validating drug-induced pathology prediction.	[40]
PathMMU Benchmark	Evaluation Suite	Standardized dataset for benchmarking VLM performance in pathology.	[41]
PathPT Framework	Few-Shot Method	Boosts VLFM performance for rare cancer subtyping via prompt tuning.	[24]
Synthetic Captions	Data Augmentation	Provides fine-grained, scalable training data for VLFMs.	Generated by AI copilots (e.g., PathChat) [10]

The development of visual-language foundation models (VLFMs) is transforming computational pathology by enabling AI systems to learn from images and associated text without extensive manual labeling. A significant capability of these models is zero-shot classification, where a model can diagnose or categorize histopathology images without having been explicitly trained on labeled examples for that specific task [1]. However, achieving high performance in this setting is challenging. Advanced fine-tuning strategies, particularly loss relaxation and random sentence sampling, have emerged as powerful techniques to enhance the robustness and accuracy of VLFMs for pathology applications, allowing them to better handle the nuanced, multi-labeled nature of medical data [26].

Core Concepts and Their Role in Pathology

The Challenge of False-Negative Pairs in Medical Data

In standard contrastive learning, models are trained to identify positive image-text pairs (e.g., a whole-slide image and its corresponding report) from negative pairs. However, medical image-report pairs often share overlapping semantic labels [26]. For instance, two different chest X-ray reports might both mention "Lung Opacity" and "Edema," meaning they are semantically related. Standard contrastive loss functions incorrectly treat these pairs as entirely negative, leading to suboptimal model performance. Loss relaxation addresses this by softening the penalty for these "false-negative" pairs [26].

The Information Density of Medical Reports

Pathology and radiology reports are brief yet dense with critical clinical information. Traditional text augmentation techniques like random deletion or synonym replacement risk altering clinical meaning [26]. Random sentence sampling is a tailored augmentation method that treats a report as a collection of informative sentences. By randomly sub-sampling sentences during training, it teaches the model to align images with rich, fine-grained textual descriptions rather than a single, global report representation, thereby improving the model's language understanding [26].

The table below summarizes the performance improvements afforded by these fine-tuning techniques across various medical imaging tasks and datasets.

Table 1: Performance Gains from Advanced Fine-Tuning Techniques

Fine-Tuning Technique	Task/Dataset	Model(s)	Key Metric	Performance Gain	Significance
Loss Relaxation + Random Sentence Sampling	Zero-shot Chest X-ray Pathology Classification (CheXpert)	Multiple pre-trained image-text encoders [26] [42]	Macro AUROC	Average increase of 4.3% across four datasets [26]	Outperformed state-of-the-art and marginally surpassed board-certified radiologists [26] [42]
Loss Relaxation + Random Sentence Sampling	Zero-shot Chest X-ray Pathology Classification	Pre-trained contrastive models (e.g., CLIP-based) [26]	Macro AUROC	Consistent improvements across three distinct pre-trained models [26]	Method is model-agnostic and does not require external data [26]
Multi-modal Whole-Slide Model (TITAN)	Slide-level Cancer Subtyping (TCGA NSCLC)	TITAN (using SSL and vision-language alignment) [10] [14]	Zero-shot Accuracy	Achieved 90.7% accuracy [10]	Outperformed existing slide foundation models by a wide margin (e.g., 12.0% over PLIP) [10]
Visual-Language Model (CONCH)	Zero-shot Gleason Pattern Classification (SICAP)	CONCH [1]	Quadratic Kappa (QK)	Achieved 0.690 QK [1]	Outperformed BiomedCLIP by 0.140 [1]

Experimental Protocols

Protocol A: Implementing Random Sentence Sampling

Objective: To enhance the model's ability to learn from fine-grained, sentence-level information in medical reports.

Materials:

• A batch of N medical image-report pairs.
• A pre-trained visual-language model (e.g., a CLIP-like architecture).
• A text tokenizer.

Procedure:

• Text Preprocessing: For each medical report in the batch, split the text into its constituent sentences.
• Stochastic Sampling: For every training step and for each report, randomly sub-sample n sentences from the total m sentences available. The value of n can be fixed or randomly chosen within a range.
• Prompt Construction: Use the selected n sentences as the positive text input for the corresponding image.
• Batch Formation: The positive image-text pairs for the contrastive learning batch are now composed of the image and its stochastically sampled sentences.
• Model Training: Proceed with the contrastive learning objective. This process forces the image encoder to align with multiple valid, but partial, textual descriptions, teaching it the rich semantics of individual findings [26].

Protocol B: Implementing Loss Relaxation for False-Negative Mitigation

Objective: To modify the contrastive loss function to prevent over-penalization of semantically similar image-text pairs.

Materials:

• A batch of N image-text pairs.
• A pre-trained visual-language model with image encoder ( E{img} ) and text encoder ( E{txt} ).
• Standard InfoNCE (Noise-Contrastive Estimation) loss function.

Procedure:

• Embedding Extraction: Compute normalized image embeddings ( ui ) and text embeddings ( vi ) for all pairs in the batch.
• Similarity Calculation: Compute the cosine similarity matrix for all possible image-text pairs in the batch.
• Loss Modification: Replace the standard similarity function in the InfoNCE loss with a modified version that clips the upper bound of the similarity for negative pairs. This reduces the attractive force between positive pairs, thereby mitigating the penalty for false negatives. The specific modification involves a relaxed similarity function that makes the model focus on perfectly negative pairs [26].
• Loss Computation: Calculate the relaxed InfoNCE loss using the modified similarity scores. Given a batch of N pairs, the loss is computed as: $$ \mathcal{L} = -\frac{1}{2N}\left(\sum_{i=1}^{N}\log\frac{\exp(\text{sim}'(u_i, v_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}'(u_i, v_j)/\tau)} + \sum_{i=1}^{N}\log\frac{\exp(\text{sim}'(v_i, u_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}'(v_i, u_j)/\tau)}\right) $$ Where sim' is the modified, relaxed similarity function [26].
• Backpropagation: Update the model parameters using the gradients from the relaxed loss.

Workflow Visualization

Diagram 1: Fine-tuning Workflow. This diagram illustrates the integrated training process, highlighting the two key techniques: Random Sentence Sampling (A) and computing a Relaxed Loss (C).

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Resources for Fine-tuning Pathology VLFMs

Resource Name/Type	Function in Fine-tuning	Specific Examples & Notes
Pre-trained VLMs	Provides the foundational model to be adapted for pathology tasks.	CONCH [1]: A VLFMs pretrained on 1.17M histopathology image-caption pairs. TITAN [10] [14]: A multimodal whole-slide foundation model. CLIP [26] [1]: General-domain models (e.g., OpenAI's CLIP) that can be adapted.
Histopathology Datasets	Provides paired image-text data for fine-tuning.	Mass-340K [10]: 335,645 WSIs and medical reports. TCGA [1]: Provides WSIs across cancer types (e.g., BRCA, NSCLC). In-house WSI Collections [31] [43]: For specific organs or rare diseases.
Synthetic Caption Generators	Augments limited text data by generating fine-grained descriptions for image regions.	PathChat [10]: A generative AI copilot used by TITAN to create 423k synthetic ROI captions, crucial for detailed vision-language alignment.
Text Prompt Templates	Used during zero-shot inference to convert class names into descriptive text the model can understand.	Templates like "an image of {CLASSNAME}" or "microscopic view of {CLASSNAME} cells" [31]. Ensembling multiple prompts per class improves robustness [1] [31].
Computing Framework	Manages the efficient processing of gigapixel WSIs and model training.	Feature Extraction Tools [31]: (e.g., CLAM) for segmenting tissue and extracting patch features from WSIs. Deep Learning Libraries: PyTorch or TensorFlow.

Zero-shot classification represents a paradigm shift in computational pathology, enabling the diagnosis of diseases without task-specific training data by leveraging semantic knowledge and auxiliary information [44]. This capability is particularly vital for diagnosing rare cancers and in settings where annotated data is scarce. Visual-language foundation models (VLFMs), trained on millions of image-text pairs, achieve this by aligning image and text representations in a shared semantic space, allowing for classification of unseen categories by matching images with textual descriptions of classes [1] [44]. However, the performance of these models is highly sensitive to the specific wording, or "prompts," used to represent class labels—a challenge known as the Prompt Sensitivity Problem. Variations in terminology, phrasing, or level of detail can lead to inconsistent and unpredictable model performance, potentially impacting diagnostic reliability [1]. This Application Note details the underlying causes of prompt sensitivity and provides structured experimental protocols and reagent solutions to develop robust, prompt-agnostic zero-shot classification systems for pathology research and drug development.

The Prompt Sensitivity Challenge in Pathology

In zero-shot classification, a model trained on a set of "seen" classes must generalize to "unseen" classes during inference. It does this by leveraging semantic side information—such as textual descriptions, attributes, or structured knowledge—to form a connection between visual features and novel class concepts [44]. The general workflow involves: (1) embedding both input images and class descriptions into a shared semantic space, (2) computing similarity scores between the input embedding and each class embedding, and (3) assigning the class with the highest similarity [44].

The core of the prompt sensitivity problem lies in the fact that the semantic embedding for a class is highly dependent on the specific natural language phrasing used in the prompt. For instance, a model might respond differently to "invasive lobular carcinoma of the breast" compared to "breast ILC," despite their clinical equivalence [1]. This sensitivity arises because:

• Language Model Embedding Variance: Language encoders generate distinct vector representations for different phrasings, even if they are semantically similar.
• Domain-Specific Terminology: The extensive vocabulary of pathology, including synonyms, acronyms, and nested hierarchical terms (e.g., "lung squamous cell carcinoma" is a subtype of "non-small cell lung cancer"), increases the likelihood of creating prompts that lie outside the model's optimized embedding space [45].
• Training Data Bias: Models are trained on specific corpora, making them more responsive to language styles and terms present in that data.

This variability poses a significant risk in clinical and research applications, where consistent and reliable performance is paramount. The following sections outline a systematic approach to quantify, mitigate, and overcome this challenge.

Quantitative Analysis of Prompt Sensitivity

To effectively address prompt sensitivity, researchers must first quantify its impact on model performance. The following experiment benchmarks the robustness of a visual-language foundation model against a range of prompt variations relevant to pathology tasks.

Experimental Protocol: Benchmarking Prompt Robustness

Objective: To evaluate the performance variance of a zero-shot classifier across systematically varied text prompts for cancer subtyping and tissue classification tasks.

Materials:

• Model: A pre-trained visual-language foundation model for pathology (e.g., CONCH [1] or KEEP [45]).
• Datasets:
  • TCGA NSCLC: For non-small cell lung cancer subtyping (Adenocarcinoma vs. Squamous Cell Carcinoma) [1].
  • TCGA RCC: For renal cell carcinoma subtyping [1].
  • CRC100K: For colorectal cancer tissue classification [1].
• Prompt Variations (per class):
  • Baseline: Standard class name (e.g., "lung adenocarcinoma").
  • Synonym: Medical synonym (e.g., "lung glandular cancer").
  • Formal: Formal pathological description (e.g., "a malignant epithelial tumor with glandular differentiation").
  • Acronym: Common acronym (e.g., "LUAD").
  • Hierarchical: Term including hypernym relation (e.g., "a subtype of non-small cell lung cancer") [45].

Methodology:

• Feature Extraction: For each image in the test set, compute the image embedding using the model's vision encoder.
• Text Embedding Generation: For each class in a task, generate text embeddings for all prompt variations using the model's language encoder.
• Similarity Calculation: Compute the cosine similarity between each image embedding and all class-prompt text embeddings.
• Prediction & Aggregation: For a single prompt, classify the image by selecting the class with the highest similarity score. For the ensemble method, average the similarity scores across all prompt variations for a given class before selecting the winning class.
• Evaluation: Calculate the balanced accuracy for each prompt variation and for the ensemble across all test images.

Expected Outcomes: Significant performance variance is expected across different prompts. The ensemble method should stabilize performance, achieving results that are competitive with or superior to the best single prompt.

Results and Comparative Analysis

Table 1: Performance variance of a VLFM (CONCH) across different prompt types on cancer subtyping tasks. Data presented as balanced accuracy (%).

Task / Prompt Type	Baseline	Synonym	Formal	Acronym	Hierarchical	Ensemble
TCGA NSCLC	90.7	88.2	85.9	82.5	87.4	91.5
TCGA RCC	90.2	88.7	86.1	84.3	89.5	91.1
CRC100K (ROI)	79.1	76.5	74.8	72.1	77.9	80.3

Table 2: Performance of KEEP, a knowledge-enhanced model, on rare cancer diagnosis, demonstrating the utility of structured knowledge. BA = Balanced Accuracy.

Model	Task	Metric	Performance
KEEP [45]	Subtyping 30 Rare Brain Cancers	Median BA	0.456
CONCH [45]	Subtyping 30 Rare Brain Cancers	Median BA	0.371

The data in Table 1 confirms the prompt sensitivity problem, with performance fluctuations exceeding 8 percentage points on the same task. The ensemble method consistently mitigates this issue. Furthermore, as shown in Table 2, models like KEEP that explicitly incorporate hierarchical knowledge show notably strong performance on challenging tasks like rare cancer subtyping, suggesting that integrating structured information is a powerful strategy for enhancing robustness [45].

Core Strategies for Robust Zero-Shot Classification

Strategy 1: Prompt Ensemble Methods

Principle: Aggregate predictions from multiple, semantically distinct prompts for a single class to smooth out variances and produce a more stable and accurate final prediction [1].

Implementation:

• Prompt Curation: For each class, generate a set of ( N ) prompts ( {P1, P2, ..., P_N} ) that cover the diversity of clinical terminology.
• Similarity Averaging: For a given image embedding ( I ) and class ( C ), compute the final similarity score as the mean of the similarities to all prompts for ( C ): ( S{final}(I, C) = \frac{1}{N} \sum{i=1}^{N} \text{cosine_sim}(I, E(P_i^C)) ), where ( E ) is the text encoder.
• Classification: Assign the class with the highest ( S_{final} ).

Considerations:

• The ensemble set should be clinically relevant and vetted by pathologists.
• While effective, this method increases computational cost linearly with the number of prompts.

Strategy 2: Knowledge-Enhanced Semantic Alignment

Principle: Move beyond simple image-text pairs by integrating structured domain knowledge to guide the model's learning process, creating a more nuanced and hierarchically-aware semantic space [45].

Implementation:

• Knowledge Graph (KG) Construction: Build or utilize a comprehensive disease ontology (e.g., from Disease Ontology [45]) that includes entities, synonyms, definitions, and hypernym relations (e.g., "X is a subtype of Y").
• Knowledge Encoding: Train a language model to encode this graph, learning the hierarchical relationships between diseases.
• Structured Pre-training: Reorganize noisy image-text pairs into semantic groups linked by the KG's hierarchical relations. During pre-training, align images not just with a single caption, but within the context of these structured semantic groups [45].

Visualization of Knowledge-Enhanced Workflow: The following diagram illustrates the architecture of a knowledge-enhanced foundation model like KEEP, which integrates a knowledge graph to refine vision-language alignment.

Strategy 3: Graph-Based Semantic Reasoning

Principle: Model the relationships among classes (both seen and unseen) explicitly using a semantic graph, and use label propagation algorithms to refine initial predictions and ensure coherence across related classes [44].

Implementation:

• Graph Construction: Create a graph where nodes represent all possible classes (e.g., diseases, tissue types). Connect nodes with edges weighted by their semantic similarity (e.g., cosine similarity of their canonical name embeddings, or based on a formal ontology).
• Zero-Shot Prediction: Obtain initial similarity scores from the VLFM for all classes.
• Label Propagation: Use a graph propagation algorithm (e.g., absorbing Markov chains) to diffuse these initial scores across the graph. This allows semantically related classes to reinforce each other, improving accuracy for fine-grained and rare subtypes [44].

Integrated Protocol for Robust Diagnosis

This protocol combines the above strategies into a comprehensive workflow for deploying a robust zero-shot classifier for cancer diagnosis using whole slide images (WSIs).

Objective: To perform accurate, slide-level cancer detection and subtyping in a zero-shot setting, minimizing the impact of prompt sensitivity.

Materials:

• VLFM: A knowledge-enhanced model like KEEP [45] or a robust base model like CONCH [1].
• Prompt Library: A pre-defined, curated set of prompt variations for all target diagnostic classes.
• Knowledge Graph: A pathology-specific ontology (e.g., Disease Ontology [45]).
• Computing Infrastructure: A GPU-enabled server capable of processing gigapixel WSIs.

Step-by-Step Workflow:

• WSI Tiling: Divide the input WSI into small, non-overlapping image tiles at a specified magnification (e.g., 20X).
• Tile-Level Embedding: Use the vision encoder of the VLFM to extract a feature vector for each tile.
• Prompt Ensemble Similarity Calculation:
  • For each diagnostic class, compute the tile-text similarity score using the ensemble method described in Section 4.1.
  • This produces a similarity score matrix: Tiles x Classes.
• Tile-Level Classification: Classify each tile by assigning the class with the highest ensemble similarity score.
• Slide-Level Aggregation:
  • Option 1 (Majority Voting): The most frequent tile-level prediction becomes the slide-level diagnosis.
  • Option 2 (Score Aggregation): Average the top-K similarity scores for each class across all tiles, then select the class with the highest average score.
• Graph-Based Refinement (Optional):
  • Construct a semantic graph of all possible diagnostic outcomes.
  • Use the initial slide-level similarity scores as a starting point for label propagation across this graph to produce the final, refined diagnosis.
• Visualization and Reporting:
  • Generate a heatmap of the WSI, visualizing the spatial distribution of the top predicted class or its similarity score to provide model explainability [1].
  • Report the final slide-level diagnosis and confidence metrics.

Visualization of Integrated Workflow: The following diagram summarizes the end-to-end protocol for robust zero-shot WSI classification.

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential computational tools and resources for developing robust zero-shot classifiers in computational pathology.

Reagent / Resource	Type	Function / Application	Source / Example
CONCH Model	Vision-Language Foundation Model	A foundational model for zero-shot tasks; serves as a strong baseline or feature extractor.	HuggingFace: MahmoodLab/CONCH [12]
KEEP Framework	Knowledge-Enhanced Foundation Model	A model architecture blueprint that integrates disease knowledge graphs for improved semantic alignment.	Code & models to be released (see ARXIV:2412.13126) [45]
Disease Ontology (DO)	Knowledge Graph	A structured, controlled vocabulary for human diseases, providing hierarchical relations and synonyms for knowledge enhancement.	Disease Ontology Consortium [45]
TCGA Datasets	Benchmark Data	Annotated whole slide images for various cancers; used for training and, critically, for evaluating model generalizability.	NIH Genomic Data Commons [1]
OpenPath & Quilt1M	Pretraining Data	Public collections of pathology image-caption pairs used for pre-training vision-language models.	Source: Public websites (Twitter, YouTube) [45]
Prompt Ensemble Library	Methodological Tool	A curated set of text prompts for each disease class, designed to cover clinical terminology variation and stabilize predictions.	Manually curated from textbooks and clinical reports [1]

In computational pathology, the development of robust visual-language foundation models for tasks such as zero-shot classification has been historically constrained by the scarcity of large-scale, expertly annotated histopathology image datasets. The annotation of whole-slide images (WSIs) is labor-intensive and not scalable to open-set recognition problems or rare diseases, which are common in pathology practice [1]. Recently, the strategic use of synthetic data has emerged as a transformative solution to these challenges. By leveraging Large Language Models (LLMs) and multimodal generative AI to generate captions, rewrite text, and create descriptive semantic prototypes, researchers can overcome data bottlenecks and enhance the performance of foundation models, enabling them to capture fine-grained pathological features with greater accuracy [10] [17]. This document details the application notes and experimental protocols for leveraging synthetic data in pathology AI research.

Application Notes

The Role of Synthetic Data in Pathology Foundation Models

Synthetic data serves multiple critical functions in the development of visual-language foundation models for pathology, from pretraining to zero-shot inference.

• Augmenting Pretraining Data: Foundation models require massive amounts of aligned image-text data. LLMs and generative AI copilots can generate synthetic captions for vast repositories of histopathology images that lack detailed textual descriptions. For instance, the TITAN model was pretrained using 423,122 synthetic captions generated from a multimodal generative AI copilot for pathology, which provided fine-grained morphological descriptions at the region-of-interest (ROI) level [10]. This approach scales pretraining datasets cost-effectively while preserving privacy.
• Enhancing Fine-Grained Classification: Zero-shot classification of histopathological images, especially for fine-grained tumor subtypes, requires high class separability in the model's semantic space. LLMs can generate rich, pathology-aware, class-specific text descriptions that act as superior semantic prototypes. The Fine-Grained Patch Alignment Network (FG-PAN) leverages LLM-generated fine-grained descriptions to significantly widen the gap between different class representations in the feature space, thereby improving zero-shot brain tumor subtype classification [17].
• Enabling Instruction Tuning for Interactive AI: The creation of interactive AI assistants for pathology requires datasets of instruction-following question-answer pairs. Synthetic data generation is pivotal for building these datasets without exhaustive manual effort. Models like HistoChat are instruction-tuned on datasets developed by generating question-answer pairs, enabling them to conduct open-ended conversations about cell distribution and other pathological features [46].

Key Advantages and Performance Metrics

The integration of LLM-generated synthetic data consistently leads to measurable improvements in model performance across diverse benchmarks.

Table 1: Performance Impact of Synthetic Data in Pathology Models

Model / Component	Synthetic Data Type	Task	Performance Improvement
TITAN [10]	423k synthetic ROI captions	Pathology report generation, zero-shot classification	Outperformed existing slide foundation models in low-data regimes and rare cancer retrieval.
FG-PAN [17]	LLM-generated fine-grained class descriptions	Zero-shot brain tumor subtype classification	Increased balanced accuracy on EBRAINS dataset from 0.493 (CONCH) to 0.572.
HistoChat [46]	Synthetically generated image-QA pairs	Cell-distribution analysis in colon histopathology	Achieved 69.1% accuracy in human evaluation, demonstrating efficacy with a small dataset of 231 images.
Open-source LLMs [47]	3000 synthetic thyroid nodule dictations	Free-text to structured data conversion in radiology	Performance comparable to GPT-4 (5-shot), with the Yi-34B model achieving an F1 score of 0.95.

Beyond quantitative metrics, synthetic data offers key advantages. It facilitates privacy preservation by allowing model development without direct use of sensitive patient data [47]. It also promotes scalability, as synthetic data generation can be massively scaled to cover rare diseases and diverse tissue types that are underrepresented in real-world datasets [10].

Experimental Protocols

Protocol 1: Generating Synthetic Fine-Grained Descriptions for Zero-Shot Classification

This protocol outlines the procedure for using an LLM to generate detailed, morphology-focused text descriptions for disease subtypes to improve zero-shot classification.

1. Define Classification Schema and Class Labels

• Identify the set of disease classes or histopathological patterns for classification (e.g., brain tumor subtypes: glioblastoma, oligodendroglioma, astrocytoma).
• For each class, compile a list of key histopathological terms and morphological features from medical literature and expert knowledge (e.g., "microvascular proliferation," "serpentine necrosis," "nuclear atypia") [17].

2. Construct LLM Prompts

• Design a prompt template that instructs the LLM to act as a expert pathologist. The prompt should request a comprehensive description of the given disease class, emphasizing visual morphological characteristics.
• Example Prompt: "You are a board-certified pathologist. Describe the detailed histopathological features of [Class Label] as seen on a hematoxylin and eosin (H&E) stained whole-slide image. Focus on cellular and tissue-level morphology, including cell appearance, tissue architecture, and key diagnostic features such as [List of Key Terms]. Do not mention the disease name in your description." [17]

3. Generate and Curate Descriptions

• Execute the prompt for each class label using a powerful LLM (e.g., GPT-4, Llama 3).
• Collect the generated descriptions. It is recommended to generate an ensemble of multiple descriptions per class (e.g., 5-10) using slight variations in the prompt to capture a broader semantic range [1].
• Optionally, have a pathologist review and refine the generated descriptions for accuracy.

4. Integrate with Vision-Language Model

• Use the generated fine-grained descriptions as the text prototypes for each class in a pre-trained visual-language model (e.g., CONCH).
• For a given WSI, compute the similarity between the image embedding and the text embeddings of all class descriptions.
• Classify the image by selecting the class with the highest similarity score [17].

Protocol 2: Generating Synthetic Captions for Vision-Language Pretraining

This protocol describes generating synthetic, fine-grained captions for histopathology image patches (ROIs) to augment the pretraining of foundation models.

1. Image Patch Selection

• Process whole-slide images to extract relevant region-of-interest (ROI) patches at high resolution (e.g., 8,192 x 8,192 pixels at 20x magnification) [10].
• Ensure the ROIs cover diverse tissue types, staining variations, and disease states.

2. Leverage a Multimodal Generative AI Copilot

• Employ a specialized multimodal model capable of understanding histopathology images and generating text.
• Input the ROI image to the model with a carefully designed instruction prompt that requests a detailed morphological description.
• Example Prompt: "Describe the histopathological features in this image patch in detail. Include observations on tissue architecture, cell morphology, and any notable pathological findings." [10]

3. Data Curation and Quality Control

• Collect the generated captions to form image-text pairs.
• Implement a quality control step, which can involve:
  • Automated Filtering: Using a separate model to check for coherence and relevance.
  • Pathologist-in-the-Loop: A subset of generated captions can be reviewed by an expert for accuracy and used to fine-tune the captioning model iteratively.
• The final curated set of synthetic image-caption pairs can be used alongside any available real report data for large-scale multimodal pretraining [10].

Workflow Visualization

The following diagram illustrates the integrated workflow for leveraging synthetic data in a pathology visual-language foundation model, from data generation to zero-shot inference.

The Scientist's Toolkit

This section details the essential research reagents, models, and datasets used in the featured experiments for leveraging synthetic data in computational pathology.

Table 2: Essential Research Reagents and Solutions

Item Name	Type	Function & Application Notes
CONCH [1] [12]	Vision-Language Foundation Model	A foundational model pretrained on 1.17M histopathology image-caption pairs. Serves as a robust base for transfer learning, zero-shot classification, and feature extraction.
TITAN [10]	Multimodal Whole-Slide Foundation Model	A transformer-based model designed to encode entire WSIs. It demonstrates the use of 423k synthetic captions for pretraining, enabling superior slide-level tasks.
LLMs (e.g., GPT-4, Llama 3) [17] [47]	Large Language Model	Used as a "semantic engine" to generate fine-grained class descriptions, rewrite text, and create synthetic pathology reports based on expert-crafted prompts.
PathChat [10]	Multimodal Generative AI Copilot	A specialized model for pathology used in the TITAN pipeline to generate fine-grained, synthetic captions for histopathology ROIs.
Lizard Dataset [46]	Histopathology Dataset	Provides annotated data on cell types and distributions in colon tissue. Serves as a base for generating synthetic question-answer pairs for instruction tuning.
ST-bank Dataset [48]	Spatial Transcriptomics Dataset	A curated dataset of 2.2M tissue patches with paired H&E images and transcriptomics data. Used for training visual-omics models like OmiCLIP.

The adoption of whole-slide images (WSIs) in computational pathology represents a paradigm shift from traditional microscopy, enabling the application of artificial intelligence (AI) for diagnostic and research purposes. WSIs are gigapixel-sized digital scans of entire glass slides, producing high-resolution images that can span several gigabytes per file [49] [50]. While these detailed images capture comprehensive tissue information necessary for accurate diagnosis, their massive size creates significant computational challenges for storage, transmission, and analysis. Within the context of zero-shot classification using visual-language foundation models, efficient handling of WSIs is not merely an engineering concern but a fundamental prerequisite for enabling real-time inference and practical deployment in clinical and research settings. This document outlines standardized protocols and optimized methodologies for managing WSIs efficiently while maintaining diagnostic integrity.

Storage and Compression Strategies for WSIs

The WSI Storage Challenge

Whole-slide images impose substantial burdens on digital infrastructure. A single WSI can generate files ranging from 1 to 5 gigabytes, and when considering longitudinal patient records that must be maintained for decades, the cumulative storage requirements become enormous [50]. Furthermore, in cloud-based medical ecosystems, these large files must frequently be transferred between institutions for collaborative diagnosis, second opinions, or multi-center research, creating bandwidth bottlenecks [50].

Table 1: Comparison of Lossless Compression Methods for WSIs

Compression Method	Type	Average Compression Ratio	Key Characteristics
PNG	Image-specific	~1x	Ineffective for WSI content
LZMA	Dictionary-based	~2x	Better handles WSI volatility
Huffman Coding	Entropy-based	~1.23x	Limited effectiveness
Neural Network-based	Predictive	1.6-2x	Limited by high entropy
WISE Framework	Hierarchical/Dictionary	36x (up to 136x)	Specifically designed for WSI

Lossless Compression Protocols

While lossy compression techniques exist, they risk altering diagnostically critical information and are generally unsuitable for primary diagnosis [50]. Therefore, lossless compression that preserves all original image data is essential. Standard lossless compressors like PNG perform poorly on WSIs due to the unique "information irregularity" in pathological images, characterized by high-frequency signals distributed widely across the image with high volatility [50].

Protocol 1: WISE Compression Framework The WISE (Whole-slide Image lossless Compression) framework employs a hierarchical encoding strategy specifically designed for WSI characteristics [50]:

• Input Preparation: Begin with a full, uncompressed WSI in standard format (e.g., SVS, TIFF).
• Background Identification and Removal:
  • Convert the color image to grayscale.
  • Apply binarization, assigning 0 to background pixels and 1 to foreground (tissue) pixels.
  • Perform morphological operations (hole-filling and dilation) to refine the foreground mask.
  • Extract connected components to identify all tissue-containing regions.
• Hierarchical Projection Coding:
  • Process the identified tissue regions through a hierarchical projection to reduce entropy.
  • This step effectively separates and prepares the data for efficient dictionary encoding.
• Bitmap Coding: Apply bitmap coding to further disentangle locality patterns optimal for dictionary-based compression.
• Dictionary-Based Compression: Implement a final compression pass using optimized dictionary methods (e.g., LZ77 variants) to achieve the final compressed output.

This method has demonstrated an average compression ratio of 36x, reaching up to 136x for certain WSIs, significantly outperforming generic compression tools [50].

Protocol 2: Background Removal and Tiling Algorithm An alternative or complementary approach focuses on eliminating non-diagnostic background areas [51]:

• Image Binarization: Convert the WSI to a binary mask using optimal thresholding.
• Morphological Processing: Apply dilation to ensure all tissue peripheries are included.
• Connected Component Analysis: Identify all discrete tissue regions on the slide.
• Bounding Box Extraction: Calculate the smallest rectangular bounding boxes that enclose each tissue component.
• Optimal Rectangle Packing: Use a rectangle-packing algorithm (e.g., rectangle-packer package) to arrange all bounding boxes into the smallest possible composite image.
• Cropping and Assembly: Crop the tissue regions from the original WSI using the coordinates of the packed bounding boxes and assemble them into a new, significantly smaller image file.

This algorithm has achieved a mean size reduction of 7.11x while preserving the original resolution of the tissue areas, and has been validated to maintain the performance of downstream deep learning models for tasks like dysplasia grading [51].

Computational Architectures for Zero-Shot Classification

Efficient WSI handling is critical for vision-language foundation models (VLMs) that perform zero-shot tasks, such as diagnosing unseen tumor subtypes or predicting genetic mutations without task-specific training.

Model Architecture and Workflow

VLMs like HistoGPT, CONCH, and Quilt-LLaVA are designed to process gigapixel WSIs and generate diagnostic reports or classifications through integrated vision and language components [52] [9]. Their efficiency stems from specialized architectures that avoid processing the entire image at full resolution simultaneously.

Diagram 1: VLM Architecture for Zero-Shot WSI Analysis

Table 2: Foundation Models for Zero-Shot Pathology Tasks

Model	Architecture	Parameter Count	Key Features	Supported Tasks
HistoGPT	Vision + Autoregressive LLM	~430M (Small)	Processes multiple WSIs; Generates full reports	Tumor subtyping, Thickness, Margins [52]
CONCH	Contrastive Captioning (CoCa)	~200M	Excels with precise anatomical prompts	Cancer subtyping, Prognosis [9]
Quilt-LLaVA	Visual Encoder + Large LLM	~7B	Generative Q&A capabilities	Interactive diagnosis [9]
Quilt-Net	Contrastive (CLIP-based)	~150M	Dual-encoder for image-text alignment	Classification, Retrieval [9]

Efficient Inference Protocol for Zero-Shot Tasks

Protocol 3: Zero-Shot Inference with Prompt Optimization Effective prompt design dramatically impacts the performance and efficiency of VLMs on zero-shot tasks [9]:

• Feature Extraction (Offline/Preprocessing):

  • Use a pre-trained vision backbone (e.g., CTransPath, UNI) to extract features from tissue-containing patches at 10× or 20× magnification [52].
  • Employ a Perceiver Resampler or similar token-consuming layer to aggregate thousands of patch features into a manageable number of slide-level visual tokens [52].

• Prompt Design and Optimization:

  • Domain Specificity: Use precise medical terminology (e.g., "invasive ductal carcinoma" vs. "breast cancer").
  • Anatomical Precision: Include specific anatomical references when available (e.g., "colonic mucosa," "lymph node cortex").
  • Instruction Framing: Structure the prompt as an explicit instruction (e.g., "Describe the tumor morphology and estimate the tumor thickness.").
  • Output Constraints: Specify the desired format (e.g., "Respond with one of the following classes: benign, dysplasia, invasive carcinoma.").

• Cross-Modal Fusion and Inference:

  • Feed the processed visual tokens and text prompt tokens into the cross-attention blocks of the model.
  • The language model generates the output autoregressively, attending to visual features at each step.

• Ensemble Refinement (Optional for Report Generation):

  • To avoid fixed text structures, sample multiple reports with different random seeds.
  • Use a general-purpose LLM (e.g., GPT-4) to aggregate the sampled reports into a final, comprehensive version [52].

Protocol 4: Cross-Modal Knowledge Representation (CMKR) For complex zero-shot classification tasks, the CMKR framework enhances performance by leveraging both implicit and explicit knowledge [53]:

• Implicit Knowledge Extraction: Use a large multimodal model to generate textual descriptions and features from the WSI.
• Explicit Knowledge Integration: Retrieve structured information from biomedical knowledge graphs (e.g., disease ontologies, anatomical hierarchies).
• Cross-Modal Alignment: Apply a contrastive loss to align the image features, implicit text features, and explicit knowledge graph embeddings in a shared representation space. This enforces that semantically similar concepts from different modalities are close together.
• Zero-Shot Classification: The aligned multi-modal representation enables accurate classification of concepts not seen during training by leveraging the rich semantic relationships encoded in the shared space.

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Computational Tools for WSI Analysis

Resource / Tool	Type	Primary Function	Application in Zero-Shot Learning
CTransPath	Vision Backbone	Feature extraction from WSI patches [52]	Creates foundational visual representations for VLMs.
UNI	Vision Backbone	Large-scale feature extraction [52]	Provides high-quality visual features for complex tasks.
BioGPT	Language Model	Biomedical text generation [52]	Serves as the language backbone for generative reports.
Perceiver Resampler	Neural Module	Aggregates patch features into slide-level tokens [52]	Drastically reduces computational load for downstream processing.
WISE Compressor	Compression Tool	Lossless WSI size reduction [50]	Enables efficient storage and faster transfer of WSIs for analysis.
Rectangle Packing Algorithm	Preprocessing	Removes background and assembles tissue regions [51]	Reduces effective image size before model processing.

Diagram 2: WSI Analysis Workflow

Computational efficiency is not an ancillary concern but a central enabler for the practical application of zero-shot visual-language models in pathology. By implementing the structured protocols outlined here—including advanced lossless compression like WISE, background removal algorithms, and efficient feature extraction pipelines—researchers can overcome the significant data bottlenecks presented by gigapixel WSIs. Furthermore, understanding the architecture of modern foundation models and applying systematic prompt engineering allows for effective zero-shot inference on tasks ranging from tumor classification to comprehensive report generation. These optimized workflows ensure that the transformative potential of AI in pathology can be realized in real-world clinical and research environments, ultimately accelerating drug development and improving diagnostic precision.

Context Modulation and Other Architectural Innovations for Enhanced Alignment

The application of visual-language foundation models to zero-shot classification in pathology represents a paradigm shift in computational pathology. These models, pre-trained on vast datasets of image-text pairs, learn a shared embedding space where visual concepts and their textual descriptions are aligned. This alignment enables zero-shot inference—the ability to recognize and classify pathological entities without task-specific training data. However, a significant adaptation gap often exists between the general-domain knowledge of foundation models and the fine-grained, specialized requirements of medical image analysis. Context modulation refers to a class of architectural innovations designed to dynamically adjust the flow of information and the interactions between vision and language modalities based on the specific context of the input data. This article details the core architectural principles, experimental protocols, and key reagent solutions for implementing context modulation to achieve enhanced alignment in pathology vision-language models.

Core Architectural Principles and Quantitative Performance

Innovations in model architecture are crucial for bridging the adaptation gap in medical zero-shot learning. The table below summarizes key architectural innovations and their demonstrated performance.

Table 1: Architectural Innovations for Enhanced Alignment in Zero-Shot Pathology Classification

Architectural Innovation	Core Mechanism	Model/Example	Reported Performance Gains	Key Applicability
Random Sentence Sampling [26]	Randomly sub-samples sentences from a pathology report during training, enabling the image to align with multiple positive text descriptors.	Fine-tuned CLIP-style models on chest X-rays [26]	Average macro AUROC increase of 4.3% across four chest X-ray datasets; surpassed board-certified radiologists on five CheXpert pathologies [26].	Training with multi-sentence medical reports
Loss Relaxation [26]	Modifies the contrastive loss function to clip the upper bound of similarity, reducing the penalty for false-negative pairs with partial semantic overlap.	Fine-tuned CLIP-style models on chest X-rays [26]	As above; mitigates sub-optimal representation learning from multi-labeled image-report pairs [26].	Multi-label medical datasets
Parameter-Efficient Conv-LoRA Adapter [54]	Injects local inductive biases into Vision Transformers (ViTs) via a parallel multi-branch convolutional design within a Low-Rank Adaptation (LoRA) bottleneck.	ACD-CLIP for Zero-Shot Anomaly Detection (ZSAD) [54]	Provides fine-grained visual details critical for dense prediction tasks like segmentation [54].	Adapting ViT-based encoders for fine-grained medical imagery
Dynamic Fusion Gateway (DFG) [54]	A vision-guided mechanism that dynamically weights and fuses multi-level text features to generate context-aware semantic descriptors for each visual feature group.	ACD-CLIP for Zero-Shot Anomaly Detection (ZSAD) [54]	Enables flexible, bidirectional fusion policy, overcoming static layer-wise alignment [54].	Complex tasks requiring adaptive, context-dependent vision-language interaction
Visual-Language Foundation Pretraining [3] [25]	Large-scale, task-agnostic pretraining on domain-specific image-caption pairs to learn a foundational, aligned representation.	CONCH (Histopathology) [3] [25]	State-of-the-art (SOTA) on 14 benchmarks for classification, segmentation, and retrieval [3].	Building powerful base models for pathology

Diagram 1: Context Modulation and Architectural Innovation Workflow

Experimental Protocols for Key Methodologies

Protocol: Fine-Tuning with Random Sentence Sampling and Loss Relaxation

This protocol is designed for adapting a pre-trained visual-language model (e.g., CLIP) to a dataset of medical image-report pairs for zero-shot pathology classification [26].

• Data Preparation:

  • Collect a dataset of medical images (e.g., chest X-rays, whole slide images) and their corresponding textual reports.
  • Pre-process the text reports by segmenting them into individual sentences using a natural language processing (NLP) library like spaCy.

• Model Setup:

  • Load a pre-trained visual-language model, which includes an image encoder ((E{img})) and a text encoder ((E{txt})).

• Training Loop with Random Sentence Sampling:

  • For each image-text pair in a training batch:
    • From the associated report containing (m) sentences, randomly sub-sample (n) sentences (e.g., (n=1) or (2)).
    • The sub-sampled sentences form the positive text for the image in this training iteration.
  • Encode the image and the sampled text to get their embeddings.

• Loss Calculation with Relaxation:

  • Compute the standard InfoNCE contrastive loss ((\mathcal{L})), which aims to maximize the similarity of positive image-text pairs and minimize the similarity of negative pairs in the batch [26].
  • Apply loss relaxation by modifying the similarity function (({\textbf {sim}})) used in the loss calculation. This is done by clipping the upper bound of the similarity, preventing the model from over-penalizing so-called "false-negative" pairs that may have partial semantic overlap [26].

• Inference for Zero-Shot Classification:

  • For a target pathology (c), create positive and negative text prompts (e.g., "{label}" and "no {label}").
  • Compute the image embedding (E{img}(I{trg})) for the target image.
  • Compute the text embeddings (E{txt}(T{+, c})) and (E{txt}(T{-, c})) for the prompts.
  • Calculate the cosine similarity between the image embedding and each prompt embedding.
  • The probability for pathology (c) is obtained via softmax over the two similarity scores [26].

Protocol: Architectural Co-Design with ACD-CLIP for Anomaly Detection

This protocol outlines the procedure for implementing the Architectural Co-Design CLIP (ACD-CLIP) framework, which is highly relevant for dense prediction tasks in pathology, such as localizing anomalous regions [54].

• Model Partitioning:

  • Partition the vision and text encoders of a pre-trained VLM (e.g., CLIP) into (N) sequential groups.

• Integration of Conv-LoRA Adapters:

  • Into each vision group, integrate a Conv-LoRA Adapter.
  • The adapter processes the output of a ViT block. It uses a parallel multi-branch design (e.g., with (3\times3) and (5\times5) convolutions) within a LoRA bottleneck to capture multi-scale local patterns.
  • The output of the adapter is a residual update ((\Delta X)) that is added to the main vision stream, enhancing it with fine-grained spatial details [54].

• Dynamic Fusion Gateway (DFG) Implementation:

  • For each vision feature group (V_i), extract a global context vector.
  • Pass this vector through an MLP to generate logits, which are normalized via Softmax to produce dynamic weight vectors ((\bm{\omega}i^N, \bm{\omega}i^A)) over all (N) text feature levels.
  • Use these weights to dynamically fuse the multi-level text features, creating context-aware "normal" ((Ti^N)) and "abnormal" ((Ti^A)) descriptors for each vision group [54].

• Training Objective:

  • Compute a segmentation loss ((\mathcal{L}_{seg})), such as a combination of Focal and Dice loss, between the predicted anomaly map and the ground-truth mask.
  • Compute a classification loss ((\mathcal{L}_{cls})) using cross-entropy on the cosine similarity between the final global visual feature and the original, unfused text features.
  • The total training objective is a weighted sum: (\mathcal{L}{total} = \mathcal{L}{seg} + \lambda{cls}\mathcal{L}{cls}) [54].

Diagram 2: ACD-CLIP Framework for Dense Prediction

The Scientist's Toolkit: Research Reagent Solutions

The following table catalogues essential "research reagents"—datasets, models, and software tools—for developing and experimenting with alignment innovations in pathology.

Table 2: Essential Research Reagent Solutions for Pathology VLM Research

Reagent Solution	Type	Primary Function in Research	Example Instances
Pathology VLMs	Foundation Model	Provides the base pre-trained model with aligned image-text representations for transfer learning and zero-shot evaluation.	CONCH (Histopathology) [3] [25], CLIP (General Domain) [26] [54]
Medical Image-Text Datasets	Dataset	Serves as the training and evaluation corpus for fine-tuning models and benchmarking performance.	MIMIC-CXR (Chest X-rays & Reports) [26], MZSL-50 (Multimodal Zero-Shot Videos) [55], TCGA (Whole-Slide Images) [3]
Interpretability & Visualization Toolkits	Software Library	Enables model debugging and explanation by visualizing feature maps, attentions, and saliency.	PyTorchViz (Architecture Graphs) [56], Grad-CAM & Saliency Map libraries [57]
Parameter-Efficient Fine-Tuning (PEFT) Libraries	Software Library	Facilitates the integration and training of adapters (e.g., LoRA, Conv-LoRA) with minimal computational overhead.	Libraries supporting LoRA and custom adapter integration (e.g., Hugging Face PEFT) [54]
Whole Slide Image (WSI) Processing Frameworks	Software Library	Manages the loading, tiling, and analysis of gigapixel-sized whole slide images for model input.	OpenSlide, TIAToolbox

Benchmarking Performance: Zero-Shot Models vs. Supervised Learning and Human Experts

The emergence of vision-language models (VLMs) has introduced a new paradigm in artificial intelligence (AI), showing particular promise for specialized domains like digital pathology [58] [59] [60]. However, the initial development and evaluation of these models predominantly focused on general-purpose datasets, providing limited understanding of their effectiveness in the complex, high-stakes field of histopathology interpretation [58]. This gap highlighted the urgent need for specialized, high-quality benchmarks that could accurately measure model capabilities in pathological reasoning and understanding.

The establishment of robust evaluation frameworks is especially critical within the context of zero-shot classification with visual-language foundation models in pathology research. Zero-shot evaluation tests a model's inherent ability to understand and reason about pathology images without task-specific training, providing a true measure of its generalization capabilities and potential for real-world clinical application [58] [4]. To address these needs, two complementary frameworks have emerged: PathMMU as a massive multimodal expert-level benchmark, and PathVLM-Eval as a comprehensive evaluation system for benchmarking VLM performance on this challenging dataset [61] [58].

Benchmark Framework Specifications

PathMMU: A Massive Multimodal Expert-Level Benchmark

PathMMU represents the largest and highest-quality expert-validated pathology benchmark for Large Multimodal Models (LMMs) [61] [62]. Its construction harnessed GPT-4V's advanced capabilities in a cascading process that utilized over 30,000 image-caption pairs to enrich captions and generate corresponding question-and-answer sets [61]. To maximize the benchmark's authority, seven pathologists rigorously scrutinized each question under strict standards in PathMMU's validation and test sets, simultaneously setting an expert-level performance benchmark for comparison [61] [62].

The benchmark's structure encompasses several key components:

• Total Volume: 33,428 multimodal multiple-choice questions and 24,067 images sourced from diverse pathological resources [61]
• Question Format: Multiple-choice questions (MCQs) designed to aid pathologists in diagnostic reasoning and support professional development initiatives in histopathology [58]
• Explanatory Support: Each question includes an explanation for the correct answer, facilitating deeper understanding and model interpretability [61]
• Subset Composition: Includes specialized subsets such as PubMed, SocialPath, and EduContent, featuring diverse formats and difficulty levels [58]

Table 1: PathMMU Benchmark Composition

Component	Scale/Number	Description
Multimodal MCQs	33,428 questions	Multiple-choice questions with images
Pathology Images	24,067 images	Sourced from diverse pathological resources
Pathologist Validators	7 experts	Rigorous scrutiny of validation/test sets
Human Performance Benchmark	71.8% accuracy	Expert-level performance baseline [61]

PathVLM-Eval: Comprehensive Evaluation System

The PathVLM-Eval framework was developed to conduct extensive benchmarking of VLMs on histopathology image understanding using the PathMMU dataset [58] [59]. This system utilizes VLMEvalKit, a widely used open-source evaluation framework, to bring publicly available pathology datasets under a single evaluation umbrella, ensuring unbiased and contamination-free assessments of model performance [58] [60].

The framework's key characteristics include:

• Model Coverage: Evaluation of more than 60 state-of-the-art visual language models, significantly expanding the range assessed in prior literature [58]
• Model Diversity: Inclusion of models from leading families including LLaVA, Qwen-VL, Qwen2-VL, InternVL, Phi3, Llama3, MOLMO, and XComposer series [58] [60]
• Zero-shot Focus: Strict zero-shot evaluation across all models to assess inherent capabilities without task-specific fine-tuning [58]
• Public Leaderboard: Complete evaluation results available through a public leaderboard for ongoing comparison and transparency [58] [59]

Performance Evaluation and Key Findings

Quantitative Performance Analysis

The extensive evaluations conducted through PathVLM-Eval reveal several crucial insights about current VLM capabilities in pathological understanding. The empirical findings indicate that even advanced LMMs struggle with the challenging PathMMU benchmark, with the top-performing LMM, GPT-4V, achieving only a 49.8% zero-shot performance, significantly lower than the 71.8% demonstrated by human pathologists [61] [62].

Among the open-source models tested in the PathVLM-Eval benchmark, Qwen2-VL-72B-Instruct achieved superior performance with an average score of 63.97%, outperforming other models across all PathMMU subsets [58] [59] [60]. This performance is particularly notable as it substantially exceeds GPT-4V's results, though still trails human expert performance.

Table 2: Key Performance Results on PathMMU Benchmark

Model/Human	Zero-shot Accuracy	Parameters	Notes
Human Pathologists	71.8%	N/A	Expert-level performance benchmark [61]
Qwen2-VL-72B-Instruct	63.97%	72B	Top-performing open-source model [58]
GPT-4V	49.8%	Proprietary	Top-performing closed-source model [61]
Fine-tuned Smaller LMMs	>49.8%	Varies	Can outperform GPT-4V but still below humans [61]

Specialized Clinical Application: Ki-67 Index Prediction

Beyond general pathological understanding, benchmarking efforts have extended to specific clinical tasks such as Ki-67 proliferation index quantification in breast cancer histopathology [63]. This task is crucial for prognosis but remains subjective and labor-intensive when performed manually [63].

In a strict zero-shot setting evaluating eight commercial models using guideline-based prompts, substantial differences between providers emerged [63]. On the BCData dataset (n=402), OpenAI's GPT-4.5 achieved the best concordance with expert annotations (R²=0.8570, RMSE=7.9708) and reached a macro-F1 of 74.56% for three-class cut-off classification (Low: Ki-67 <16%, Medium: 16%≤Ki-67<30%, High: Ki-67≥30%) [63]. On the SHIDC-B-Ki-67 dataset (n=700), GPT-4.1 mini led with R²=0.5465, RMSE=16.8202, and macro-F1 of 66.86% across the same ranges [63].

The Impact of Prompt Design on Zero-shot Performance

Recent investigations have systematically examined how prompt design affects zero-shot diagnostic pathology in VLMs [4]. Through structured ablative studies on cancer invasiveness and dysplasia status, researchers developed a comprehensive prompt engineering framework that systematically varies domain specificity, anatomical precision, instructional framing, and output constraints [4].

The findings demonstrate that prompt engineering significantly impacts model performance, with the CONCH model achieving the highest accuracy when provided with precise anatomical references [4]. Performance consistently degraded when reducing anatomical precision, highlighting the critical importance of anatomical context in histopathological image analysis [4]. This research establishes foundational guidelines for prompt engineering in computational pathology and highlights the potential of VLMs to enhance diagnostic accuracy when properly instructed with domain-appropriate prompts.

Experimental Protocols and Methodologies

Zero-shot Evaluation Protocol

The standard zero-shot evaluation protocol employed in PathVLM-Eval involves several critical steps to ensure consistent, unbiased assessment of model capabilities [58] [60]:

• Model Preparation: Models are evaluated without any task-specific fine-tuning using their standard pre-trained weights
• Prompt Standardization: A unified, guideline-based prompt is employed across all evaluations to ensure consistency [63]
• Evaluation Framework: Utilization of VLMEvalKit for standardized assessment and metric calculation [58]
• Input Modalities: Processing of both image and text question inputs through the models' multimodal architecture
• Output Assessment: Comparison of model-generated answers against expert-validated ground truth responses

Robustness Evaluation Methodology

To understand model reliance on visual inputs and their robustness to image quality variations, PathVLM-Eval incorporates evaluations under different visual conditions [60]:

• Original Image: Evaluation with unmodified pathology images
• Blackout: Complete removal of visual information to test textual reasoning capabilities
• Fully Blurred: Heavy blurring applied to entire images
• Partially Blurred: Selective blurring of specific image regions

This evaluation sheds light on the ability of VLMs to process textual information when visual cues are degraded or absent, highlighting a limitation in current histopathology benchmarking datasets which requires future work to increase the dependency between textual questions and images [60].

Advanced Training Methodologies

Recent advancements have introduced specialized training approaches to enhance VLM performance on pathology-specific tasks. The PathVLM-R1 model employs a reinforcement learning-driven approach with several innovative components [64]:

• Supervised Fine-tuning Phase: Initial knowledge injection using pathological data to imbue the model with foundational pathological knowledge
• Dual Reward Mechanism: Incorporation of cross-modal process rewards to supervise reasoning quality and outcome accuracy rewards to evaluate final answers
• Group Relative Policy Optimization (GRPO): Reinforcement learning algorithm that enhances model reasoning by optimizing policy gradients through group-based relative advantage estimation and regularization [64]

This approach has demonstrated significant improvements, with PathVLM-R1 showing a 14% improvement in accuracy compared to baseline methods and superior performance compared to the Qwen2-VL-32B version despite having significantly fewer parameters [64].

Visualization of Benchmark Workflows

PathMMU Dataset Creation Workflow

PathVLM-Eval Assessment Workflow

Prompt Engineering Framework

Table 3: Key Research Reagents and Computational Resources

Resource Name	Type/Function	Application in Pathology VLM Research
PathMMU Dataset	Expert-validated benchmark	Gold-standard evaluation for pathology reasoning capabilities [61]
VLMEvalKit	Open-source evaluation framework	Standardized assessment of VLM performance on pathology tasks [58]
Qwen2-VL Model Series	Vision-language models	State-of-the-art open-source architecture for pathology image understanding [58]
PathVLM-R1	Specialized pathology VLM	Reinforcement learning-optimized model for pathological reasoning [64]
LLM-Generated Rewrites	Data augmentation technique	Enhancing pathology image-caption datasets for improved training [65]
Dual Reward Mechanism	Training methodology	Combines process and outcome rewards for better reasoning transparency [64]

The establishment of the PathMMU and PathVLM-Eval frameworks represents a significant advancement in the rigorous evaluation of vision-language models for pathology research. These complementary systems provide the specialized benchmarking infrastructure necessary to drive meaningful progress in zero-shot classification capabilities for histopathology images. The comprehensive evaluations conducted to date reveal that while current VLMs show promising capabilities, they still significantly trail human expert performance, highlighting the need for continued research and development.

The findings from these benchmark frameworks offer valuable guidance for researchers and drug development professionals working to implement VLM technologies in pathological applications. The demonstrated impact of factors such as model architecture, scale, prompt engineering, and training methodology provide concrete directions for future work. As these evaluation frameworks continue to evolve and incorporate more diverse pathological tasks and datasets, they will undoubtedly accelerate the development of more capable, reliable, and clinically useful vision-language models for pathology.

The adoption of visual-language foundation models (VLMs) represents a paradigm shift in computational pathology, enabling powerful zero-shot classification capabilities. These models leverage aligned image and text embeddings to make diagnostic predictions without task-specific training data. Evaluating their performance, however, requires careful consideration of specialized metrics including the Area Under the Receiver Operating Characteristic Curve (AUROC), accuracy, and retrieval scores, which provide complementary insights into model behavior and clinical utility [9] [66]. Proper metric selection and interpretation is essential for benchmarking model performance across diverse diagnostic tasks including cancer subtyping, biomarker prediction, and disease prognosis [66].

This application note provides a structured framework for quantifying and interpreting model performance in zero-shot pathology applications, with specific protocols for implementing these assessments in research settings.

Performance Metrics for Zero-Shot Classification

Core Metric Definitions and Interpretation

Table 1: Core Performance Metrics for Zero-Shot Pathology Classification

Metric	Calculation	Interpretation	Strengths	Limitations
AUROC	Area under ROC curve plotting TPR vs. FPR across thresholds	Probability that random positive sample ranks higher than random negative sample; value 0.5-1.0	Threshold-independent; robust to class imbalance	May overestimate performance in severe class imbalance
Accuracy	(TP + TN) / (TP + TN + FP + FN)	Proportion of correct predictions among total predictions	Intuitive interpretation; simple calculation	Misleading with class imbalance; threshold-dependent
Precision	TP / (TP + FP)	Proportion of true positives among predicted positives	Measures false positive rate; crucial for screening	Sensitive to data distribution shifts
Recall (Sensitivity)	TP / (TP + FN)	Proportion of actual positives correctly identified	Measures false negative rate; crucial for diagnosis	Trade-off with precision
F1 Score	2 × (Precision × Recall) / (Precision + Recall)	Harmonic mean of precision and recall	Balanced measure for binary classification	Does not account for true negatives
AUPRC	Area under precision-recall curve	Performance in imbalanced datasets; value 0.5-1.0	More informative than AUROC for imbalanced data	More complex interpretation than AUROC

For zero-shot classification, AUROC values typically range from 0.63-0.77 across different diagnostic tasks, with vision-language models like CONCH achieving AUROCs of 0.77 for morphology tasks, 0.73 for biomarker prediction, and 0.63 for prognosis tasks [66]. Accuracy must be interpreted cautiously in imbalanced datasets, where a high accuracy may mask poor performance on minority classes.

Benchmarking Foundation Models

Table 2: Performance Benchmark of Pathology Foundation Models Across Task Types

Foundation Model	Model Type	Morphology AUROC	Biomarker AUROC	Prognosis AUROC	Overall AUROC
CONCH	Vision-Language	0.77	0.73	0.63	0.71
Virchow2	Vision-Only	0.76	0.73	0.61	0.71
Prov-GigaPath	Vision-Only	0.69	0.72	0.66	0.69
DinoSSLPath	Vision-Only	0.76	0.67	0.63	0.69
UNI	Vision-Only	0.68	0.68	0.67	0.68
PLIP	Vision-Language	0.64	0.64	0.63	0.64

Recent benchmarking of 19 foundation models across 31 clinical tasks on 6,818 patients revealed that CONCH and Virchow2 achieved the highest overall AUROC of 0.71, significantly outperforming other models in numerous tasks [66]. Vision-language models like CONCH demonstrate particular strength in morphology-related tasks (AUROC: 0.77), while larger vision-only models excel in biomarker prediction [66].

Experimental Protocols for Metric Evaluation

Protocol 1: Zero-Shot Classification Using VLMs

Purpose: To evaluate the performance of vision-language models on pathology classification tasks without task-specific training.

Materials:

• Whole Slide Images (WSIs) in SVS or compatible formats
• Pre-trained VLM (e.g., CONCH, Quilt-Net, Quilt-LLaVA)
• Computational resources with GPU acceleration
• Prompt engineering framework

Procedure:

• WSI Preprocessing: Tessellate WSIs into non-overlapping patches (typically 512×512 pixels at 20× magnification) [10].
• Feature Extraction: Extract patch-level features using pre-trained vision encoders (e.g., CONCHv1.5 generates 768-dimensional features per patch) [10].
• Prompt Design: Systematically vary prompts across domain specificity, anatomical precision, instructional framing, and output constraints [9].
• Similarity Calculation: Compute cosine similarity between image embeddings and class description embeddings in the joint vision-language space [9].
• Prediction: Select class with highest similarity score as predicted label.
• Performance Calculation: Compare predictions with ground truth labels to compute AUROC, accuracy, precision, recall, and F1 scores.

Expected Outcomes: CONCH typically achieves zero-shot performance improvements of 15-20% on cancer subtyping compared to models without hierarchical visual processing [9]. Prompt engineering significantly impacts performance, with precise anatomical references improving accuracy [9].

Protocol 2: Content-Based Image Retrieval (CBIR) Evaluation

Purpose: To assess retrieval performance for similar case finding and rare disease diagnosis.

Materials:

• Reference database of annotated pathology images
• Feature extraction backbone (e.g., YOLOv5 + EfficientNet fusion)
• Similarity measurement algorithm (cosine, Euclidean)

Procedure:

• Feature Representation: Extract compact, descriptive image representations using deep learning-based feature extractors [67].
• Database Indexing: Create search-optimized index of feature representations.
• Query Processing: For input image, extract features and compute similarity against database.
• Retrieval: Return top-k most similar images with their annotations.
• Performance Measurement:
  • Calculate mean Average Precision (mAP) at different similarity thresholds
  • Compute precision@k for various k values (k=5, 10, 20)
  • Measure retrieval accuracy as proportion of cases where correct diagnosis appears in top results

Expected Outcomes: Advanced CBIR systems can achieve mAP of 0.488 at threshold 0.9 and precision of 0.864 across pathologies, significantly outperforming conventional approaches [67].

Visualization of Zero-Shot Classification Workflow

Figure 1: Zero-shot classification workflow showing the parallel processing of whole slide images and text prompts through vision and language encoders, with similarity calculation in the joint embedding space leading to final predictions and performance metrics.

The Scientist's Toolkit

Table 3: Essential Research Reagent Solutions for Zero-Shot Pathology

Tool/Category	Specific Examples	Function/Application	Key Characteristics
Vision-Language Models	CONCH, Quilt-Net, Quilt-LLaVA, TITAN	Zero-shot classification, cross-modal retrieval	Pre-trained on histopathology image-text pairs (1.17M+ pairs)
Feature Extractors	CONCHv1.5, Virchow2, DinoSSLPath	Convert image patches to feature embeddings	Generate 768-dimensional features per patch
Evaluation Frameworks	MR-PHE, Benchmarking suites	Comprehensive model assessment	Multi-resolution analysis, cross-validation
Digital Pathology Tools	QuPath, CellProfiler, ImageJ	WSI preprocessing, annotation, analysis	Open-source platforms with ML support
Prompt Engineering	Structured prompt templates	Optimize zero-shot performance	Vary domain specificity, anatomical precision
Performance Analysis	AUROC, AUPRC, Accuracy, F1	Quantify diagnostic performance	Threshold-independent and dependent metrics

The MR-PHE (Multi-Resolution Prompt-guided Hybrid Embedding) framework exemplifies advanced tooling for zero-shot histopathology, incorporating multi-resolution patch extraction to mimic pathologists' workflow and similarity-based patch weighting to emphasize diagnostically important regions [68].

Performance metrics including AUROC, accuracy, and retrieval scores provide essential quantitative assessment of zero-shot classification capabilities in computational pathology. Vision-language foundation models like CONCH achieve competitive performance with AUROCs of 0.71-0.77 across diverse tasks, demonstrating significant potential for clinical application. Proper experimental implementation using the protocols and tools described enables robust evaluation and comparison of model performance, accelerating the development of clinically applicable AI solutions for pathology.

In the field of computational pathology, the scarcity of high-quality, annotated datasets remains a significant bottleneck for the development of artificial intelligence (AI) systems [69] [70]. While supervised deep learning models have demonstrated remarkable performance in pathological image classification, they heavily rely on labeled data, demanding extensive human annotation efforts [69]. Recently, vision-language foundation models (VLFMs) have emerged as a promising alternative, leveraging pre-trained knowledge for tasks such as zero-shot classification, thereby reducing dependency on annotated datasets [69] [71]. This application note provides a comparative analysis of these two approaches, focusing on their application in pathology research, particularly for zero-shot classification scenarios. We present structured quantitative comparisons, detailed experimental protocols, and essential research tools to guide researchers and drug development professionals in selecting and implementing the appropriate methodology for their specific research context.

Fundamental Architectural Differences

Supervised Deep Learning Models typically rely on convolutional neural networks (CNNs) or vision transformers (ViTs) trained exclusively on annotated pathology datasets. These models require extensive labeled data for training and formulate slide-level modeling as multiple instance learning, where each image tile is treated as an independent instance [70] [30]. They excel in specific diagnostic tasks when trained on large, high-quality datasets but struggle with generalization across diverse clinical scenarios and institutions.

Vision-Language Foundation Models (VLFMs) represent a paradigm shift by leveraging pre-trained knowledge from large-scale image-text pairs. Models like PLIP, BiomedCLIP, and CONCH demonstrate remarkable capabilities in histopathological image interpretation through visual-language alignment [71]. VLFMs consist of dual encoders—a visual encoder for image processing and a text encoder for language understanding—enabling zero-shot inference by comparing image features with textual descriptions of pathological concepts [69] [72]. This architecture allows them to recognize novel pathology categories without task-specific training.

Quantitative Performance Comparison

The table below summarizes the comparative performance of VLFMs and supervised deep learning models across various pathology tasks:

Table 1: Performance Comparison of VLFM and Supervised Learning Approaches

Model Category	Specific Model	Task	Dataset	Performance Metrics	Key Strengths
VLFM (Zero-shot)	VLM-CPL [69]	Patch-level pathological image classification	Five public datasets	Substantially outperformed pure zero-shot VLMs; superior to noisy label learning methods	Annotation-free; utilizes pseudo-labels; handles domain gap
VLFM (Few-shot)	MOC [71]	Few-shot Whole Slide Image Classification	TCGA-NSCLC	10.4% AUC improvement over SOTA few-shot VLFM methods; up to 26.25% gain under 1-shot	Resilient to data scarcity; meta-optimized classifier
Pathology Foundation Model	Prov-GigaPath [30]	Mutation prediction (18 biomarkers)	Providence network (171,189 slides)	3.3% macro-AUROC and 8.9% macro-AUPRC improvement over best competing method	Whole-slide context modeling; scales to billion-tile datasets
Supervised Deep Learning	Conventional MIL [71]	WSI Classification	Large annotated datasets	Outperforms zero-shot VLFMs but requires extensive annotations	High accuracy with sufficient data; well-established methodology
Self-Supervised Learning	DINO-MX [73]	Multiple medical imaging tasks	Diverse medical datasets	Competitive performance with reduced computational costs	Flexible framework; avoids annotation requirements

Table 2: Scenario-Based Model Recommendation

Research Scenario	Recommended Approach	Rationale	Implementation Considerations
Annotation-Rich Environment	Supervised Deep Learning (MIL)	Maximizes performance when extensive labeled data is available	Requires dedicated annotation resources and computational infrastructure
Zero-Shot Classification	VLFM with consensus pseudo-labels (VLM-CPL)	Functions without human annotation; leverages pre-trained knowledge	Requires prompt engineering and noise filtering strategies
Few-Shot Learning	Meta-Optimized Classifier (MOC)	Specifically designed for data-scarce environments	Meta-learning component optimizes classifier configuration dynamically
Whole-Slide Analysis	Prov-GigaPath [30]	Models slide-level context across thousands of tiles	Computationally intensive; requires specialized LongNet architecture
Limited Computational Resources	DINO-MX with PEFT [73]	Reduces training costs while maintaining performance	Supports parameter-efficient fine-tuning and knowledge distillation

Experimental Protocols

Protocol for VLFM Zero-Shot Classification with Consensus Pseudo-Labels

Application Context: This protocol describes the implementation of VLM-CPL for annotation-free pathological image classification, suitable for scenarios where labeled training data is unavailable or scarce [69].

Materials and Equipment:

• Pre-trained vision-language model (e.g., PLIP, BiomedCLIP, CONCH)
• Whole Slide Images (WSIs) in standard formats (SVS, NDPI)
• Computational infrastructure with GPU acceleration
• Patch extraction tools (e.g., OpenSlide)

Procedure:

• Input Preparation and Patch Extraction:
  • Segment WSIs into non-overlapping patches at appropriate magnification (typically 20×)
  • Apply multiple augmented views to each patch (rotation, color jittering, flipping)
  • Exclude background and artifact-heavy regions using tissue segmentation algorithms

• Prompt-Based Pseudo-Label Generation:

  • Design text prompts for each disease category using domain-specific terminology (e.g., "A pathology image of lung adenocarcinoma")
  • Generate initial pseudo-labels through zero-shot inference using the VLM
  • Compute uncertainty estimates by aggregating predictions across multiple augmented views

• Feature-Based Pseudo-Label Generation:

  • Extract patch features using the visual encoder of the VLM
  • Perform sample clustering in the feature space (K-means or hierarchical clustering)
  • Derive cluster-based labels by assigning majority class within clusters

• Prompt-Feature Consensus Filtering:

  • Identify reliable samples where prompt-based and feature-based pseudo-labels show agreement
  • Discard samples with conflicting labels or high uncertainty scores
  • Generate a refined training set with consensus pseudo-labels

• Semi-Supervised Learning:

  • Implement cross-supervision between reliable and uncertain samples
  • Train a final classifier using the refined pseudo-labels
  • Apply open-set prompting to filter irrelevant patches from whole slides

Troubleshooting Tips:

• For poor consensus rates, adjust the augmentation strategies or feature clustering parameters
• If model performance plateaus, refine text prompts with more specific pathological terminology
• To handle class imbalance, implement stratified sampling during consensus filtering

Protocol for Few-Shot WSI Classification with Meta-Optimized Classifier

Application Context: This protocol implements the MOC framework for few-shot whole slide image classification, particularly valuable for rare cancers or specialized diagnostic tasks with minimal annotated examples [71].

Materials and Equipment:

• Pre-trained pathology VLFMs (visual and text encoders)
• Limited annotated WSIs (as few as 1-8 examples per class)
• Meta-learning framework (PyTorch or TensorFlow)

Procedure:

• WSI Preprocessing and Feature Extraction:
  • Extract non-overlapping patches from WSIs
  • Generate patch embeddings using the pre-trained visual encoder: ( u{i,j} = \frac{\mathcal{F}(x{i,j})}{\|\mathcal{F}(x{i,j})\|} )
  • Compute prompt embeddings for each class using the text encoder: ( wc = \frac{\mathcal{G}(tc)}{\|\mathcal{G}(tc})\|} )

• Classifier Bank Configuration:

  • Implement multiple non-parametric candidate classifiers focusing on:
    • Maximal similarity: Top-K cosine similarity between patch and text embeddings
    • Categorical prominence: Difference between highest and second-highest similarity scores
    • Minimal irrelevance: Inverse of similarity to negative prompts
  • Each candidate classifier ( \psih ) produces patch scores ( S^{\Psih}{x{i,j}} = \psih(u{i,j}, W) )

• Patch Filtering via Bank Nomination:

  • Each candidate selects top-q patches with highest scores: ( Bag{Xi}^{\psih} = {x{i,j} | x{i,j} \in \mathop{argmax}\limits^{(q)} S{x{i,j}}^{\psih}} )
  • Create bank-nominated set through union: ( Bag{Xi}^{\Psi} = \bigcup{\psih \in \Psi} Bag{Xi}^{\psi_h} )

• Meta-Learner Optimization:

  • Train a two-layer perceptron meta-learner ( \mathcal{M} ) to predict classifier weights
  • Process each nominated patch embedding to generate weights: ( \Lambda{i,j} = \mathcal{M}(u{i,j}) )
  • Compute final patch predictions as weighted combination of classifier outputs

• Slide-Level Classification:

  • Aggregate patch-level predictions to generate slide-level diagnosis
  • Implement attention mechanism or MIL pooling to weight informative patches

Validation Methods:

• Use leave-one-out cross-validation in few-shot settings
• Compare with conventional MIL baselines and prompt-tuning methods
• Evaluate performance sensitivity to shot number (1-shot to 8-shot)

Visualization of Workflows

VLFM Zero-Shot Classification with Consensus Pseudo-Labels

VLFM Zero-shot Workflow

Meta-Optimized Classifier for Few-Shot Learning

MOC Few-shot Workflow

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Research Tools for VLFM Implementation

Tool/Category	Specific Examples	Function	Implementation Notes
Pre-trained VLFMs	PLIP, BiomedCLIP, CONCH [71]	Provide foundational visual-language understanding for pathology images	Select based on training dataset diversity and specific pathology domain
Whole Slide Processing	OpenSlide, CuCIM	Extract patches from gigapixel WSIs at multiple magnifications	Optimize patch size and overlap for specific tissue types
Data Augmentation	Multi-view augmentations [69]	Generate diverse image variations for robust pseudo-label generation	Include color normalization to address stain variability
Feature Extraction	DINOv2, ViT architectures [73] [30]	Convert image patches to compact embeddings for processing	Use consistent normalization across all patches
Meta-Learning Framework	MOC meta-learner [71]	Dynamically optimize classifier configurations for few-shot learning	Implement as two-layer perceptron with ReLU activation
Parameter-Efficient FT	LoRA, Layer Freezing [73]	Adapt foundation models with minimal computational resources	Particularly valuable for institutions with limited GPU access
Evaluation Benchmark	TCGA-NSCLC, Providence [71] [30]	Standardized assessment of model performance across tasks	Include both patch-level and slide-level classification tasks

The comparative analysis reveals that VLFMs and supervised deep learning models offer complementary strengths for computational pathology. Supervised approaches maintain superiority in annotation-rich environments, while VLFMs provide unprecedented flexibility for zero-shot and few-shot learning scenarios. The emerging paradigm of meta-optimized classifiers combined with consensus pseudo-labeling strategies demonstrates particular promise for clinical deployments where diagnostic training data is severely limited. As pathology foundation models continue to evolve—incorporating larger datasets, whole-slide context modeling, and advanced vision-language pretraining—they hold significant potential to transform cancer diagnostics and biomarker discovery, ultimately advancing personalized medicine through improved accessibility and generalizability.

This application note details a groundbreaking achievement in medical artificial intelligence: the development of zero-shot visual-language foundation models that match or exceed the performance of board-certified radiologists in detecting pathologies from chest X-rays on the CheXpert dataset. By leveraging novel fine-tuning strategies and contrastive learning architectures, these models demonstrate the transformative potential of self-supervised learning in medical imaging, eliminating the dependency on large-scale, expert-annotated datasets while achieving expert-level diagnostic performance.

The application of deep neural networks to medical imaging has long been constrained by the scarcity of high-quality, expert-labeled training data. The process of creating annotated medical image datasets is resource-intensive, requiring specialized expertise and significant time investment. Zero-shot classification with visual-language foundation models represents a paradigm shift by leveraging naturally occurring image-text pairs—specifically chest X-rays and their corresponding radiology reports—to learn pathological representations without explicit disease-specific annotations.

Models like CLIP (Contrastive Language-Image Pre-training) have demonstrated remarkable capabilities in general computer vision tasks by learning from web-scale image-text pairs. The medical adaptation of this approach, exemplified by CheXzero, aligns image embeddings with textual embeddings from radiology reports through contrastive learning, creating a shared representation space where images and pathological findings can be matched without supervised fine-tuning. This case study examines the architectural innovations and methodological refinements that have enabled these models to not only address the data scarcity challenge but actually surpass human expert performance on specific diagnostic tasks.

Quantitative Performance Analysis

Model vs. Radiologist Performance on CheXpert

Table 1: Comparative performance between zero-shot models and board-certified radiologists on CheXpert test dataset (5 competition pathologies)

Pathology	Model MCC	Radiologist MCC	Difference (95% CI)	Model F1	Radiologist F1	Difference (95% CI)
Atelectasis	0.445	0.523	-0.078 (-0.154, 0.000)	0.575	0.620	-0.045 (-0.090, -0.001)
Cardiomegaly	0.588	0.530	0.058 (-0.016, 0.133)	0.684	0.619	0.065 (0.013, 0.115)
Consolidation	0.543	0.525	0.018 (-0.090, 0.123)	0.570	0.620	-0.050 (-0.146, 0.036)
Edema	0.545	0.530	0.015 (-0.070, 0.099)	0.637	0.619	0.018 (-0.053, 0.086)
Pleural Effusion	0.595	0.635	-0.040 (-0.096, 0.013)	0.665	0.699	-0.034 (-0.078, 0.008)
Average	0.523	0.530	-0.005 (-0.043, 0.034)	0.606	0.619	-0.009 (-0.038, 0.018)

Performance metrics collected from [74] demonstrate that the self-supervised model achieved an average Matthews Correlation Coefficient (MCC) of 0.523 compared to radiologists' 0.530, with no statistically significant difference. Notably, the model outperformed radiologists in detecting cardiomegaly with statistical significance in F1 score (0.065 improvement, 95% CI 0.013, 0.115) [74].

Comparative Model Performance

Table 2: Performance comparison across different models on chest X-ray pathology classification (AUC scores)

Model / Pathology	Atelectasis	Cardiomegaly	Consolidation	Edema	Pleural Effusion	Average
CheXzero	0.758	0.825	0.783	0.880	0.836	0.816
MoCoCLIP	0.700	0.860	0.780	0.890	0.870	0.820
Enhanced Fine-tuning	0.790	0.910	0.780	0.910	0.850	0.848
ConVIRT (100% labels)	0.781	0.861	0.731	0.901	0.931	0.841
MoCo-CXR (100% labels)	0.824	0.861	0.748	0.931	0.964	0.866

The enhanced fine-tuning approach with loss relaxation and random sentence sampling achieved an average macro AUROC increase of 4.3% across four chest X-ray datasets and three pre-trained models, demonstrating consistent improvements in zero-shot pathology classification [26] [75]. MoCoCLIP showed particular strength in edema detection (AUC 0.890) and effusion identification (AUC 0.870) [76].

Experimental Protocols & Methodologies

Core Zero-Shot Classification Protocol

Objective: To perform pathology classification without disease-specific annotations during training by aligning image and text embeddings in a shared representation space.

Training Phase:

• Data Preparation: Collect paired chest X-ray images and corresponding radiology reports. The MIMIC-CXR dataset containing 377,110 image-text pairs is typically used [74].
• Image Encoding: Process images through a visual encoder (typically ViT-B/32). Images are resized to 224×224 pixels and normalized using ImageNet statistics.
• Text Encoding: Process radiology reports through a text encoder (Transformer-based). For medical reports, employ random sentence sampling rather than full reports to enhance learning of individual findings.
• Contrastive Learning: Optimize image-text alignment using InfoNCE loss or modified variants that account for false-negative pairs inherent in medical data.

Inference Phase:

• Prompt Engineering: Create positive and negative prompts for each target pathology using templates: "{label}" and "no {label}" (e.g., "Atelectasis" and "No atelectasis").
• Similarity Calculation: For a target image Iₜᵣg and pathology c, compute:
  • s₊,꜀ = Eᵢₘg(Iₜᵣg) · Eₜₓₜ(T₊,꜀)
  • s₋,꜀ = Eᵢₘg(Iₜᵣg) · Eₜₓₜ(T₋,꜀)
• Probability Calculation: Apply softmax to obtain classification probability: prob꜀ = softmax(s₊,꜀, s₋,꜀) [26] [75] [74].

Advanced Fine-tuning with Loss Relaxation

Problem Addressment: Standard contrastive learning treats all non-matching image-report pairs as negative samples, ignoring the multi-labeled nature of medical data where pairs may share some pathologies (false negatives).

Methodology:

• Random Sentence Sampling: For each training iteration, randomly select n sentences from medical reports containing m sentences, creating ₘCₙ positive samples and enabling the model to learn rich semantics of individual findings.
• Loss Relaxation: Modify similarity function to clip upper bound of positive pair attraction:

sim(uᵢ,vⱼ) = 1/(1+exp(-α(uᵢ·vⱼ-t))) if i=j and uᵢ·vⱼ ≥ t uᵢ·vⱼ/2t else if i=j and t > uᵢ·vⱼ ≥ 0 uᵢ·vⱼ otherwise

where t is a threshold (0[26] [75].<="" attraction="" between="" determining="" level="" p="" pairs="" positive="">

MoCoCLIP Integration Protocol

Architecture Enhancement: Integrate Momentum Contrast (MoCo) into CLIP framework to improve representation learning.

Implementation:

• Dual Encoder Setup: Maintain two copies of image encoder - main encoder (θq) and momentum encoder (θk).
• Momentum Update: Update θk using: θk ← mθk + (1-m)θq, where m is momentum coefficient (typically 0.999).
• Queue Implementation: Initialize queue to store encoded features (keys) with size 65,536 and feature dimension matching encoder output.
• Multi-Objective Optimization: Combine losses: ℒ = ℒcon + λℒMoCo, where ℒcon is image-text contrastive loss and ℒMoCo is MoCo contrastive loss [76].

Visual Framework for Zero-Shot Medical Classification

Zero-Shot Medical Classification Framework

Multi-Scale Feature Learning Architecture

Multi-Scale Feature Learning

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential research reagents and resources for zero-shot medical imaging research

Resource	Type	Function	Example Specifications
MIMIC-CXR	Dataset	Provides 377,110 paired chest X-ray images and radiology reports for training	Frontal-view images, de-identified reports, multi-institutional [74]
CheXpert	Dataset	Benchmark dataset for evaluation containing 500 frontal CXRs with 14 pathology labels	Competition pathologies: Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion [76]
CLIP Architecture	Model Framework	Base vision-language model for medical adaptation	ViT-B/32 image encoder, transformer text encoder, contrastive pre-training [26]
NIH ChestXray14	Dataset	Large-scale dataset with 112,120 X-ray images from 30,805 patients	14 disease categories + "No Finding" class, used for training and evaluation [76]
InfoNCE Loss	Algorithm	Contrastive loss function for aligning image and text representations	Modifiable with relaxation mechanisms for medical false-negative pairs [26]
Momentum Contrast (MoCo)	Algorithm	Self-supervised learning method for robust visual representations	Queue size: 65,536, momentum coefficient: 0.999, compatible with CLIP framework [76]
Random Sentence Sampling	Text Processing	Medical report augmentation preserving clinical information	Selects n sentences from m-sentence reports, creating ₘCₙ learning opportunities [75]

The demonstrated capability of zero-shot visual-language models to match or exceed board-certified radiologists in specific chest X-ray interpretation tasks on the CheXpert dataset marks a significant milestone in medical AI. The key innovations—including advanced fine-tuning strategies with loss relaxation, multi-scale feature learning, and integration of momentum contrast—have collectively addressed fundamental challenges in medical image analysis, particularly the multi-labeled nature of image-report pairs and the prevalence of false-negative samples in contrastive learning.

These advancements highlight the viability of self-supervised learning paradigms in reducing dependency on expensively annotated medical datasets while achieving expert-level performance. The consistent improvements across multiple datasets and model architectures suggest robust methodologies that can be extended to other medical imaging domains beyond chest radiography. As these foundation models continue to evolve, they hold significant promise for enhancing diagnostic accuracy, streamlining clinical workflows, and expanding access to expert-level medical image interpretation across diverse healthcare settings. Future research directions include extending these approaches to multi-modal medical data integration, improving model interpretability for clinical trust, and adapting the frameworks for real-time diagnostic applications.

The integration of complex 3D imaging tasks with visual-language foundation models represents a transformative frontier in computational pathology and biomedical research. While models like TITAN (Transformer-based pathology Image and Text Alignment Network) demonstrate remarkable capabilities in zero-shot classification for whole-slide images, significant performance gaps emerge when these approaches are extended to complex 3D imaging domains [10]. This application note examines the current limitations, provides standardized experimental protocols for evaluating these gaps, and outlines essential reagent solutions for researchers working at this intersection. The transition from 2D histopathology to volumetric analysis introduces unique challenges in data handling, computational resource allocation, and multimodal alignment that must be addressed to realize the full potential of zero-shot learning in three-dimensional biomedical contexts.

Performance Gap Analysis in 3D Imaging

Technical Limitations in 3D Imaging Modalities

Current 3D imaging modalities present inherent constraints that directly impact the performance of visual-language foundation models. The table below summarizes key technical limitations across major imaging techniques:

Table 1: Technical Limitations of 3D Imaging Modalities

Imaging Modality	Primary Limitations	Impact on Foundation Models
Structured Light	Limited depth penetration; sensitive to surface reflectivity [77]	Restricted to surface topology; fails with complex internal structures
Time-of-Flight (ToF)	Multipath interference; ambient light noise; limited resolution [77]	Noisy depth measurements reduce feature extraction accuracy
Stereo Vision	Requires texture-rich surfaces; computationally intensive [77]	Fails with textureless tissues; high computational load limits scalability
CT/MRI	Radiation exposure (CT); motion artifacts; high cost [78]	Limited training data availability; artifacts confuse model predictions
Ultrasound	Speckle noise; shadowing artifacts; operator dependency [79]	Inconsistent image quality hampers robust feature learning

These technical limitations create fundamental barriers for foundation models expecting clean, standardized 2D image inputs. The translation of these constraints to model performance gaps is particularly evident in zero-shot settings where the model cannot rely on task-specific fine-tuning to compensate for modality-specific artifacts.

Performance Metrics Across Imaging Tasks

Rigorous evaluation of foundation models on 3D imaging tasks reveals consistent performance degradation compared to their 2D counterparts. The HA-U3Net study, while demonstrating advanced capabilities in 3D medical image segmentation, highlighted critical limitations in cross-modality generalization [79]. When evaluated on the BraTS dataset for brain tumor segmentation, model performance dropped by approximately 15-20% compared to within-modality performance, exposing the sensitivity of these systems to variations in resolution, contrast, and noise profiles across imaging sources.

The TITAN model, despite its strong performance on whole-slide images, faces architectural constraints when processing volumetric data [10]. Its pretraining on 335,645 whole-slide images focused primarily on 2D region-of-interest encodings, lacking explicit mechanisms for capturing inter-slice dependencies essential for true 3D understanding. This limitation manifests particularly in tasks requiring spatial reasoning across planes, such as tracking tumor invasion patterns through tissue volumes or understanding vascular networks in organ systems.

Experimental Protocols for Evaluating 3D Zero-Shot Performance

Protocol 1: Cross-Modality Generalization Assessment

Objective: Quantify zero-shot classification performance across diverse 3D imaging modalities.

Materials:

• Pre-trained visual-language foundation model (e.g., TITAN, CONCH)
• Multi-modal 3D imaging dataset (e.g., BraTS, ABUS, TotalSegmentator)
• Computational resources with minimum 16GB GPU memory

Methodology:

• Data Preparation: Curate paired image-text datasets from at least three imaging modalities (MRI, CT, ultrasound)
• Feature Extraction: Process volumetric data using sliding window approach (64×64×64 voxels per window)
• Text Prompt Engineering: Develop standardized prompt templates for each diagnostic class
• Similarity Computation: Calculate cosine similarity between image features and text embeddings
• Evaluation: Assess zero-shot accuracy using multi-class ROC analysis

Quality Control:

• Implement intensity normalization within and across modalities
• Validate text prompts with domain experts for clinical relevance
• Conduct ablation studies on window size impact on performance

Protocol 2: Spatial Reasoning Capability Evaluation

Objective: Measure model performance on tasks requiring 3D spatial understanding.

Materials:

• Volumetric annotations with spatial relationships
• Distance metrics for spatial accuracy quantification
• Synthetic spatial reasoning dataset

Methodology:

• Task Design: Create spatial relationship prompts (e.g., "tumor adjacent to ventricle," "mass superior to kidney")
• Volume Processing: Extract features from orthogonal planes independently
• Feature Fusion: Implement cross-attention mechanisms between planar features
• Spatial Alignment: Compute spatial consistency metrics between predictions and ground truth
• Benchmarking: Compare against human expert performance on identical tasks

Quality Control:

• Establish inter-rater reliability for spatial annotations
• Control for volume size and complexity across test cases
• Validate with progressively complex spatial relationships

Visualization of 3D Imaging Challenges in Computational Pathology

Diagram 1: 3D Imaging Performance Gap Architecture

This architecture illustrates how inherent limitations in 3D imaging modalities interact with foundation model constraints to produce performance gaps in zero-shot classification tasks. The critical pathway shows how modality-specific artifacts propagate through the system, ultimately reducing classification accuracy and reliability.

Research Reagent Solutions for 3D Pathology

Table 2: Essential Research Toolkit for 3D Visual-Language Pathology

Research Reagent	Function	Application Notes
CONCH Vision-Language Model [12]	Provides foundational embeddings for histopathology images and text	Use as feature extractor; compatible with non-H&E stains
Merge DICOM Toolkit [80]	Standardizes whole-slide imaging data formats	Enables interoperability across scanner vendors
HA-U3Net Architecture [79]	3D medical image segmentation across modalities	Nested V-Net structure with hybrid attention
PathPT Framework [24]	Few-shot prompt tuning for rare cancer subtyping	Enables spatially-aware visual aggregation
TITAN Whole-Slide Model [10]	Multimodal slide representation learning	Pretrained on 335K WSIs; enables zero-shot tasks

The performance gaps in applying visual-language foundation models to complex 3D imaging tasks stem from both technical limitations in imaging modalities and architectural constraints in current models. The experimental protocols outlined provide standardized methodologies for quantifying these limitations, while the research reagent toolkit offers practical solutions for researchers addressing these challenges. Future work must focus on developing volumetric-aware architectures, expanding multi-modal pretraining, and creating comprehensive benchmarks specifically designed for 3D zero-shot classification in pathology. As foundation models continue to evolve, addressing these performance gaps will be essential for unlocking the full potential of AI-assisted diagnosis in volumetric medical imaging.

The Role of Large-Scale Benchmarking (e.g., 60+ Models) in Guiding Model Selection

Large-scale benchmarking represents a critical methodology in computational pathology, enabling the systematic evaluation of numerous foundation models (e.g., 60+ models) across standardized tasks and datasets. For pathology research, particularly in zero-shot classification with visual-language foundation models (VLMs), these comprehensive evaluations provide empirical evidence to guide model selection beyond theoretical capabilities. Recent studies have demonstrated the necessity of such benchmarks, as they reveal significant performance variations across models when applied to real-world clinical tasks, including biomarker prediction, morphological analysis, and prognostic outcome prediction [66]. The transition from patch-level to whole-slide image (WSI) analysis has further complicated model selection, necessitating benchmarks that evaluate performance across multiple dimensions, including data efficiency, multimodal capabilities, and robustness in low-prevalence scenarios.

Benchmarking studies help mitigate risks associated with selective reporting and data leakage by employing external cohorts that were never part of foundation model training. This approach ensures that performance metrics reflect true generalizability rather than dataset-specific optimization. For drug development professionals and researchers, these benchmarks provide critical decision-support data when selecting AI models for pathology applications, ultimately accelerating the translation of computational tools into clinical and research workflows [66]. The integration of VLMs has introduced additional complexity, as performance depends not only on architectural considerations but also on prompt design and cross-modal alignment strategies, further underscoring the value of systematic comparative studies.

Performance Benchmarking of Pathology Foundation Models

Quantitative Performance Across Clinical Tasks

Recent large-scale benchmarking efforts have evaluated 19 foundation models and 14 ensembles across 31 clinically relevant tasks using datasets comprising 6,818 patients and 9,528 slides from lung, colorectal, gastric, and breast cancers [66]. The models were assessed on weakly supervised tasks related to morphology (n=5), biomarkers (n=19), and prognostication (n=7) using the area under the receiver operating characteristic curve (AUROC) as the primary metric. As illustrated in Table 1, vision-language models, particularly CONCH, demonstrated superior overall performance, with Virchow2 as a close second among vision-only models [66].

Table 1: Performance Summary of Top-Performing Foundation Models Across Task Categories

Model	Model Type	Morphology Tasks (Mean AUROC)	Biomarker Tasks (Mean AUROC)	Prognosis Tasks (Mean AUROC)	Overall Mean AUROC
CONCH	Vision-Language	0.77	0.73	0.63	0.71
Virchow2	Vision-Only	0.76	0.73	0.61	0.71
Prov-GigaPath	Vision-Only	0.69	0.72	0.66	0.69
DinoSSLPath	Vision-Only	0.76	0.68	0.62	0.69
UNI	Vision-Only	0.72	0.68	0.63	0.68

Statistical comparisons across 29 binary classification tasks revealed that CONCH significantly outperformed other models in numerous tasks: PLIP (16 tasks), Phikon and BiomedCLIP (13 tasks each), Kaiko (11 tasks), and 7 tasks each for Hibou-L, H-optimus-0, CTransPath, Virchow, Panakeia, UNI, and DinoSSLPath [66]. Conversely, Virchow2 outperformed CONCH in 6 tasks, demonstrating the complementary strengths of these top-performing models. These findings underscore the importance of task-specific model selection rather than relying on a one-size-fits-all approach.

Performance in Low-Data and Low-Prevalence Scenarios

For drug development applications involving rare diseases or biomarkers, model performance in low-data regimes is particularly relevant. Benchmarking studies have evaluated foundation models using progressively smaller training cohorts (300, 150, and 75 patients) while maintaining similar ratios of positive samples [66]. In the largest sampled cohort (n=300), Virchow2 demonstrated superior performance in 8 tasks, followed closely by PRISM with 7 tasks. With the medium-sized cohort (n=150), PRISM led in 9 tasks, while Virchow2 led in 6 tasks. The smallest cohort (n=75) showed more balanced results, with CONCH leading in 5 tasks, while PRISM and Virchow2 each led in 4 tasks [66].

Notably, performance metrics remained relatively stable between n=75 and n=150 cohorts, suggesting that certain foundation models can maintain effectiveness even with limited training data. This characteristic is particularly valuable for pharmaceutical research involving rare cancers or molecular subtypes with limited available samples. For low-prevalence biomarkers (<15% positive cases), such as BRAF mutations (10%) and CpG island methylator phenotype (CIMP) status (13%), the choice of foundation model significantly impacted predictive performance, with CONCH and Virchow2 consistently outperforming alternatives [66].

Experimental Protocols for Benchmarking Studies

Whole-Slide Image Processing and Feature Extraction

The benchmarking methodology for pathology foundation models follows a standardized protocol for whole-slide image analysis. The process begins with WSI tessellation into small, non-overlapping patches, typically 512×512 pixels at 20× magnification [66] [10]. These patches serve as input to foundation models for feature extraction, generating embeddings that capture morphological patterns in histology tissue. For slide-level foundation models like TITAN, patch features are spatially arranged in a two-dimensional grid replicating their original positions within the tissue, preserving spatial context [10].

Table 2: Key Research Reagents and Computational Tools for Pathology Benchmarking

Resource Type	Specific Examples	Function in Benchmarking
Foundation Models	CONCH, Virchow2, TITAN, Quilt-Net, Quilt-LLaVA	Generate feature representations from histology images for downstream tasks
Dataset Cohorts	TCGA, Mass-340K, CHUM Digestive Pathology (3,507 WSIs)	Provide diverse, multi-organ histology data for training and evaluation
Visualization Tools	Cellxgene, Spotfire, Tableau, Custom Shiny Apps	Enable exploration and interpretation of high-dimensional feature spaces
Analysis Frameworks	M.E.D.V.I.S., ABMIL, Transformer-based MIL	Standardize evaluation metrics and experimental workflows

For vision-language models, the protocol includes cross-modal alignment strategies. The TITAN model, for instance, employs a three-stage pretraining approach: (1) vision-only unimodal pretraining on region crops, (2) cross-modal alignment of generated morphological descriptions at the region-of-interest level, and (3) cross-modal alignment at the WSI level with clinical reports [10]. This structured approach ensures that the resulting slide-level representations capture both visual semantics and their corresponding clinical context.

Evaluation Methodology for Zero-Shot Classification

The evaluation of zero-shot capabilities in VLMs requires specialized protocols that assess classification performance without task-specific fine-tuning. Benchmarking studies typically employ multiple prompt designs varying in domain specificity, anatomical precision, instructional framing, and output constraints [9]. For example, recent research has systematically evaluated VLMs using an in-house digestive pathology dataset comprising 3,507 WSIs across seven tissue types, assessing performance on cancer invasiveness and dysplasia status classification [9].

The evaluation protocol for zero-shot classification involves feeding image-text pairs to the model and measuring the similarity between image embeddings and text embeddings of class descriptions. For CLIP-inspired models like Quilt-Net, this involves processing input images through a visual encoder (e.g., ViT-B/32) and class labels through a text encoder (e.g., GPT-2), then computing cosine similarity between the resulting embeddings [9]. The class with the highest similarity score is selected as the prediction. Performance is measured using standard classification metrics, including accuracy, AUROC, and F1 score, with emphasis on performance across different tissue types and disease states.

Prompt Engineering Framework for VLMs

Benchmarking studies have revealed that VLM performance in computational pathology is highly sensitive to prompt design, necessitating structured frameworks for prompt optimization. Research has identified four critical dimensions for prompt variation in pathology VLMs: (1) domain specificity, (2) anatomical precision, (3) instructional framing, and (4) output constraints [9]. Studies demonstrate that precise anatomical references in prompts significantly enhance model performance, with the CONCH model achieving highest accuracy when provided with detailed anatomical context [9].

The prompt engineering protocol involves systematic ablation studies comparing different phrasing strategies for the same diagnostic task. For example, a prompt for dysplasia classification might vary from a generic "Is dysplasia present?" to a more specific "Based on the histological features of this colon wall tissue, is there evidence of epithelial dysplasia?" [9]. Benchmarking results indicate that model complexity alone does not guarantee superior performance, with effective domain alignment and domain-specific training being critical factors. This finding has important implications for drug development pipelines, where consistent and standardized prompt designs ensure reproducible model performance across different studies and sites.

Implementation Guidelines for Model Selection

Decision Framework for Model Selection

Based on comprehensive benchmarking data, researchers can implement a structured decision framework for model selection in pathology applications. For zero-shot classification tasks, vision-language models like CONCH generally outperform vision-only alternatives, particularly when precise anatomical references can be incorporated into well-designed prompts [9]. However, for applications with limited training data or low-prevalence targets, Virchow2 may be preferable, as it has demonstrated superior performance in several low-data scenarios [66].

The decision framework should also consider computational requirements and inference speed, particularly for high-throughput drug screening applications. While models like Quilt-LLaVA with approximately 7B parameters enable sophisticated reasoning, they come with significant computational costs compared to more efficient alternatives like CONCH (200M parameters) or Quilt-Net (150M parameters) [9]. For whole-slide analysis, models like TITAN offer advantages in slide-level representation learning, outperforming both region-of-interest and other slide foundation models across multiple machine learning settings, including linear probing, few-shot, and zero-shot classification [10].

Ensemble Strategies for Enhanced Performance

Benchmarking results have demonstrated that foundation models trained on distinct cohorts learn complementary features to predict the same labels, creating opportunities for ensemble approaches that outperform individual models. Research shows that an ensemble combining CONCH and Virchow2 predictions outperformed individual models in 55% of tasks, leveraging their complementary strengths in classification scenarios [66]. This suggests that for critical applications in drug development, where predictive accuracy is paramount, ensemble methods may be worth the additional computational complexity.

The ensemble protocol involves training multiple foundation models independently and combining their predictions through weighted averaging or meta-learning approaches. The benchmarking study evaluated 14 ensembles derived from the 19 foundation models, with the CONCH-Virchow2 ensemble demonstrating particularly strong performance across multiple task categories [66]. This ensemble strategy effectively mitigates the risk of model-specific biases and enhances robustness across diverse tissue types and disease states.

Large-scale benchmarking of 60+ models provides an evidence-based foundation for model selection in computational pathology, particularly for zero-shot classification with visual-language foundation models. The comprehensive evaluation of models across diverse tasks and datasets reveals that while CONCH and Virchow2 generally lead in performance, optimal model selection depends on specific research contexts, including data availability, task requirements, and computational resources. The experimental protocols and implementation guidelines presented herein offer researchers and drug development professionals a structured approach to leveraging these benchmarking insights in their pathology research workflows.

The field continues to evolve rapidly, with emerging foundation models like TITAN demonstrating the potential of whole-slide representation learning and multimodal capabilities. As benchmarking methodologies mature, incorporating more diverse datasets and real-world clinical scenarios, they will play an increasingly vital role in translating computational pathology advances into tangible improvements in drug development and patient care.

Conclusion

Visual-language foundation models represent a paradigm shift in computational pathology, offering a powerful solution to the field's pressing data annotation challenges. The synthesis of research confirms that zero-shot models like CONCH and TITAN, when enhanced with strategic fine-tuning and prompt engineering, can achieve state-of-the-art performance across diverse tasks—sometimes rivaling or surpassing human experts. Key takeaways include the critical importance of addressing the multi-label nature of medical data, the effectiveness of synthetic data for training enrichment, and the need for standardized, comprehensive benchmarking. Future directions should focus on improving model interpretability for clinical trust, expanding capabilities to 3D medical images, enabling seamless integration with multimodal patient data for drug development, and conducting large-scale real-world validation to fully translate this promising technology into routine clinical and research practice.

References

• 1. A visual-language foundation model for computational ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11384335/]
• 2. Preparing Data for Artificial Intelligence in Pathology with ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10572440/]
• 3. A visual-language foundation model for computational ... [https://www.nature.com/articles/s41591-024-02856-4]
• 4. Investigating Zero-Shot Diagnostic Pathology in Vision ... [https://arxiv.org/abs/2505.00134]
• 5. 【阅读笔记】MI-Zero：视觉语言模型+ 多实例学习，零样本迁移 ... [https://blog.csdn.net/qq_43538018/article/details/143957558]
• 6. [论文评述] Investigating Zero-Shot Diagnostic Pathology in ... [https://www.themoonlight.io/zh/review/investigating-zero-shot-diagnostic-pathology-in-vision-language-models-with-efficient-prompt-design]
• 7. Foundation models for radiology [https://dirjournal.org/articles/foundation-models-for-radiology-fundamentals-applications-opportunities-challenges-risks-and-prospects/dir.2025.253445]
• 8. Fusing Multiple Foundation Models for Pathology Image ... [https://www.sciencedirect.com/science/article/abs/pii/S1566253525009029]
• 9. Investigating Zero-Shot Diagnostic Pathology in Vision- ... [https://arxiv.org/html/2505.00134v1]
• 10. A multimodal whole-slide foundation model for pathology [https://www.nature.com/articles/s41591-025-03982-3]
• 11. Enhancing Zero-Shot Brain Tumor Subtype Classification ... [https://arxiv.org/html/2508.01602v2]
• 12. mahmoodlab/CONCH: Vision-Language Pathology ... [https://github.com/mahmoodlab/CONCH]
• 13. Comparing Computational Pathology Foundation Models ... [https://arxiv.org/html/2509.15482v1]
• 14. A multimodal whole-slide foundation model for pathology [https://pubmed.ncbi.nlm.nih.gov/41193692/]
• 15. MahmoodLab/CONCH [https://huggingface.co/MahmoodLab/CONCH]
• 16. Benchmarking pathology foundation models for non- ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11957174/]
• 17. Enhancing Zero-Shot Brain Tumor Subtype Classification ... [https://www.sciencedirect.com/science/article/abs/pii/S0957417425037765]
• 18. Navigating Gigapixel Pathology Images with Large ... [https://arxiv.org/html/2511.19652v1]
• 19. Rethinking Foundation Models in Pathology: A New Lens [https://beyondtheslide.substack.com/p/rethinking-foundation-models-pathology?utm_source=substack&utm_medium=email&utm_content=share&action=share]
• 20. Pathology Foundation Models - PMC - PubMed Central [https://pmc.ncbi.nlm.nih.gov/articles/PMC11799676/]
• 21. A visual-language foundation model for computational ... [https://www.broadinstitute.org/publications/broad1348226]
• 22. Multimodal Data Fusion for Whole-Slide Histopathology ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12602746/]
• 23. Multiscale transformers and multi-attention mechanism ... [https://www.nature.com/articles/s41598-025-90397-2]
• 24. Boosting Pathology Foundation Models via Few-shot ... [https://www.arxiv.org/abs/2508.15904]
• 25. A visual-language foundation model for computational ... [https://pubmed.ncbi.nlm.nih.gov/38504017/]
• 26. Significantly improving zero-shot X-ray pathology ... [https://www.nature.com/articles/s41598-024-73695-z]
• 27. \dataset: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature [https://arxiv.org/html/2501.07171v1]
• 28. ImageCLEFmed Caption | ImageCLEF / LifeCLEF - Multimedia Retrieval in CLEF [https://www.imageclef.org/2025/medical/caption]
• 29. Enhancing image caption generation through context- ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11399578/]
• 30. A whole-slide foundation model for digital pathology from ... [https://www.nature.com/articles/s41586-024-07441-w]
• 31. Zero-shot segmentation of skin tumors in whole-slide ... [https://arxiv.org/html/2511.18978v1]
• 32. Cutting-Edge Pathology & WSI Research: November 2025 [https://jobs-staging.ilipra.org/blog/cutting-edge-pathology-and-wsi]
• 33. BC-predict: mining of signal biomarkers and production ... [https://www.frontiersin.org/journals/bioinformatics/articles/10.3389/fbinf.2025.1644695/full]
• 34. Novel cancer subtyping method guided by tumor-normal ... [https://www.nature.com/articles/s41598-025-07813-w]
• 35. Pathology Image Segmentation: Annotating Slides for AI [https://datavlab.ai/post/pathology-image-segmentation-for-ai-annotating-slides-and-tissue-samples]
• 36. An intelligent healthcare system for rare disease diagnosis ... [https://biodatamining.biomedcentral.com/articles/10.1186/s13040-025-00487-0]
• 37. AI模型TxGNN在罕见病治疗中的应用与前景：新模型与合作进展 [https://www.forwardpathway.com/128228]
• 38. A multimodal visual–language foundation model for ... [https://www.nature.com/articles/s41746-025-01772-2]
• 39. Multi-Label Classification of Thoracic Diseases using ... [https://arxiv.org/html/2202.03583v4]
• 40. A multi-label learning model for predicting drug-induced ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9451100/]
• 41. PathVLM-Eval: Evaluation of open vision language models ... [https://www.sciencedirect.com/science/article/pii/S2153353925000409]
• 42. [PDF] Significantly improving zero-shot X-ray pathology ... [https://www.semanticscholar.org/paper/Significantly-improving-zero-shot-X-ray-pathology-Jang-Kyung/3a8c0e08821bb9265ad5d9032449288282b89a1a]
• 43. Self-supervised deep metric learning for prototypical zero- ... [https://www.sciencedirect.com/science/article/pii/S0010482525009850]
• 44. Zero-Shot Classification [https://www.emergentmind.com/topics/zero-shot-classification]
• 45. A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis [https://arxiv.org/html/2412.13126v1]
• 46. HistoChat: Instruction-tuning multimodal vision language ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12365532/]
• 47. Synthetic data trained open-source language models are ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12287339/]
• 48. A visual–omics foundation model to bridge histopathology ... [https://www.nature.com/articles/s41592-025-02707-1]
• 49. Whole Slide Imaging (WSI) in Pathology: Emerging Trends ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12308189/]
• 50. A Framework for Gigapixel Whole-Slide-Image Lossless ... [https://arxiv.org/html/2503.18074v1]
• 51. An Efficient Algorithm for Whole Slide Image Size Reduction [https://www.sciencedirect.com/science/article/pii/S294976122300055X]
• 52. Generating dermatopathology reports from gigapixel whole ... [https://www.nature.com/articles/s41467-025-60014-x]
• 53. Zero-shot medical image classification via large multimodal ... [https://www.sciencedirect.com/science/article/abs/pii/S1046202325002038]
• 54. Architectural Co-Design for Zero-Shot Anomaly Detection [https://arxiv.org/html/2508.07819v1]
• 55. A multimodal benchmark and improved architecture for ... [https://www.amazon.science/publications/a-multimodal-benchmark-and-improved-architecture-for-zero-shot-learning]
• 56. How to Visualize Deep Learning Models [https://neptune.ai/blog/deep-learning-visualization]
• 57. Interpretation and Visualization Techniques for Deep Learning ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8236074/]
• 58. Evaluation of open vision language models in histopathology [https://pubmed.ncbi.nlm.nih.gov/40688392/]
• 59. PathVLM-Eval: Evaluation of open vision language models ... [https://elmi.hbku.edu.qa/en/publications/pathvlm-eval-evaluation-of-open-vision-language-models-in-histopa]
• 60. Evaluation of open vision language models in histopathology [https://pmc.ncbi.nlm.nih.gov/articles/PMC12276438/]
• 61. PathMMU: A Massive Multimodal Expert-Level Benchmark ... [https://arxiv.org/abs/2401.16355]
• 62. ECCV 2024 PathMMU : A Massive Multimodal Expert-Level Benchmark... [https://eccv.ecva.net/virtual/2024/oral/878]
• 63. Benchmarking of Closed Vision-Language Models for Ki ... [https://www.sciencedirect.com/science/article/pii/S0957417425038114]
• 64. PathVLM-R1: A Reinforcement Learning-Driven ... [https://arxiv.org/html/2504.09258v1]
• 65. WACV 2025 Open Access Repository [https://openaccess.thecvf.com/content/WACV2025/html/Bahadir_LLM-Generated_Rewrite_and_Context_Modulation_for_Enhanced_Vision_Language_Models_WACV_2025_paper.html]
• 66. Benchmarking foundation models as feature extractors for ... [https://www.nature.com/articles/s41551-025-01516-3]
• 67. Improving diagnosis accuracy with an intelligent image ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10547797/]
• 68. Leveraging Vision-Language Embeddings for Zero-Shot ... [https://pubmed.ncbi.nlm.nih.gov/40601464/]
• 69. VLM-CPL: Consensus Pseudo-Labels from Vision-Language Models... [https://pubmed.ncbi.nlm.nih.gov/40758498/]
• 70. Deep learning approaches for pathological image ... [https://pubmed.ncbi.nlm.nih.gov/41274680/?utm_source=FeedFetcher&utm_medium=rss&utm_campaign=None&utm_content=0j_EYqAqdyLBptvlCsBvl6eCciDZQXaTQav0gvcoh-y&fc=None&ff=20251123105047&v=2.18.0.post22+67771e2]
• 71. MOC: Meta-Optimized Classifier for Few- Shot Whole Slide Image... [https://arxiv.org/html/2508.09967v1]
• 72. Vision-language foundation model for 3D medical imaging [https://www.nature.com/articles/s44387-025-00015-9]
• 73. DINO-MX: A Modular & Flexible Framework for Self ... [https://arxiv.org/html/2511.01610v1]
• 74. Expert-level detection of pathologies from unannotated ... [https://www.nature.com/articles/s41551-022-00936-9]
• 75. Significantly improving zero-shot X-ray pathology ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11455863/]
• 76. ENHANCING ZERO-SHOT LEARNING IN MEDICAL ... [https://arxiv.org/html/2503.13134v1]
• 77. Understand 3D imaging techniques and hardware [https://www.activesilicon.com/news-media/news/tech-focus-3d-imaging/]
• 78. Artificial Intelligence in Clinical Medicine: Challenges ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12468291/]
• 79. Net: A modality-agnostic framework for 3D medical image ... [https://www.sciencedirect.com/science/article/abs/pii/S0950705125011682]
• 80. Enabling the future of digital pathology with DICOM at ... [https://www.merative.com/blog/enabling-the-future-of-digital-pathology-with-dicom-at-pathology-visions-2025]