Linear Probing Reveals Hidden Transferability: A Multi-Domain CLIP Encoder Benchmark for White Blood Cell Classification

Abstract

White blood cell (WBC) classification from peripheral blood smear images is essential for diagnosing infections, immunedisorders, and hematological malignancies. Vision-language models based on Contrastive Language-Image Pretraining (CLIP) offer apromising approach by enabling inference with minimal or no labelled data, yet no prior study has jointly examined models acrossmultiple pretraining domains, prompt strategies, and supervision levels for white blood cell classification. In this work, we presentCLIPBlood, a comprehensive benchmark of six CLIP-based encoders for WBC classification on the WBCAtt dataset comprising 10,298images across five leukocyte classes. The evaluated models span three pretraining domains: general (OpenCLIP ViT-B/32, ViT-L/14),biomedical (BiomedCLIP, PubMedCLIP), and pathology (PLIP, QuiltNet). We assess all models under two complementary settings:zero-shot classification with five prompt templates of increasing domain specificity, and few-shot linear probing with k ∈ {1, 5, 10, 25,50, 100} labelled samples per class. Our results show that zero-shot performance remains poor across all 30 model-prompt combinations,with no configuration exceeding 0.26 macro-F1, indicating that current CLIP encoders lack the text-image alignment necessary for fine-grained blood cell classification. In contrast, few-shot linear probing dramatically improves performance, confirming that frozen CLIPencoders capture discriminative visual structure that can be leveraged with minimal supervision. Among all models, BiomedCLIPconsistently achieves the highest scores across all values of k, reaching 87.5% macro-F1 with only 100 samples per class, despite rankingamong the lowest in zero-shot performance. This paradox highlights that visual representation quality, rather than text-image alignment,is the primary determinant of downstream performance. The t-SNE visualizations further confirm that BiomedCLIP produces the mostseparable embedding clusters. Overall, our findings demonstrate that pretraining domain alignment is a stronger predictor of performancethan model scale, and that BiomedCLIP is the most suitable foundation encoder for WBC analysis under low-resource conditions.