Design and architecture of the SAMJ plugin
It is important to note that SAMJ does not introduce a novel model for bioimage segmentation. Instead, it provides a software tool that facilitates the use of an existing foundational model (SAM), which we believe can be highly valuable for bioimage annotation, especially within Fiji where its environment, usability, and macro language make it an ideal setting for high-throughput annotation and automation.
SAMJ leverages the core architecture of SAM, which is built upon three key components: an image encoder, a prompt encoder, and a mask decoder. The image encoder is based on a Visual Transformer (ViT)11 that encodes the input image, offering high representational power at the expense of significant computational resources. The prompt encoder is another transformer to encode the prompt given by the user to point or hint to the location of the object of interest. Finally, the encoded image and prompt are used by the mask decoder, which combines elements of both a transformer and a Convolutional Neural Network, to generate the mask of interest.
Accessibility and usability
The SAMJ plugin is installed as any standard Fiji extension, eliminating the technical IT barriers often associated with Python packages and making advanced image segmentation accessible to a wider audience. Unlike most Python-based libraries that require command-line interfaces for environment configuration and dependency management, SAMJ simplifies the process with a seamless, one-click installation, enabling effortless deployment.
In addition, SAMJ provides access to five SAM variants: SAM-212 (Tiny, Small, and Large), EfficientSAM13, and EfficientViTSAM-L214. All variants perform the same segmentation task and are trained on comparable generalist datasets, but they differ substantially in computational requirements and performance characteristics (see Supplementary Information section 1). This diversity enables users to select a model that fits both their hardware and application needs, ensuring that at least one option remains practical to run on any virtual-machine, workstation or mid-range laptop. For instance, users may prioritize speed and efficiency on less powerful machines or higher annotation accuracy on complex images.
Integration with the Fiji ecosystem and Java–Python interoperability
Beyond its ease of use, SAMJ is designed for adaptability and seamless integration. Powered by SAM, it can enhance annotation workflows across a range of Java-based platforms, for example, in addition to Labkit, we have also integrated it into BigDataViewer15. Moreover, to support broader adoption, SAMJ provides a well-documented API, enabling developers to easily incorporate SAM functionality into their preferred Java software effortlessly. Furthermore, it includes a software-agnostic Java GUI, facilitating straightforward integration into any Java-based environment.
A key strength of SAMJ is its integration of Python methods into Java environments. This is achieved through Appose16 (see section 16), a Java package that allows Java and Python to run as separate processes yet communicate in real time. Leveraging Micromamba17, Appose also automates Python environment setup, totally eliminating manual configuration via the command line. As a result, SAMJ is remarkably user-friendly, granting effortless access to advanced methods.
Interactive annotation workflow in Fiji
The annotation process in the SAMJ plugin mirrors SAM’s workflow (Fig. 1). Each image is encoded once by the image encoder, the most computationally intensive step. Afterwards, prompts are processed and masks generated almost instantly, allowing users to interactively produce multiple annotations with real-time responsiveness. SAMJ integrates seamlessly with Fiji’s existing tools, allowing users to create prompts such as points and rectangles directly through Fiji’s familiar toolbar, thereby minimizing the learning curve for new users.
This figure illustrates the typical workflow of SAMJ for annotating objects in an image, including the time required for each step on standard workstations. a Model Installation: The most time-consuming step, where SAMJ installs the selected model and, if necessary, sets up the environment, taking approximately 1000 s. b Image Encoding: When the user clicks “Go” in the plugin, the image is processed to generate an embedding, which takes around 10 s. c Object Annotation: Since no re-embedding is required, this step is immediate, with each annotation generated in approximately 0.1 s per user click. The annotation process can be repeated multiple times as desired for different objects, offering rapid and interactive segmentation.
The SAMJ plugin offers two annotation modes: Live and Batch mode (called BatchSAMize). In Live mode, prompts are drawn directly on the encoded image, and annotations are generated interactively, one at a time. In Batch mode, users can define multiple seed points to serve as prompts for segmenting multiple objects in a raw image. These seeds can be generated automatically using traditional Fiji commands (e.g., Find Maxima, Watershed) within an ImageJ macro, followed by a call to SAMJ’s Batch mode to segment the whole image. Additionally, users can enhance annotations by providing seeds from other segmentation methods as prompts, enabling iterative refinement. Prompts for BatchSAMize can be supplied either manually or automatically. Thanks to SAMJ’s integration with Fiji, users can combine bounding boxes or point prompts generated by deep learning models, macros, or files imported through the ROI Manager. Users can even use the macros to annotate batches of images with SAMJ, as illustrated in the Supplementary Materials. This flexibility enhances both the reproducibility and the automation of image annotation.
Representative use cases
To illustrate the power of SAMJ and its seamless integration with the Fiji ecosystem, we present four representative use cases highlighting its capabilities (Fig. 2). These examples illustrate SAMJ’s ability to adapt to diverse image annotation tasks while taking advantage of Fiji’s powerful image processing tools. In all cases, the targets are compact and well-defined–conditions under which SAMJ performs particularly well. Elongated or branched structures may pose more of a challenge; However, users can refine prompts, adjust scale settings, or combine SAMJ with complementary methods. Experimentation is encouraged, as SAMJ often proves effective even under suboptimal imaging conditions (see section 16).
a Nuclei Segmentation using Fiji’s capabilities and BatchSAMize: An image from the CellPose dataset21 is used to demonstrate the segmentation of nuclei in the red channel. Pixel intensity maxima are identified in Fiji to generate single-point prompts for each nucleus. These prompts are processed in batch mode using SAMJ, resulting in semantic segmentation of individual nuclei. b Tumor Area Quantification and Nuclei Analysis with SAMJ and StarDist plugins for Fiji: Breast cancer TMA (Tissue Microarray) images, stained with H& E (Hematoxylin and Eosin) provided by the British Columbia Cancer Agency (BCCA)28, are used to quantify tumoral regions and their nuclei. Tumoral areas are annotated with SAMJ’s rectangle prompt, generating masks that are combined with the original image in Fiji using an AND operation to obtain the intersection. StarDist is then applied through deepImageJ to segment individual nuclei in the tumoral areas. c Accelerating annotation of Bacterial Motility: SAMJ is applied to annotate motile bacteria using a single rectangle annotation. When compared to Ground Truth, SAMJ’s annotations achieve comparable or superior precision, showcasing its efficiency in handling complex shapes and streamlining high-throughput workflows. Each use case highlights SAMJ’s integration with Fiji, combining SAM’s advanced annotation capabilities with Fiji’s extensive image processing tools. d Efficient annotation of 3D Electron Microscopy images of mitochondria29 using SAMJ and Labkit: SAMJ is integrated into Labkit to support 3D and multi-label annotation. Users can annotate structures on arbitrarily oriented and scaled slices, improving visibility and accuracy of objects with complex spatial orientation. The multi-class labeling capability of Labkit allows the annotation of several distinct structures within the same volume. This workflow reduces annotation effort and ensures spatial consistency across slices.
In the first use case, we illustrate nuclei segmentation in a fluorescence image using the BatchSAMize mode of SAMJ. Classical Fiji commands are used to detect nuclei, which serve as single-point prompts for batch processing in SAMJ. This workflow highlights the synergy between the preprocessing capabilities of Fiji and the segmentation power of SAMJ, streamlining large-scale annotation tasks.
The second use case intended for tumor area quantification focuses on breast cancer TMA images stained with H&E. The task involved quantifying tumoral areas and the number of nuclei within these regions. Using SAMJ’s rectangle annotation feature, tumoral regions were outlined, generating masks for these areas. Subsequently, individual nuclei were segmented with StarDist18 using the deepImageJ7,19 plugin. This workflow demonstrates SAMJ’s role as a flexible and efficient annotation tool that seamlessly integrates with Fiji, enabling efficient annotation and precise quantification for tumor analysis.
The third use case highlights SAMJ’s efficiency in annotating complex shapes20. Bacteria from motility studies were annotated using a single rectangle prompt per image, significantly reducing manual effort. When compared to manually generated ground truth, SAMJ’s annotations are highly consistent with the reference labels, as supported by the IoU results reported in Table 4. This use case demonstrates how SAMJ accelerates annotation tasks for intricate biological structures, making it a powerful tool for high-throughput studies.
The final use case demonstrates SAMJ’s capability for efficient annotation of mitochondria instances in 3D electron microscopy volumes through its integration with Labkit. Annotating objects in 3D is inherently challenging due to the difficulty of maintaining spatial coherence across slices and the substantial effort required for manual annotation. SAMJ streamlines this process, and its integration into Labkit–designed for interactive 3D visualization–enables users to annotate structures on arbitrarily oriented slices, ensuring optimal views of the object of interest. Moreover, Labkit support for multi-class labeling allows users to assign distinct labels to multiple structures within the same volume. This combination significantly reduces annotation effort while improving both accuracy and consistency, highlighting the power of integrating SAMJ with complementary Fiji tools for complex 3D bioimage annotation tasks. In this example, every slice of the volume was annotated with SAMJ, effectively following a pragmatic 2.5 D strategy. Thanks to SAMJ’s rapid interactive prompting, this approach still provided a substantial gain in speed compared to fully manual slice-by-slice annotation.
Together, these use cases highlight the versatility and strength of SAMJ integration within the Fiji ecosystem. By combining the advanced segmentation capabilities of SAM with the rich set of image processing tools of Fiji, SAMJ enables more efficient, precise, and scalable annotation workflows across a wide range of biological imaging tasks. This integration lowers technical barriers and supports diverse use cases, from 2D fluorescence images to complex 3D volumes. Consequently, it empowers the broader life science community–particularly biologists who primarily rely on GUI software platforms–with powerful deep-learning tools that would otherwise remain accessible only to a few, accelerating and improving their bioimage analysis.
Annotating complex objects with multi-step prompting
The Segment Anything Model was originally trained with a one-prompt/one-object paradigm on a wide variety of objects, mostly from natural images. As a result, biological structures with intricate morphologies, such as neurons, are often difficult to annotate with a single prompt, regardless of prompt placement.
However, thanks to SAMJ interactivity, users can overcome this limitation by enabling a multi-step, compositional annotation workflow. While SAM may struggle to segment a branched structure globally, it remains highly effective at detecting local boundaries. SAMJ leverages this strength by allowing users to segment distinct sub-structures (e.g., soma, axons, and dendrites) individually. These partial masks can then be merged into a single, coherent object. This piecewise strategy yields significantly more accurate annotations for elongated or highly branched structures than single-prompt inference (see Fig. 3).
SAMJ’s interactive workflow allows users to segment multiple parts of an object that the model recognizes and subsequently merge them. The figure shows a fluorescent neuron30 with two ramifications. Using a single box prompt (middle image), the model captures only one ramification. By annotating the second ramification with additional point prompts (purple dots on the right image), the user obtains two separate labels whose union yields the final mask of the complex neuron.
