iloncka/culico-net-segm-v1-nano

Model Card for culico-net-segm-v1-nano

culico-net-segm-v1-nano - instance segmentation model focused on segmenting mosquito specimens in images and videos. This model is a result of the CulicidaeLab project and was developed by fine-tuning the SAM2 (Segment Anything Model 2) architecture.

The culico-net-segm-v1-nano is based on SAM2's efficient Hiera-Tiny backbone and was specifically adapted for mosquito specimen segmentation using a dedicated dataset. This model can handle both image segmentation and video segmentation tasks with prompt-based interaction.

Model Details:

• Model Type: Instance segmentation / Mask generation
• Architecture: SAM2 with Hiera-Tiny backbone
• Input: Images or video frames with optional prompts (points, boxes)
• Output: Binary masks for mosquito specimens
• Backbone: Hiera-Tiny (efficient transformer architecture)

Papers:

• SAM 2: Segment Anything in Images and Videos: https://arxiv.org/abs/2408.00714
• Original GitHub Repository: https://github.com/facebookresearch/sam2

Dataset: The model was trained on the iloncka/mosquito-species-segmentation-dataset. This is one of a suite of datasets which also includes iloncka/mosquito-species-detection-dataset and iloncka/mosquito-species-classification-dataset. These datasets contain images of various mosquito species with detailed segmentation annotations, crucial for training accurate segmentation models.

Pretrain Dataset: ImageNet-1k, SA-1B (Segment Anything 1B dataset)

Model Usage:

The model can be used for both image and video segmentation tasks. Below are code snippets demonstrating how to use the model:

For image prediction:

import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("iloncka/culico-net-segm-v1-nano")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

For video prediction:

import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("iloncka/culico-net-segm-v1-nano")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Refer to the official SAM2 demo notebooks for details.

The CulicidaeLab Project:

The culico-net-segm-v1-nano model is a component of the larger CulicidaeLab project. This project aims to provide a comprehensive suite of tools for mosquito monitoring and research. Other parts of the project include:

• Related Models:

  • iloncka/culico-net-cls-v1 - Classification model
  • iloncka/culico-net-det-v1 - Detection model

• Datasets:

  • iloncka/mosquito-species-detection-dataset
  • iloncka/mosquito-species-segmentation-dataset
  • iloncka/mosquito-species-classification-dataset

• Python Library: https://github.com/iloncka-ds/culicidaelab

• Mobile Applications:

  • https://gitlab.com/mosquitoscan/mosquitoscan-app
  • https://github.com/iloncka-ds/culicidaelab-mobile

• Web Application: https://github.com/iloncka-ds/culicidaelab-server

Practical Applications:

The culico-net-segm-v1-nano model and the broader CulicidaeLab project have several practical applications:

• Precise Morphological Analysis: Enables detailed examination of mosquito anatomical features for species identification and research.
• Automated Specimen Measurement: Can be used to automatically measure wing dimensions, body size, and other morphological characteristics.
• Training Data Generation: Creates high-quality segmentation masks for training other computer vision models.
• Embedded Systems (Edge AI): The efficient architecture allows deployment on edge devices for real-time mosquito monitoring in field conditions.
• Expert Systems Integration: Provides precise segmentation capabilities that can assist entomologists in detailed specimen analysis.

Acknowledgments:

The development of CulicidaeLab is supported by a grant from the Foundation for Assistance to Small Innovative Enterprises (FASIE).

Citation:

To cite the SAM2 paper, model, or software, please use the below:

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}