|
|
--- |
|
|
library_name: transformers |
|
|
tags: [] |
|
|
--- |
|
|
|
|
|
# Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
MiewID-msv2 is a feature extractor trained for re-identification using contrastive learning on a large, high-quality dataset of 54 wildlife species - terrestrial and aquatic - including fins, flukes, flanks, faces. |
|
|
|
|
|
- **Model Type:** Wildlife re-identification feature backbone |
|
|
- **Model Stats:** |
|
|
- Params (M): 51.11 |
|
|
- GMACs: 24.38 |
|
|
- Activations (M): 91.11 |
|
|
- Image size: 440 x 440 |
|
|
|
|
|
### Model Sources [optional] |
|
|
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
|
|
- **Repository:** https://github.com/WildMeOrg/wbia-plugin-miew-id |
|
|
- **Backbone:** https://huggingface.co/timm/efficientnetv2_rw_m.agc_in1k |
|
|
|
|
|
# Usage |
|
|
|
|
|
Intended use is re-identification of individuals from different species by matching against a database of grount-truth samples. Model features can also be used for species classification by retrieval. |
|
|
|
|
|
### Embedding Extraction |
|
|
``` |
|
|
import numpy as np |
|
|
from PIL import Image |
|
|
import torch |
|
|
import torchvision.transforms as transforms |
|
|
from transformers import AutoModel |
|
|
|
|
|
model_tag = f"conservationxlabs/miewid-msv2" |
|
|
model = AutoModel.from_pretrained(model_tag, trust_remote_code=True) |
|
|
|
|
|
def generate_random_image(height=440, width=440, channels=3): |
|
|
random_image = np.random.randint(0, 256, (height, width, channels), dtype=np.uint8) |
|
|
return Image.fromarray(random_image) |
|
|
|
|
|
random_image = generate_random_image() |
|
|
|
|
|
preprocess = transforms.Compose([ |
|
|
transforms.Resize((440, 440)), |
|
|
transforms.ToTensor(), |
|
|
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), |
|
|
]) |
|
|
|
|
|
input_tensor = preprocess(random_image) |
|
|
input_batch = input_tensor.unsqueeze(0) |
|
|
|
|
|
with torch.no_grad(): |
|
|
output = model(input_batch) |
|
|
|
|
|
print(output) |
|
|
print(output.shape) |
|
|
|
|
|
``` |
|
|
|
|
|
### More Examples |
|
|
|
|
|
View more usage examples at https://github.com/WildMeOrg/wbia-plugin-miew-id/tree/main/wbia_miew_id/examples |
|
|
|
|
|
# Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
|
|
The dataset used for these experiments was a combination of data from Wildbook platforms (multiple users), Happywhale Kaggle competitions multi-species dataset and multiple publicly available datasets. A small subset of data from Wildbook platforms is available at https://lila.science/datasets. |
|
|
|
|
|
|
|
|
### Example images |
|
|
|
|
|
|
|
|
 |
|
|
|
|
|
# Evaluation Results |
|
|
|
|
|
The multi-species model shows improvements over models trained on single species. Moreover the model shows strong generalization ability for majority of the species when trained and evaluated in a leave-one-out fashion. |
|
|
|
|
|
### Detailed results |
|
|
|
|
|
|Source |group |map |rank-1|rank-5|rank-10|rank-20|viewpoints |n_test_annots|n_test_names|n_train_annots|n_train_names| |
|
|
|--------------|--------------------------------|-----|------|------|-------|-------|--------------------------------------------------------------------------------------------------------------------|-------------|------------|--------------|-------------| |
|
|
|Wildbook |amur_tiger |91.8 |100.0 |100.0 |100.0 |100.0 |['left', 'right'] |233 |47 |691 |75 | |
|
|
|Wildbook |beluga_whale |61.36|71.85 |81.63 |85.51 |89.28 |['up'] |849 |228 |2767 |354 | |
|
|
|Happywhale |blue_whale |36.98|38.68 |55.11 |61.52 |69.04 |['unknown'] |998 |392 |1185 |339 | |
|
|
|Wildbook |bottlenose_dolphin |85.69|93.99 |96.18 |96.85 |97.52 |['right', 'left'] |4911 |838 |11607 |1062 | |
|
|
|Happywhale |brydes_whale |69.96|80.95 |92.86 |95.24 |100.0 |['unknown'] |42 |9 |81 |15 | |
|
|
|Lomas Capuchin|capuchin |34.8 |47.97 |68.58 |76.69 |85.14 |['front'] |296 |35 |1708 |29 | |
|
|
|Wildbook |cheetah |57.37|71.15 |84.92 |87.87 |92.13 |['left', 'right'] |624 |139 |1522 |144 | |
|
|
|Wildbook |chimpanzee |94.7 |100.0 |100.0 |100.0 |100.0 |['unknown'] |52 |8 |138 |9 | |
|
|
|C-Tai |chimpanzee_ctai |51.67|69.64 |83.68 |89.18 |94.12 |['front'] |527 |63 |2680 |44 | |
|
|
|C-Zoo |chimpanzee_czoo |77.66|86.67 |93.33 |97.5 |99.17 |['front'] |240 |24 |1258 |17 | |
|
|
|Happywhale |commersons_dolphin |0.0 |0.0 |0.0 |0.0 |0.0 |['unknown'] |10 |5 |7 |2 | |
|
|
|Happywhale |cuviers_beaked_whale |52.89|55.71 |75.71 |82.86 |90.0 |['unknown'] |70 |28 |89 |21 | |
|
|
|DogFaceNet |dog |82.96|87.38 |94.78 |96.74 |97.99 |['unknown'] |1838 |544 |5750 |1088 | |
|
|
|Happywhale |dusky_dolphin |85.82|83.54 |93.04 |94.3 |94.94 |['unknown'] |158 |68 |154 |41 | |
|
|
|Wildbook |eurasianlynx |58.8 |67.82 |83.78 |88.83 |92.02 |['left', 'right'] |726 |199 |2251 |285 | |
|
|
|Happywhale |fin_whale |67.65|76.92 |87.18 |90.11 |93.04 |['unknown'] |273 |71 |533 |55 | |
|
|
|Happywhale |frasiers_dolphin |0.0 |0.0 |0.0 |0.0 |0.0 | |-1 |-1 |-1 |-1 | |
|
|
|Wildbook |giraffe |99.18|99.08 |99.08 |100.0 |100.0 |['right', 'front', 'left', 'back'] |423 |162 |2540 |426 | |
|
|
|Wildbook |horse_wild+face |0.0 |0.0 |0.0 |0.0 |0.0 |['front'] |17 |7 |174 |53 | |
|
|
|Wildbook |horse_wild_tunisian+face |79.59|98.53 |99.51 |100.0 |100.0 |['unknown'] |204 |39 |999 |37 | |
|
|
|Wildbook |hyena |64.17|79.32 |89.22 |93.14 |96.41 |['right', 'left'] |1231 |301 |1154 |201 | |
|
|
|Wildbook |hyperoodon_ampullatus |91.14|96.11 |98.12 |98.34 |98.84 |['left', 'right'] |1930 |354 |24952 |759 | |
|
|
|Wildbook |jaguar |67.45|81.72 |91.04 |92.91 |94.4 |['left', 'right'] |540 |124 |980 |103 | |
|
|
|PrimFace |japanese_monkey |89.05|93.85 |98.46 |98.46 |100.0 |['unknown'] |65 |12 |219 |14 | |
|
|
|Happywhale |killer_whale |62.82|77.13 |85.48 |88.81 |91.82 |['unknown'] |599 |145 |1379 |177 | |
|
|
|Wildbook |leopard |65.03|82.64 |90.23 |92.83 |94.46 |['right', 'left'] |595 |108 |4261 |381 | |
|
|
|Wildbook |lion |87.12|93.86 |100.0 |100.0 |100.0 |['front', 'unknown', 'frontright', 'frontleft', 'upfront', 'right', 'upleft', 'left', 'upright', 'backleft', 'back']|1364 |242 |3981 |241 | |
|
|
|Happywhale |long_finned_pilot_whale |99.06|98.11 |100.0 |100.0 |100.0 |['unknown'] |53 |21 |33 |10 | |
|
|
|Wildbook |lynx_pardinus |48.22|57.68 |73.44 |77.18 |84.96 |['left', 'right'] |467 |142 |1376 |259 | |
|
|
|MacaqueFaces |macaque_face |89.87|95.88 |97.94 |99.12 |100.0 |['front'] |340 |34 |3480 |25 | |
|
|
|Happywhale |melon_headed_whale |90.8 |93.42 |96.05 |96.71 |98.68 |['unknown'] |152 |55 |180 |43 | |
|
|
|Wildbook |nyala |64.64|70.65 |90.05 |94.03 |97.01 |['left', 'right'] |418 |99 |1303 |149 | |
|
|
|Happywhale |pantropic_spotted_dolphin |78.15|85.19 |90.74 |94.44 |96.3 |['unknown'] |54 |11 |55 |8 | |
|
|
|Happywhale |pygmy_killer_whale |89.07|84.62 |100.0 |100.0 |100.0 |['unknown'] |26 |10 |32 |5 | |
|
|
|PrimFace |rhesus_monkey |78.12|89.13 |97.28 |98.91 |100.0 |['unknown'] |184 |28 |478 |24 | |
|
|
|Happywhale |rough_toothed_dolphin |0.0 |0.0 |0.0 |0.0 |0.0 |['unknown'] |6 |3 |7 |2 | |
|
|
|SealID |seal |37.93|59.51 |80.0 |86.1 |91.71 |['unknown'] |410 |41 |1448 |36 | |
|
|
|Happywhale |sei_whale |67.86|71.55 |87.07 |93.1 |95.69 |['unknown'] |116 |35 |106 |29 | |
|
|
|Wildbook |short_fin_pilot_whale+fin_dorsal|92.49|92.31 |98.08 |100.0 |100.0 |['right', 'left'] |272 |64 |1290 |137 | |
|
|
|Happywhale |short_finned_pilot_whale |94.39|94.96 |95.8 |95.8 |97.48 |['unknown'] |119 |57 |67 |17 | |
|
|
|Wildbook |snow_leopard |62.43|82.45 |88.91 |92.15 |94.53 |['left', 'right'] |927 |144 |2307 |99 | |
|
|
|Happywhale |spinner_dolphin |99.88|100.0 |100.0 |100.0 |100.0 |['left', 'right'] |335 |97 |1562 |251 | |
|
|
|Happywhale |spotted_dolphin |84.92|83.67 |89.8 |95.92 |97.96 |['unknown'] |49 |22 |128 |28 | |
|
|
|Wildbook |turtle_green |100.0|100.0 |100.0 |100.0 |100.0 |['left', 'right', 'up', 'front'] |543 |109 |5097 |76 | |
|
|
|Wildbook |turtle_green+head |95.1 |96.02 |100.0 |100.0 |100.0 |['left', 'right', 'up', 'front'] |544 |108 |5737 |76 | |
|
|
|Wildbook |turtle_hawksbill |78.84|81.15 |88.37 |92.56 |95.81 |['up', 'left', 'right', 'front'] |506 |106 |5397 |71 | |
|
|
|Wildbook |turtle_hawksbill+head |86.78|91.74 |95.45 |96.69 |98.35 |['right', 'left', 'up', 'front'] |589 |109 |7033 |71 | |
|
|
|SeaTurtleID |turtle_loggerhead_ext |63.67|75.43 |86.61 |90.17 |92.87 |['unknown', 'left', 'top', 'right'] |815 |217 |6573 |355 | |
|
|
|SeaTurtleID |turtle_loggerhead_ext+head |94.14|94.63 |100.0 |100.0 |100.0 |['left', 'right', 'topright', 'top', 'topleft', 'below', 'front'] |778 |209 |6361 |348 | |
|
|
|Wildbook |whale_fin+fin_dorsal |82.12|85.12 |93.87 |95.04 |95.75 |['left', 'right'] |394 |110 |1629 |186 | |
|
|
|Wildbook |whale_grey |90.97|91.55 |95.54 |96.18 |96.82 |['right', 'left'] |381 |103 |5662 |758 | |
|
|
|Wildbook |whale_humpback+fin_dorsal |80.3 |84.07 |90.08 |91.6 |95.42 |['right', 'left'] |509 |194 |3734 |541 | |
|
|
|Wildbook |whale_humpback+fluke |97.11|96.67 |100.0 |100.0 |100.0 |['back'] |150 |64 |1151 |356 | |
|
|
|Wildbook |whale_orca |97.01|98.36 |99.18 |100.0 |100.0 |['right', 'left'] |249 |49 |2790 |435 | |
|
|
|Wildbook |whale_orca+fin_dorsal |95.41|95.87 |97.52 |97.52 |99.17 |['right', 'left'] |623 |187 |3978 |430 | |
|
|
|Wildbook |whale_sperm+fluke |96.14|97.87 |98.33 |98.48 |99.09 |['back'] |658 |125 |25849 |558 | |
|
|
|Wildbook |whaleshark |57.61|72.25 |79.96 |82.27 |87.21 |['left', 'right'] |997 |212 |5713 |376 | |
|
|
|Wildbook |white_shark+fin_dorsal |91.4 |94.89 |97.87 |98.54 |99.27 |['left', 'right'] |322 |88 |1067 |162 | |
|
|
|Happywhale |white_sided_dolphin |81.82|76.0 |88.0 |88.0 |100.0 |['unknown'] |25 |11 |13 |4 | |
|
|
|Wildbook |wilddog |73.4 |85.11 |90.93 |92.38 |94.07 |['left', 'right'] |3460 |848 |4217 |656 | |
|
|
|Wildbook |zebra_grevys |91.79|96.79 |97.57 |97.74 |98.04 |['right'] |1684 |332 |8927 |362 | |
|
|
|
|
|
|
