Add Sentence Transformers integration with CLS pooling

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -28,6 +28,30 @@ PubMedNCL: Working with biomedical papers? Try [PubMedNCL](https://huggingface.c
 
 ## How to use the pretrained model
 
+### Sentence Transformers
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# Load the model
+model = SentenceTransformer("malteos/scincl")
+
+# Concatenate the title and abstract with the [SEP] token
+papers = [
+    "BERT [SEP] We introduce a new language representation model called BERT",
+    "Attention is all you need [SEP] The dominant sequence transduction models are based on complex recurrent or convolutional neural networks",
+]
+# Inference
+embeddings = model.encode(papers)
+
+# Compute the (cosine) similarity between embeddings
+similarity = model.similarity(embeddings[0], embeddings[1])
+print(similarity.item())
+# => 0.8440517783164978
+```
+
+### Transformers
+
 ```python
 from transformers import AutoTokenizer, AutoModel
 
@@ -49,6 +73,12 @@ result = model(**inputs)
 
 # take the first token ([CLS] token) in the batch as the embedding
 embeddings = result.last_hidden_state[:, 0, :]
+
+# calculate the similarity
+embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
+similarity = (embeddings[0] @ embeddings[1].T)
+print(similarity.item())
+# => 0.8440518379211426
 ```
 
 ## Triplet Mining Parameters

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.0.0",
+    "transformers": "4.41.2",
+    "pytorch": "2.3.0+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}