Hugging Face's logo

JetBrains
/
Mellum-4b-sft-all-gguf

Transformers
GGUF
code
Eval Results
Model card Files
xet
Community
1
topshik ivan-bondyrev commited on
Commit
9422dbc
·
verified ·
1 Parent(s): 3a4809d

Add evaluation results for RepoBench, SAFIM, HumanEval (#1)

Browse files

- Add evaluation results for RepoBench, SAFIM, HumanEval (e3e5887b059b90936b33ca6e03e1d092240782fa)


Co-authored-by: Ivan Bondyrev <[email protected]>

Files changed (1) hide show
  1. README.md +274 -0
README.md CHANGED
@@ -10,6 +10,241 @@ tags:
10
  - code
11
  base_model:
12
  - JetBrains/Mellum-4b-base
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
  # Model Description
@@ -37,6 +272,45 @@ llama-cli -m mellum-4b-sft-all.Q8_0.gguf --temp 0 -p $'<filename>Utils.kt\npacka
37
 
38
  ```
39
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  # Citation
41
  If you use this model, please cite:
42
 
 
10
  - code
11
  base_model:
12
  - JetBrains/Mellum-4b-base
13
+ model-index:
14
+ - name: Mellum-4b-sft-all
15
+ results:
16
+ # --------------------------- RepoBench 1.1 – Python ---------------------------
17
+ - task:
18
+ type: text-generation
19
+ dataset:
20
+ type: tianyang/repobench_python_v1.1
21
+ name: RepoBench 1.1 (Python)
22
+ metrics:
23
+ - name: EM
24
+ type: exact_match
25
+ value: 0.2823
26
+ verified: false
27
+ - name: EM ≤ 8k
28
+ type: exact_match
29
+ value: 0.2870
30
+ verified: false
31
+
32
+ - task:
33
+ type: text-generation
34
+ dataset:
35
+ type: tianyang/repobench_python_v1.1
36
+ name: RepoBench 1.1 (Python, 2k)
37
+ metrics:
38
+ - name: EM
39
+ type: exact_match
40
+ value: 0.2638
41
+ verified: false
42
+
43
+ - task:
44
+ type: text-generation
45
+ dataset:
46
+ type: tianyang/repobench_python_v1.1
47
+ name: RepoBench 1.1 (Python, 4k)
48
+ metrics:
49
+ - name: EM
50
+ type: exact_match
51
+ value: 0.2930
52
+ verified: false
53
+
54
+ - task:
55
+ type: text-generation
56
+ dataset:
57
+ type: tianyang/repobench_python_v1.1
58
+ name: RepoBench 1.1 (Python, 8k)
59
+ metrics:
60
+ - name: EM
61
+ type: exact_match
62
+ value: 0.3042
63
+ verified: false
64
+
65
+ - task:
66
+ type: text-generation
67
+ dataset:
68
+ type: tianyang/repobench_python_v1.1
69
+ name: RepoBench 1.1 (Python, 12k)
70
+ metrics:
71
+ - name: EM
72
+ type: exact_match
73
+ value: 0.2685
74
+ verified: false
75
+
76
+ - task:
77
+ type: text-generation
78
+ dataset:
79
+ type: tianyang/repobench_python_v1.1
80
+ name: RepoBench 1.1 (Python, 16k)
81
+ metrics:
82
+ - name: EM
83
+ type: exact_match
84
+ value: 0.2818
85
+ verified: false
86
+
87
+ # --------------------------- RepoBench 1.1 – Java ----------------------------
88
+ - task:
89
+ type: text-generation
90
+ dataset:
91
+ type: tianyang/repobench_java_v1.1
92
+ name: RepoBench 1.1 (Java)
93
+ metrics:
94
+ - name: EM
95
+ type: exact_match
96
+ value: 0.2867
97
+ verified: false
98
+ - name: EM ≤ 8k
99
+ type: exact_match
100
+ value: 0.3023
101
+ verified: false
102
+
103
+ - task:
104
+ type: text-generation
105
+ dataset:
106
+ type: tianyang/repobench_java_v1.1
107
+ name: RepoBench 1.1 (Java, 2k)
108
+ metrics:
109
+ - name: EM
110
+ type: exact_match
111
+ value: 0.2883
112
+ verified: false
113
+
114
+ - task:
115
+ type: text-generation
116
+ dataset:
117
+ type: tianyang/repobench_java_v1.1
118
+ name: RepoBench 1.1 (Java, 4k)
119
+ metrics:
120
+ - name: EM
121
+ type: exact_match
122
+ value: 0.3228
123
+ verified: false
124
+
125
+ - task:
126
+ type: text-generation
127
+ dataset:
128
+ type: tianyang/repobench_java_v1.1
129
+ name: RepoBench 1.1 (Java, 8k)
130
+ metrics:
131
+ - name: EM
132
+ type: exact_match
133
+ value: 0.2958
134
+ verified: false
135
+
136
+ - task:
137
+ type: text-generation
138
+ dataset:
139
+ type: tianyang/repobench_java_v1.1
140
+ name: RepoBench 1.1 (Java, 12k)
141
+ metrics:
142
+ - name: EM
143
+ type: exact_match
144
+ value: 0.2447
145
+ verified: false
146
+
147
+ - task:
148
+ type: text-generation
149
+ dataset:
150
+ type: tianyang/repobench_java_v1.1
151
+ name: RepoBench 1.1 (Java, 16k)
152
+ metrics:
153
+ - name: EM
154
+ type: exact_match
155
+ value: 0.2821
156
+ verified: false
157
+
158
+ # --------------------------- SAFIM ------------------------------------------
159
+ - task:
160
+ type: text-generation
161
+ dataset:
162
+ type: gonglinyuan/safim
163
+ name: SAFIM
164
+ metrics:
165
+ - name: pass@1
166
+ type: pass@1
167
+ value: 0.5285
168
+ verified: false
169
+
170
+ - task:
171
+ type: text-generation
172
+ dataset:
173
+ type: gonglinyuan/safim
174
+ name: SAFIM (API)
175
+ metrics:
176
+ - name: pass@1
177
+ type: pass@1
178
+ value: 0.6548
179
+ verified: false
180
+
181
+ - task:
182
+ type: text-generation
183
+ dataset:
184
+ type: gonglinyuan/safim
185
+ name: SAFIM (Block)
186
+ metrics:
187
+ - name: pass@1
188
+ type: pass@1
189
+ value: 0.4005
190
+ verified: false
191
+
192
+ - task:
193
+ type: text-generation
194
+ dataset:
195
+ type: gonglinyuan/safim
196
+ name: SAFIM (Control)
197
+ metrics:
198
+ - name: pass@1
199
+ type: pass@1
200
+ value: 0.5303
201
+ verified: false
202
+
203
+ # --------------------------- HumanEval Infilling ----------------------------
204
+ - task:
205
+ type: text-generation
206
+ dataset:
207
+ type: loubnabnl/humaneval_infilling
208
+ name: HumanEval Infilling (Single-Line)
209
+ metrics:
210
+ - name: pass@1
211
+ type: pass@1
212
+ value: 0.8083
213
+ verified: false
214
+
215
+ - task:
216
+ type: text-generation
217
+ dataset:
218
+ type: loubnabnl/humaneval_infilling
219
+ name: HumanEval Infilling (Multi-Line)
220
+ metrics:
221
+ - name: pass@1
222
+ type: pass@1
223
+ value: 0.4819
224
+ verified: false
225
+
226
+ - task:
227
+ type: text-generation
228
+ dataset:
229
+ type: loubnabnl/humaneval_infilling
230
+ name: HumanEval Infilling (Random Span)
231
+ metrics:
232
+ - name: pass@1
233
+ type: pass@1
234
+ value: 0.3720
235
+ verified: false
236
+
237
+ - task:
238
+ type: text-generation
239
+ dataset:
240
+ type: loubnabnl/humaneval_infilling
241
+ name: HumanEval Infilling (Random Span Light)
242
+ metrics:
243
+ - name: pass@1
244
+ type: pass@1
245
+ value: 0.4024
246
+ verified: false
247
+
248
  ---
249
 
250
  # Model Description
 
272
 
273
  ```
274
 
275
+ ## Benchmarks
276
+ We are providing scores for **Mellum‑4b‑sft‑all** to give users an estimate of the model’s potential capabilities.
277
+
278
+ ### RepoBench 1.1
279
+ *Type:* single‑line  *Languages:* Python and Java  *Metric:* Exact Match (EM), %
280
+
281
+ Since Mellum has a maximum context window of 8 k, we report both the average over **all** evaluated context lengths (2 k, 4 k, 8 k, 12 k and 16 k) and the average over the lengths within its supported range (≤ 8 k).
282
+
283
+ #### Python subset
284
+
285
+ | Model | 2 k | 4 k | 8 k | 12 k | 16 k | Avg | Avg ≤ 8 k |
286
+ |---------------------|------|------|------|------|------|------|-----------|
287
+ | Mellum‑4b‑sft‑all | 26.38% | 29.30% | 30.42% | 26.85% | 28.18% | 28.23% | 28.70% |
288
+
289
+ #### Java subset
290
+
291
+ | Model | 2 k | 4 k | 8 k | 12 k | 16 k | Avg | Avg ≤ 8 k |
292
+ |---------------------|------|------|------|------|------|------|-----------|
293
+ | Mellum‑4b‑sft‑all | 28.83% | 32.28% | 29.58% | 24.47% | 28.21% | 28.67% | 30.23% |
294
+
295
+ ### Syntax‑Aware Fill‑in‑the‑Middle (SAFIM)
296
+ *Type:* mix of multi‑line and single‑line  *Languages:* multi‑language  *Metric:* pass@1, %
297
+
298
+ | Model | Algorithmic | Control | API | Average |
299
+ |---------------------|--------------|---------|-----|---------|
300
+ | Mellum‑4b‑sft‑all | 40.05% | 53.03% | 65.48% | 52.85% |
301
+
302
+ ### HumanEval Infilling
303
+ - Type: single‑line and multi‑line
304
+ - Languages: Python
305
+ - Metric: pass@1, %
306
+
307
+ | Model | Single‑Line | Multi‑Line | Random Span |
308
+ |---------------------|-------------|------------|-------------|
309
+ | Mellum‑4b‑sft‑all | 80.83% | 48.19% | 37.20% |
310
+
311
+
312
+
313
+
314
  # Citation
315
  If you use this model, please cite:
316