HuggingFaceFW
/

finepdfs_edu_classifier_afr_Latn

Safetensors

Afrikaans

modernbert

Model card Files Files and versions

xet

Community

hynky HF Staff commited on Oct 6

Commit

080fbc6

verified ·

1 Parent(s): 9088e16

Add model card for afr_Latn classifier

Browse files

Files changed (1) hide show

README.md +168 -196

README.md CHANGED Viewed

@@ -1,199 +1,171 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- af
+license: apache-2.0
+datasets:
+- HuggingFaceFW/finepdfs_fw_edu_labeled
 ---
+# FinePDFs-Edu classifier (afr_Latn)
+## Model summary
+This is a classifier for judging the educational value of web pages. It was developed to filter and curate educational content from web datasets and was trained on 176221 [annotations](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_fw_edu_labeled) generated by [Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) for web samples from [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs) dataset.
+We used this classifier to build [FinePDFs-Edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu) dataset.
+### How to use in transformers
+To load the FinePDFs-Edu classifier, use the following code:
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+import re
+CHUNK_SIZE = 2048 - 2
+MAX_CHARS = 10_000
+tokenizer = AutoTokenizer.from_pretrained("HuggingFaceFW/finepdfs_edu_classifier_afr_Latn")
+model = AutoModelForSequenceClassification.from_pretrained("HuggingFaceFW/finepdfs_edu_classifier_afr_Latn")
+regex_whitespace = re.compile(r'\s')
+def create_text_chunks(text: str, tokenizer):
+    def trim_to_whitespace(text: str, trim_start: bool = True, trim_end: bool = True):
+        if trim_start:
+            match = regex_whitespace.search(text)
+            if match:
+                text = text[match.start()+1:]
+            else:
+                text = text[10:]
+        if trim_end:
+            match = regex_whitespace.search(text[::-1])
+            if match:
+                text = text[:len(text) - match.start() - 1]
+            else:
+                text = text[:-10]
+        return text
+    # First tokenize the text
+    # Speed hack, we take at most
+    if len(text) <= 2*MAX_CHARS:
+        tokens = tokenizer.encode(text[:MAX_CHARS], return_tensors="np", add_special_tokens=False)[0]
+        # Process the top chunks
+        chunks_from_top_sampled = [tokens[:CHUNK_SIZE]]
+        chunks_top_text = tokenizer.batch_decode(chunks_from_top_sampled, skip_special_tokens=True)
+        chunks_top_text = [trim_to_whitespace(chunks_top_text[0], trim_start=False, trim_end=True)]
+        return [chunks_top_text]
+    else:
+        # We tokenize the top and bottom of text
+        text_top = text[:MAX_CHARS]
+        text_bottom = text[-MAX_CHARS:]
+        tokens = tokenizer.batch_encode_plus([text_top, text_bottom], return_tensors="np", add_special_tokens=False)["input_ids"]
+        # This ensures that the second chunks is always maxed out
+        chunks = [tokens[0][:CHUNK_SIZE], tokens[1][-CHUNK_SIZE:]]
+        chunks_text = tokenizer.batch_decode(chunks, skip_special_tokens=True)
+        chunks_top_text = [trim_to_whitespace(chunks_text[0], trim_start=False, trim_end=True)]
+        chunks_bottom_text = [trim_to_whitespace(chunks_text[1], trim_start=True, trim_end=False)]
+        return chunks_top_text + chunks_bottom_text
+text = "This is a test sentence." * 2000
+chunks = create_text_chunks(text, tokenizer)
+scores = []
+for chunk in chunks:
+    inputs = tokenizer(chunk, return_tensors="pt", padding="longest", truncation=True)
+    outputs = model(**inputs)
+    logits = outputs.logits.squeeze(-1).float().detach().numpy()
+    score = logits.item()
+    scores.append(score)
+print(max(scores))
+```
+## Training
+The classifier was trained on 302343 pairs of web samples and their scores from 0 to 5, generated by Qwen3-235B-A22B-Instruct-2507. The samples were annotated based on their educational quality with 0 being not educational and 5 being highly educational.
+Below is the prompt used for Qwen3-235B-A22B-Instruct-2507 annotations:
+```
+Below is an extract from a PDF file. Evaluate whether the extract has a high educational
+value and could be useful in an educational setting for teaching from primary school to
+grade school levels using the additive 5-point scoring system described below. Points are
+accumulated based on the satisfaction of each criterion:
+- Add 1 point if the extract provides some basic information relevant to educational topics, even if it includes some irrelevant or non-academic content like advertisements and
+promotional material.
+- Add another point if the extract addresses certain elements pertinent to education but
+does not align closely with educational standards. It might mix educational content with
+non-educational material, offering a superficial overview of potentially useful topics, or
+presenting information in a disorganized manner and incoherent writing style.
+- Award a third point if the extract is appropriate for educational use and introduces key
+concepts relevant to school curricula. It is coherent though it may not be comprehensive
+or could include some extraneous information. It may resemble an introductory section of
+a textbook or a basic tutorial that is suitable for learning but has notable limitations like
+treating concepts that are too complex for grade school students.
+- Grant a fourth point if the extract highly relevant and beneficial for educational purposes
+for a level not higher than grade school, exhibiting a clear and consistent writing style. It
+could be similar to a chapter from a textbook or a tutorial, offering substantial educational
+content, including exercises and solutions, with minimal irrelevant information, and the
+concepts aren’t too advanced for grade school students. The content is coherent, focused,
+and valuable for structured learning.
+- Bestow a fifth point if the extract is outstanding in its educational value, perfectly suited for
+teaching either at primary school or grade school. It follows detailed reasoning, the writing
+style is easy to follow and offers profound and thorough insights into the subject matter,
+devoid of any non-educational or complex content.
+The extract: {example}.
+After examining the extract:
+- Briefly justify your total score, up to 100 words.
+- Conclude with the score using the format: "Educational score: <total points>"\
+```
+We added a classification head with a single regression output to [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base), unroze the last 4 layers and trained the model for 5000 steps with a learning rate of 3e-4.
+**Training Details:**
+- Model: [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) with a classification head
+- Dataset: 302343 samples from Qwen3-235B-A22B-Instruct-2507 annotations
+- Steps: 5000
+- Learning Rate: 3e-4
+- class distribution: {0: 35960, 1: 122543, 2: 35960, 3: 35960, 4: 35960, 5: 35960}
+- Evaluation Metric: F1 score
+**Classification report**
+We treat the regression model's predictions as discrete classes to calculate the metrics on a hold-out set of 7047 Qwen3-235B-A22B-Instruct-2507-annotated samples.
+```
+Validation Report:
+|   class |   precision |   recall |   f1-score |   support |
+|--------:|------------:|---------:|-----------:|----------:|
+|       0 |        0.62 |     0.65 |       0.63 |      1003 |
+|       1 |        0.88 |     0.85 |       0.86 |      4902 |
+|       2 |        0.41 |     0.48 |       0.44 |       608 |
+|       3 |        0.42 |     0.45 |       0.44 |       273 |
+|       4 |        0.67 |     0.62 |       0.65 |       225 |
+|       5 |        0.55 |     0.33 |       0.41 |        36 |
+```
+**Confusion matrix**
+We verify that the predicted educational scores are indeed close to their ground truth, and are mostry impacted by the noisy annotation.
+```
+Confusion Matrix:
+|   class  |   0 |    1 |   2 |   3 |   4 |   5 |
+|---------:|----:|-----:|----:|----:|----:|----:|
+|        0 | 653 |  348 |   2 |   0 |   0 |   0 |
+|        1 | 405 | 4165 | 310 |  21 |   1 |   0 |
+|        2 |   0 |  223 | 289 |  88 |   8 |   0 |
+|        3 |   0 |   19 |  90 | 124 |  40 |   0 |
+|        4 |   0 |    4 |  13 |  58 | 140 |  10 |
+|        5 |   0 |    0 |   1 |   3 |  20 |  12 |
+```
+## Limitations
+While the FinePDFs-Edu classifier performs well in distinguishing high-quality educational content for FinePDFs dataset, there are some limitations:
+- Scope: The model's performance might change for other datasets, in particular for out of distribution samples. It is also focused on educational content relevant to primary and grade school levels and may not perform as well on content intended for higher education or specialized domains.
+- Bias: The model's performance is dependent on the quality and representativeness of the training data and the LLM used for the annotation. Biases in both can affect the classifier's judgments. It might overfit to academic looking content for the higher scores and we recommend using int_score >= 1.35 (top 10% for english) as a threshold for data curation.
+- Context: The classifier evaluates individual web pages or extracts without considering broader context, which might impact its effectiveness in certain scenarios.
+The training and inference code is available on GitHub
+https://github.com/huggingface/finepdfs/tree/main/classification