Update README.md

README.md

@@ -6,202 +6,438 @@ tags:
 - base_model:adapter:meta-llama/Llama-3.2-3B-Instruct
 - lora
 - transformers
+- medical
+- ner
+- named-entity-recognition
+- healthcare
+- biomedical
+language:
+- en
+license: llama3.2
+metrics:
+- f1
+- precision
+- recall
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# Llama-3.2-3B Medical NER LoRA
 
+A fine-tuned medical Named Entity Recognition (NER) model based on Llama-3.2-3B-Instruct using LoRA (Low-Rank Adaptation) for efficient parameter tuning. This model is specialized for extracting medical entities and relationships from biomedical texts.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-
+This model fine-tunes Llama-3.2-3B-Instruct for medical Named Entity Recognition across three specialized tasks:
+1. **Chemical Extraction**: Identifies drug and chemical compound names
+2. **Disease Extraction**: Identifies disease and medical condition names
+3. **Relationship Extraction**: Identifies chemical-disease interactions (which chemicals influence which diseases)
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+The model was trained on a curated dataset derived from the ChemProt corpus with 2,994 high-quality medical text samples, achieving balanced performance across all three tasks.
 
-### Model Sources [optional]
+- **Developed by:** Alberto Clemente (@albyos)
+- **Model type:** Causal Language Model with LoRA adapters
+- **Language(s):** English (medical/biomedical domain)
+- **License:** Llama 3.2 Community License
+- **Finetuned from model:** meta-llama/Llama-3.2-3B-Instruct
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/albertoclemente/medical-ner-fine-tuning
+- **Training Notebook:** `notebooks/training/Medical_NER_Fine_Tuning_run_20251111.ipynb`
+- **Evaluation Notebook:** `notebooks/evaluation/Medical_NER_Evaluation_BioMistral_7B_SLERP_AWQ_Quantized_20251115.ipynb`
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+This model is designed for extracting structured medical information from unstructured biomedical texts, including:
+- Research papers and clinical studies
+- Medical literature reviews
+- Drug interaction documentation
+- Disease characterization documents
+
+**Input format:**
+```
+The following article contains technical terms including diseases, drugs and chemicals. 
+Create a list only of the [chemicals/diseases/influences] mentioned.
 
-[More Information Needed]
+[MEDICAL TEXT]
 
-### Downstream Use [optional]
+List of extracted [chemicals/diseases/influences]:
+```
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+**Output format:**
+- For chemicals/diseases: Bullet list of entities
+- For relationships: Pipe-separated pairs (`chemical | disease`)
 
-[More Information Needed]
+### Downstream Use
+
+This model can be integrated into:
+- Medical literature mining pipelines
+- Drug discovery workflows
+- Clinical decision support systems
+- Pharmacovigilance systems
+- Biomedical knowledge graph construction
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+This model is **NOT** suitable for:
+- Clinical diagnosis or treatment recommendations
+- Patient-facing medical advice
+- Real-time critical healthcare decisions
+- Languages other than English
+- Non-medical domain NER tasks
 
-[More Information Needed]
+**Important:** This model is for research and information extraction purposes only. It should not be used as a substitute for professional medical judgment.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+### Known Limitations
 
-[More Information Needed]
+1. **Domain Specificity**: Trained on scientific/biomedical literature; may not perform well on clinical notes or patient-facing text
+2. **Entity Coverage**: Limited to chemicals, diseases, and their relationships; doesn't extract other medical entities (procedures, anatomy, etc.)
+3. **Training Data Bias**: Reflects patterns in ChemProt corpus; may not generalize to all medical subdomains
+4. **Hallucination Risk**: As with all LLMs, may occasionally generate plausible but incorrect entities
+5. **Format Sensitivity**: Performance depends on using the exact prompt format from training
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+- **Always validate** extracted entities against authoritative medical databases (ChEBI, MeSH, UMLS)
+- Use in **conjunction with human expert review** for high-stakes applications
+- Monitor for **false positives** (hallucinated entities) and **false negatives** (missed entities)
+- Implement **confidence thresholding** based on your use case requirements
+- Consider ensemble methods with other biomedical NER tools (e.g., BioMistral, PubMedBERT)
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import torch
+
+# Load base model and tokenizer
+base_model_id = "meta-llama/Llama-3.2-3B-Instruct"
+model = AutoModelForCausalLM.from_pretrained(
+    base_model_id,
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(base_model_id)
+
+# Load LoRA adapter
+adapter_model_id = "albyos/llama3-medical-ner-lora-{timestamp}"  # Replace with actual model ID
+model = PeftModel.from_pretrained(model, adapter_model_id)
+
+# Format prompt (example for chemical extraction)
+prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+You are a medical NER expert specialized in extracting entities from biomedical texts.
+Extract entities EXACTLY as they appear in the text.
+
+CRITICAL RULES:
+1. Return ONLY entities found verbatim in the article
+2. Preserve exact formatting: hyphens, capitalization, special characters
+3. Extract complete multi-word terms
+4. For relationships: use format 'chemical NAME | disease NAME'
+
+OUTPUT FORMAT:
+- One entity per line with leading dash
+- No explanations or additional text<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+The following article contains technical terms including diseases, drugs and chemicals. 
+Create a list only of the chemicals mentioned.
+
+Aspirin and ibuprofen are commonly used to treat inflammation. Recent studies show 
+that metformin may reduce the risk of type-2 diabetes complications.
+
+List of extracted chemicals:
+<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+"""
+
+# Generate
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    do_sample=False,
+    temperature=1.0,
+    repetition_penalty=1.15,
+)
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+# Extract assistant response
+if "<|start_header_id|>assistant<|end_header_id|>" in response:
+    result = response.split("<|start_header_id|>assistant<|end_header_id|>")[-1].strip()
+    print(result)
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+**Dataset**: Custom medical NER dataset derived from ChemProt corpus
+- **Total samples**: 2,994 (after cleaning and deduplication)
+- **Source**: Biomedical literature abstracts
+- **Tasks**: Chemical extraction, disease extraction, relationship extraction
+- **Split**: 80% train (2,397), 10% validation (298), 10% test (299)
+- **Quality**: 99.8% retention rate, 0 empty completions, stratified by task
+
+**Data Characteristics** (from exploration analysis):
+- **Unique chemicals**: 1,578 entities
+- **Unique diseases**: 2,199 entities
+- **Vocabulary size**: 13,710 unique words
+- **Prompt length**: Median 1,357 characters (195 words), range 345-4,018 chars
+- **Hyphenated entities**: ~459 (e.g., "type-2 diabetes", "5-fluorouracil")
+- **Format conversion**: 2,050 relationships converted from sentence to pipe format
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+#### Preprocessing
 
+1. **Deduplication**: Removed duplicate prompts by normalized hash
+2. **Format standardization**: Converted relationship format from `"chemical X influences disease Y"` to `"X | Y"`
+3. **Entity normalization**: Lowercase, whitespace normalization, hyphen preservation
+4. **Stratified splitting**: Ensures 33.3% distribution per task across all splits
+5. **Leakage prevention**: Hard assertions verify zero overlap between train/val/test
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+**LoRA Configuration:**
+- **LoRA rank (r)**: 16
+- **LoRA alpha**: 32
+- **LoRA dropout**: 0.05
+- **Target modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+
+**Training Parameters:**
+- **Training regime**: fp16 mixed precision
+- **Quantization**: 4-bit NF4 (BitsAndBytes)
+- **Epochs**: 5
+- **Batch size**: 4 per device
+- **Gradient accumulation**: 4 steps (effective batch = 16)
+- **Learning rate**: 5e-5
+- **LR scheduler**: Cosine with 3% warmup
+- **Weight decay**: 0.01
+- **Optimizer**: paged_adamw_8bit
+- **Max sequence length**: 2048 tokens
+- **Gradient checkpointing**: Enabled
+
+**Data-Driven Justification:**
+All hyperparameters were validated against dataset characteristics:
+- Batch size 4-8 optimal for 3,000 samples
+- 5 epochs sufficient for format learning without overfitting
+- Conservative LR (5e-5) for 13,710 vocabulary size
+- Max length 2048 covers 99%+ of prompts (median 1,357 chars)
+
+#### Speeds, Sizes, Times
+
+- **Training time**: ~2-3 hours on NVIDIA A100 GPU
+- **Model size**: ~3.5 GB (quantized base model + LoRA adapters)
+- **Trainable parameters**: ~1.5% of total model parameters
+- **Checkpoint frequency**: Every 50 steps
+- **Evaluation frequency**: Every 50 steps
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+- **Dataset**: Held-out test set from cleaned splits (299 samples)
+- **Split date**: November 13, 2025
+- **Distribution**: 100 chemicals, 99 diseases, 100 relationships
+- **Source**: ChemProt corpus (biomedical literature)
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+Evaluation disaggregated by task type:
+- **Chemical extraction**: Drug and chemical compound identification
+- **Disease extraction**: Disease and medical condition identification  
+- **Relationship extraction**: Chemical-disease interaction pairs
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+- **F1 Score** (primary): Harmonic mean of precision and recall
+- **Precision**: Fraction of predicted entities that are correct
+- **Recall**: Fraction of gold standard entities that were found
+- **Macro-average**: Equal weight to each task (chemicals, diseases, relationships)
 
-[More Information Needed]
+**Evaluation methodology:**
+- Enhanced filtering to reduce false positives
+- Normalized entity matching (lowercase, whitespace)
+- Hyphen preservation during normalization
+- Task-specific parsing (bullet lists for entities, pipe format for relationships)
 
 ### Results
 
-[More Information Needed]
+**Llama-3.2-3B Baseline** (before considering BioMistral):
+- **Overall F1**: 53.8% (macro-average across 3 tasks)
+- **Precision**: ~52-55%
+- **Recall**: ~54-56%
+
+**Key Insights:**
+- Model successfully learned pipe format for relationships (was 0% before fine-tuning)
+- Balanced performance across all three tasks
+- Format conversion (2,050 samples) successfully integrated during training
+- Clean data (99.8% retention) contributed to stable training
+
+**Baseline Comparison:**
+- Pre-training: 0% F1 on relationships (couldn't extract pairs)
+- Post-training: ~50% F1 on relationships (significant improvement)
+- Chemical/disease extraction improved from generic to domain-specific recognition
+
+### Planned Evaluation
+
+**Next Step**: Baseline evaluation of **BioMistral-7B-SLERP-AWQ** (quantized, no fine-tuning)
+- **Hypothesis**: Medical domain pre-training may outperform fine-tuned Llama-3.2-3B
+- **Target**: 70-80% F1 (medical domain models typically show 15-20 point advantage)
+- **Decision criteria**:
+  - If BioMistral ≥70% F1 → Deploy quantized model as-is
+  - If BioMistral 60-70% F1 → Fine-tune BioMistral (expected 75-85% F1)
+  - If BioMistral <60% F1 → Fine-tuning mandatory
 
-#### Summary
+**Tracking**: [GitHub Issue #3](https://github.com/albertoclemente/medical-ner-fine-tuning/issues/3)
 
+## Model Examination
 
+### Error Analysis
 
-## Model Examination [optional]
+Common error patterns observed:
+1. **False positives**: Generic medical terms (e.g., "pain", "treatment") occasionally extracted
+2. **False negatives**: Complex multi-word entities sometimes partially extracted
+3. **Boundary issues**: Entity boundaries unclear for nested or compound terms
+4. **Format sensitivity**: Deviations from training prompt format reduce performance
 
-<!-- Relevant interpretability work for the model goes here -->
+### Filtering Strategy
 
-[More Information Needed]
+Enhanced filtering applied during evaluation:
+- Blacklist of generic terms (drug, disease, chemical, etc.)
+- Entity type validation (disease markers shouldn't appear in chemical extractions)
+- Text grounding (only entities found in source text)
+- Minimum length threshold (≥3 characters)
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute).
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type**: NVIDIA A100 80GB GPU
+- **Hours used**: ~2.5 hours
+- **Cloud Provider**: RunPod / Cloud GPU provider
+- **Compute Region**: US (variable)
+- **Carbon Emitted**: ~0.5 kg CO2eq (estimated)
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+**Note**: LoRA fine-tuning is significantly more efficient than full model training, using only ~1.5% of trainable parameters and ~3 hours of compute time vs. days/weeks for full training.
 
-## Technical Specifications [optional]
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+**Base Architecture**: Llama-3.2-3B-Instruct (Meta AI)
+- **Parameters**: 3 billion (base model)
+- **Architecture**: Transformer decoder with grouped-query attention
+- **Context length**: 8,192 tokens
+- **Vocabulary**: 128,000 tokens (SentencePiece)
 
-### Compute Infrastructure
+**LoRA Adaptation**:
+- **Trainable parameters**: ~47 million (~1.5% of total)
+- **LoRA rank**: 16 (low-rank decomposition dimension)
+- **Adapter placement**: All attention and MLP projection layers
+- **Training objective**: Next-token prediction (causal language modeling)
 
-[More Information Needed]
+### Compute Infrastructure
 
 #### Hardware
 
-[More Information Needed]
+- **Training**: NVIDIA A100 80GB GPU
+- **Memory**: 80GB VRAM (4-bit quantization reduces to ~7GB usage)
+- **CPU**: High-memory instance (for data preprocessing)
 
 #### Software
 
-[More Information Needed]
+- **Framework**: Hugging Face Transformers 4.x
+- **Training**: Hugging Face Trainer with PEFT (Parameter-Efficient Fine-Tuning)
+- **Quantization**: BitsAndBytes (4-bit NF4 quantization)
+- **Monitoring**: Weights & Biases
+- **Python**: 3.10+
+- **PyTorch**: 2.x with CUDA 12.x
+- **Key libraries**: 
+  - `transformers` (model loading, training)
+  - `peft` (LoRA implementation)
+  - `bitsandbytes` (quantization)
+  - `accelerate` (distributed training)
+  - `datasets` (data loading)
+  - `wandb` (experiment tracking)
 
-## Citation [optional]
+## Citation
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+If you use this model in your research, please cite:
 
 **BibTeX:**
 
-[More Information Needed]
+```bibtex
+@misc{clemente2025medical-ner-lora,
+  author = {Clemente, Alberto},
+  title = {Llama-3.2-3B Medical NER with LoRA},
+  year = {2025},
+  publisher = {Hugging Face},
+  journal = {Hugging Face Model Hub},
+  howpublished = {\url{https://huggingface.co/albyos/llama3-medical-ner-lora}},
+}
+```
 
 **APA:**
 
-[More Information Needed]
+Clemente, A. (2025). *Llama-3.2-3B Medical NER with LoRA* [Computer software]. Hugging Face. https://huggingface.co/albyos/llama3-medical-ner-lora
 
-## Glossary [optional]
+## Glossary
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+- **NER (Named Entity Recognition)**: Task of identifying and classifying named entities in text
+- **LoRA (Low-Rank Adaptation)**: Parameter-efficient fine-tuning method that adds trainable low-rank matrices to model layers
+- **ChemProt**: Chemical-protein interaction corpus from biomedical literature
+- **Stratified splitting**: Data splitting that preserves class distribution across splits
+- **Quantization**: Reducing model precision (e.g., 32-bit → 4-bit) to save memory
+- **Macro-average**: Averaging metrics across classes with equal weight (vs. micro-average)
+- **Pipe format**: Relationship representation as `"entity1 | entity2"` (used for chemical-disease pairs)
 
-[More Information Needed]
+## More Information
 
-## More Information [optional]
+**Project Documentation**:
+- [Quick Start Guide](docs/QUICK_START.md)
+- [Fine-Tuning Plan](docs/FINE_TUNING_PLAN.md)
+- [Three-Way Split Guide](docs/THREE_WAY_SPLIT_GUIDE.md)
+- [Checkpoint Naming Strategy](docs/CHECKPOINT_NAMING.md)
+- [Implementation Summary](docs/IMPLEMENTATION_SUMMARY.md)
+- [Validation Strategy](docs/VALIDATION_STRATEGY.md)
 
-[More Information Needed]
+**Related Work**:
+- Base Model: [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
+- Alternative: [BioMistral-7B-SLERP](https://huggingface.co/BioMistral/BioMistral-7B-SLERP) (medical domain pre-trained)
+- Dataset Source: [ChemProt Corpus](https://biocreative.bioinformatics.udel.edu/news/corpora/chemprot-corpus-biocreative-vi/)
 
-## Model Card Authors [optional]
+**GitHub Issues**:
+- [Issue #2: Retrain with BioMistral-7B-SLERP](https://github.com/albertoclemente/medical-ner-fine-tuning/issues/2) (Closed)
+- [Issue #3: Baseline Evaluation - BioMistral-7B-SLERP-AWQ](https://github.com/albertoclemente/medical-ner-fine-tuning/issues/3) (Open)
 
-[More Information Needed]
+## Model Card Authors
+
+- Alberto Clemente (@albyos)
 
 ## Model Card Contact
 
-[More Information Needed]
-### Framework versions
+- **GitHub**: https://github.com/albertoclemente/medical-ner-fine-tuning
+- **Issues**: https://github.com/albertoclemente/medical-ner-fine-tuning/issues
+
+---
+
+### Framework Versions
+
+- **PEFT**: 0.17.1+
+- **Transformers**: 4.40.0+
+- **PyTorch**: 2.2.0+
+- **BitsAndBytes**: 0.42.0+
+- **Accelerate**: 0.27.0+
+- **Datasets**: 2.18.0+
+- **Tokenizers**: 0.19.0+
 
-- PEFT 0.17.1
+**Last Updated**: November 15, 2025