Upload Creativity ITI components for LLaMA 3.1 8B

README.md

@@ -0,0 +1,171 @@
+---
+license: apache-2.0
+tags:
+- inference-time-intervention
+- code-generation
+- creativity
+- llama
+- pytorch
+datasets:
+- neocoder
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
+---
+
+# Creativity ITI for LLaMA 3.1 8B Instruct
+
+## 🎯 Model Description
+
+This repository contains **Inference-Time Intervention (ITI)** components for enhancing creativity in code generation with LLaMA 3.1 8B Instruct.
+
+ITI modifies model activations during inference to steer behavior without retraining - think of it as "creativity steering" for AI code generation.
+
+## 📊 Key Results
+
+- **68.8% test accuracy** in creativity detection
+- **Optimal α=0.4** intervention strength
+- **48 attention heads** identified for creativity
+- Trained on **1058 coding problems** from NeoCoder
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install transformers torch numpy
+```
+
+### Basic Usage
+
+```python
+import pickle
+import json
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Load ITI components
+with open('iti_config.json', 'r') as f:
+    config = json.load(f)
+
+with open('iti_components.pkl', 'rb') as f:
+    components = pickle.load(f)
+
+# Initialize model
+model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
+model = AutoModelForCausalLM.from_pretrained(
+    model_name, 
+    torch_dtype=torch.float16, 
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Apply ITI with α=0.4
+alpha = config['metadata']['alpha']
+directions = components['directions']
+top_heads = components['top_heads']
+```
+
+## 📈 Performance Metrics
+
+| Metric | Value |
+|--------|-------|
+| Training Samples | 48 (balanced) |
+| Validation Accuracy | 62.5% |
+| Test Accuracy | 68.8% |
+| Optimal Alpha (α) | 0.4 |
+| Intervention Heads | 48 |
+| Best Single Layer | Layer 3 |
+| Top Head | Layer 17, Head 21 (AUC=0.734) |
+
+## 🔬 Technical Details
+
+### How ITI Works
+
+1. **Probe Training**: Linear probes identify which attention heads correlate with creative code
+2. **Direction Finding**: Calculate "creativity directions" in activation space
+3. **Runtime Intervention**: During inference, shift activations by α=0.4 in creative direction
+4. **Result**: Model generates more innovative code solutions
+
+### File Contents
+
+- `iti_config.json`: Configuration, metadata, and intervention directions
+- `iti_components.pkl`: Binary format with top heads and directions
+- `README.md`: This documentation
+
+## 💡 Example Output Comparison
+
+**Problem**: "Check if a number is prime"
+
+**Without ITI (Baseline):**
+```python
+def is_prime(n):
+    if n <= 1:
+        return False
+    for i in range(2, n):
+        if n % i == 0:
+            return False
+    return True
+```
+
+**With ITI (α=0.4):**
+```python
+def is_prime(n):
+    return n > 1 and all(n % i for i in range(2, int(n**0.5) + 1))
+```
+
+The ITI version is more concise and uses advanced techniques (generator expression, all()).
+
+## 📚 Dataset
+
+Trained on the **NeoCoder** dataset:
+- 1058 competitive programming problems
+- Creativity labeled based on novel technique usage
+- Only 4.3% of solutions were labeled as truly creative
+- Balanced training set: 40 creative + 40 non-creative samples
+
+## 🏗️ Implementation Details
+
+### Key Components
+
+1. **Top Heads**: 48 attention heads most predictive of creativity
+2. **Intervention Layers**: Distributed across layers 3-31
+3. **Direction Vectors**: 128-dimensional vectors per head
+4. **Activation Shift**: Applied to last token during generation
+
+## 📖 Citation
+
+If you use this work, please cite:
+
+```bibtex
+@article{li2023inference,
+  title={Inference-Time Intervention: Eliciting Truthful Answers from a Language Model},
+  author={Li, Kenneth and others},
+  journal={NeurIPS},
+  year={2023}
+}
+```
+
+## 🙏 Acknowledgments
+
+- Original ITI paper authors (Li et al., 2023)
+- NeoCoder dataset creators
+- Meta AI for LLaMA 3.1
+- NSCC Singapore for compute resources
+
+## 📜 License
+
+Apache 2.0 - See LICENSE file for details
+
+## ⚠️ Limitations
+
+- Intervention may occasionally produce syntactically invalid code
+- Effectiveness varies by problem complexity
+- Requires LLaMA 3.1 8B model (not included)
+
+## 🔗 Links
+
+- [Original ITI Paper](https://arxiv.org/abs/2306.03341)
+- [Base Model: LLaMA 3.1 8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)

iti_components.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:560cbf82915fc4ce490b08f24dbe06e60f0ecfe915b3bc93431771d0ada978cc
+size 17539

requirements.txt

@@ -0,0 +1,4 @@
+transformers>=4.30.0
+torch>=2.0.0
+numpy>=1.20.0
+huggingface-hub>=0.16.0