syed-aliredha/creativity-iti-llama31-8b

Creativity ITI for LLaMA 3.1 8B Instruct

🎯 Model Description

This repository contains Inference-Time Intervention (ITI) components for enhancing creativity in code generation with LLaMA 3.1 8B Instruct.

ITI modifies model activations during inference to steer behavior without retraining - think of it as "creativity steering" for AI code generation.

📊 Key Results

• 68.8% test accuracy in creativity detection
• Optimal α=0.4 intervention strength
• 48 attention heads identified for creativity
• Trained on 1058 coding problems from NeoCoder

🚀 Quick Start

Installation

pip install transformers torch numpy

Basic Usage

import pickle
import json
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load ITI components
with open('iti_config.json', 'r') as f:
    config = json.load(f)

with open('iti_components.pkl', 'rb') as f:
    components = pickle.load(f)

# Initialize model
model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    torch_dtype=torch.float16, 
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Apply ITI with α=0.4
alpha = config['metadata']['alpha']
directions = components['directions']
top_heads = components['top_heads']

📈 Performance Metrics

Metric	Value
Training Samples	48 (balanced)
Validation Accuracy	62.5%
Test Accuracy	68.8%
Optimal Alpha (α)	0.4
Intervention Heads	48
Best Single Layer	Layer 3
Top Head	Layer 17, Head 21 (AUC=0.734)

🔬 Technical Details

How ITI Works

1. Probe Training: Linear probes identify which attention heads correlate with creative code
2. Direction Finding: Calculate "creativity directions" in activation space
3. Runtime Intervention: During inference, shift activations by α=0.4 in creative direction
4. Result: Model generates more innovative code solutions

File Contents

• iti_config.json: Configuration, metadata, and intervention directions
• iti_components.pkl: Binary format with top heads and directions
• README.md: This documentation

💡 Example Output Comparison

Problem: "Check if a number is prime"

Without ITI (Baseline):

def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, n):
        if n % i == 0:
            return False
    return True

With ITI (α=0.4):

def is_prime(n):
    return n > 1 and all(n % i for i in range(2, int(n**0.5) + 1))

The ITI version is more concise and uses advanced techniques (generator expression, all()).

📚 Dataset

Trained on the NeoCoder dataset:

• 1058 competitive programming problems
• Creativity labeled based on novel technique usage
• Only 4.3% of solutions were labeled as truly creative
• Balanced training set: 40 creative + 40 non-creative samples

🏗️ Implementation Details

Key Components

1. Top Heads: 48 attention heads most predictive of creativity
2. Intervention Layers: Distributed across layers 3-31
3. Direction Vectors: 128-dimensional vectors per head
4. Activation Shift: Applied to last token during generation

📖 Citation

If you use this work, please cite:

@article{li2023inference,
  title={Inference-Time Intervention: Eliciting Truthful Answers from a Language Model},
  author={Li, Kenneth and others},
  journal={NeurIPS},
  year={2023}
}

🙏 Acknowledgments

• Original ITI paper authors (Li et al., 2023)
• NeoCoder dataset creators
• Meta AI for LLaMA 3.1
• NSCC Singapore for compute resources

📜 License

Apache 2.0 - See LICENSE file for details

⚠️ Limitations

• Intervention may occasionally produce syntactically invalid code
• Effectiveness varies by problem complexity
• Requires LLaMA 3.1 8B model (not included)

🔗 Links

• Original ITI Paper
• Base Model: LLaMA 3.1 8B