Parallel Decoder Transformer (PDT) adapters for GPT-OSS-20B
This repository contains PDT adapter/head weights trained against the GPT-OSS-20B trunk, plus minimal training artifacts.
Abstract (arXiv)
Autoregressive decoding in Large Language Models (LLMs) is inherently sequential, creating a latency bottleneck that scales linearly with output length. While "Decomposition-and-Fill" methods like Skeleton-of-Thought attempt to parallelize generation via external orchestration, they suffer from coherence drift due to the lack of cross-stream communication. In this work, we introduce the Parallel Decoder Transformer (PDT), a parameter-efficient architecture that embeds coordination primitives directly into the inference process of a frozen pre-trained model. Instead of retraining the base model, PDT injects lightweight Speculative Note Conditioning (SNC) adapters that allow parallel decoding streams to synchronize via a shared, dynamic latent space. We formulate coordination as a speculative consensus problem, where sibling streams broadcast semantic "notes" to a global bus, gated by a learned verification head. We validate our approach on a 50,000-step curriculum using a frozen 20B-parameter backbone. Our results demonstrate that PDT achieves effective self-correction, reaching 77.8% precision in coverage prediction and recovering approximate serial semantics without modifying the trunk weights. This establishes PDT as a scalable, efficient alternative to full model fine-tuning for structured parallel generation.
Example: PDT notes artifact (truncated)
This is a real sample from the dataset pipeline (survey_200141_ff0a0b4f.json), shown with list/string truncation to keep the model card readable.
{
"sample_id": "survey_200141_ff0a0b4f",
"domain": "survey",
"plan_path": "outputs/structured_plans/pdt_10k/survey/survey_200141_ff0a0b4f.json",
"sectional_independence": true,
"lag_delta": 1,
"note_cadence_M": 6,
"true_notes_example": {
"stream_id": "stream_1",
"ENT": [
{
"id": "E1",
"name": "Croatan",
"aliases": [
"Croatoan"
],
"type": "Ethnic Group",
"canonical": true
},
{
"id": "E2",
"name": "Dare County",
"aliases": [
"Alligator River",
"Croatan Sound",
"Roanoke Island",
"... <2 more items>"
],
"type": "Location",
"canonical": true
},
{
"id": "E3",
"name": "werowances",
"aliases": [
"chiefs"
],
"type": "Leadership Title",
"canonical": true
},
"... <4 more items>"
],
"FACT": [
{
"subj_id": "E1",
"predicate": "lived in",
"object": "coastal areas of what is now North Carolina",
"evidence_span": {
"start": 45,
"end": 87,
"text": "coastal areas of what is now North Carolina"
},
"certainty": 1.0
},
{
"subj_id": "E1",
"predicate": "might have been",
"object": "a branch of the larger Roanoke people or allied with them",
"evidence_span": {
"start": 92,
"end": 141,
"text": "a branch of the larger Roanoke people or allied with them"
},
"certainty": 0.8
},
{
"subj_id": "E2",
"predicate": "encompasses",
"object": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks",
"evidence_span": {
"start": 177,
"end": 265,
"text": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks"
},
"certainty": 1.0
},
"... <5 more items>"
],
"COVERAGE": [
{
"plan_item_id": "Define who the Croatan were, where they lived historically, and where related people live today.",
"status": "missing"
},
{
"plan_item_id": "Describe political leadership (werowances) and their responsibilities regarding wealth and decision-making.",
"status": "missing"
},
{
"plan_item_id": "Summarize core religious beliefs about a chief god, petty gods, immortality of the soul, heaven/Popogusso, and roles of priests and conjurors.",
"status": "missing"
},
"... <6 more items>"
]
},
"speculative_variant_example": {
"variant_id": "survey_200141_ff0a0b4f_variant_0",
"noise_config": {
"paraphrase_ratio": 0.15,
"drop_ratio": 0.05,
"hallucination_ratio": 0.05,
"shuffle_notes": true
},
"lag_delta": 1,
"notes_example": {
"stream_id": "stream_1",
"ENT": [
{
"id": "E1",
"name": "Croatan",
"aliases": [
"Croatoan",
"Croatian"
],
"type": "Ethnic Group",
"canonical": true
},
{
"id": "E2",
"name": "Dare County",
"aliases": [
"Alligator River",
"Croatan Sound",
"Roanoke Island",
"... <1 more items>"
],
"type": "Location",
"canonical": true
},
{
"id": "E3",
"name": "werowances",
"aliases": [
"chiefs",
"leaders"
],
"type": "Leadership Title",
"canonical": true
},
"... <1 more items>"
],
"FACT": [
{
"subj_id": "E1",
"predicate": "lived in",
"object": "coastal areas of what is now North Carolina",
"evidence_span": {
"start": 45,
"end": 87,
"text": "coastal areas of what is now North Carolina"
},
"certainty": 1.0
},
{
"subj_id": "E1",
"predicate": "might have been",
"object": "a branch of the larger Roanoke people or allied with them",
"evidence_span": {
"start": 92,
"end": 141,
"text": "a branch of the larger Roanoke people or allied with them"
},
"certainty": 0.8
},
{
"subj_id": "E2",
"predicate": "encompasses",
"object": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks",
"evidence_span": {
"start": 177,
"end": 265,
"text": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks"
},
"certainty": 1.0
},
"... <1 more items>"
],
"COVERAGE": [
{
"plan_item_id": "Define who the Croatan were, where they lived historically, and where related people live today.",
"status": "missing"
},
{
"plan_item_id": "Describe political leadership (werowances) and their responsibilities regarding wealth and decision-making.",
"status": "missing"
},
{
"plan_item_id": "Summarize core religious beliefs about a chief god, petty gods, immortality of the soul, heaven/Popogusso, and roles of priests and conjurors.",
"status": "missing"
},
"... <1 more items>"
]
}
},
"versioned_notes_snapshot_0": {
"snapshot_id": 0,
"source": "procedural_bus",
"lag_delta": 1,
"note_cadence_M": 6,
"ent_count": 9,
"fact_count": 10,
"notes_example": {
"stream_id": "stream_1",
"ENT": [
{
"id": "E1",
"name": "Croatan",
"aliases": [
"Croatoan"
],
"type": "Ethnic Group",
"canonical": true
},
{
"id": "E2",
"name": "Dare County",
"aliases": [
"Alligator River",
"Croatan Sound",
"Roanoke Island",
"... <1 more items>"
],
"type": "Location",
"canonical": true
},
{
"id": "E3",
"name": "werowances",
"aliases": [
"chiefs"
],
"type": "Leadership Title",
"canonical": true
},
"... <1 more items>"
],
"FACT": [
{
"subj_id": "E1",
"predicate": "lived in",
"object": "coastal areas of what is now North Carolina",
"evidence_span": {
"start": 45,
"end": 87,
"text": "coastal areas of what is now North Carolina"
},
"certainty": 1.0
},
{
"subj_id": "E1",
"predicate": "might have been",
"object": "a branch of the larger Roanoke people or allied with them",
"evidence_span": {
"start": 92,
"end": 141,
"text": "a branch of the larger Roanoke people or allied with them"
},
"certainty": 0.8
},
{
"subj_id": "E2",
"predicate": "encompasses",
"object": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks",
"evidence_span": {
"start": 177,
"end": 265,
"text": "the Alligator River, Croatan Sound, Roanoke Island, Ocracoke Island, and parts of the Outer Banks"
},
"certainty": 1.0
},
"... <1 more items>"
],
"COVERAGE": [
{
"plan_item_id": "Define who the Croatan were, where they lived historically, and where related people live today.",
"status": "missing"
},
{
"plan_item_id": "Describe political leadership (werowances) and their responsibilities regarding wealth and decision-making.",
"status": "missing"
},
{
"plan_item_id": "Summarize core religious beliefs about a chief god, petty gods, immortality of the soul, heaven/Popogusso, and roles of priests and conjurors.",
"status": "missing"
},
"... <1 more items>"
]
}
},
"rollback": {
"triggered": false,
"l_tokens": 0,
"events": []
}
}
To reproduce this view locally:
uv run python scripts/pretty_notes_artifact.py survey_200141_ff0a0b4f.json
How to use
- Install the reference implementation (runtime + scripts):
https://github.com/logan-robbins/parallel-decoder-transformer
- Download the base trunk model (
openai/gpt-oss-20b) via Hugging Face (or provide a local path). - Download the adapter checkpoint from this repo and point
configs/gpt_oss_transfer_production.yaml(or CLI flags) at it.
Artifacts (public GCS)
The complete training artifacts and dataset archives are mirrored publicly in GCS:
- Bucket root:
https://storage.googleapis.com/parallel-decoder-transformer/ - Upload manifest (full listing):
https://storage.googleapis.com/parallel-decoder-transformer/UPLOAD_MANIFEST.md - Training checkpoints:
https://storage.googleapis.com/parallel-decoder-transformer/checkpoints/gpt-oss-8xH100-50000steps/ - Dataset archives:
https://storage.googleapis.com/parallel-decoder-transformer/data/archives/
Training logs (Weights & Biases)
- WandB run:
https://wandb.ai/ljrweb-self/parallel-decoder-transformer/runs/fmuea63a
Why the dataset is structured this way
PDT is trained on streamed, structured supervision produced by a 5-stage pipeline:
- Stage 2 (Plans): a 3-stream decomposition plan is generated for each document.
- Stage 3 (Notes): we generate true notes (teacher) and speculative notes (student input) in a consistent schema:
ENT: entity table (stable ids)FACT: grounded tuples withevidence_spanCOVERAGE: plan-item status targets (covered|partial|missing)versioned_notes: lagged, versioned snapshots mirroring the Dynamic Notes Bus semantics
- Stage 5 (KD Export): these artifacts are converted into
kd_*.jsonlwhere each line is a stream-level training example.
This layout is required to support the teacher→student curriculum described in the training guide:
- Stage 0: planner/notes-head bootstrap (trunk frozen)
- Stage 1: stream adapters + SNC cross-attention bootstrap (speculation frozen; teacher notes forced)
- Stage 2: enable speculation + notes-bus usage (teacher-heavy mixing)
- Stage 3: train agreement + coverage heads for self-correction/rollback behavior (still trunk frozen)
Citation
@misc{robbins2025pdt,
title={Parallel Decoder Transformer: Model-Internal Parallel Decoding with Speculative Invariance via Note Conditioning},
author={Robbins, Logan},
year={2025},
eprint={2512.10054},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2512.10054}
}
What’s included
pdt_adapters.*: trainable adapter/head weights (no trunk weights unless you intentionally uploaded them)training_report.json,train_run_stages.json,train_manifest.json,agreement_thresholds.json
License
- This repo (adapters + artifacts): MIT.
- Base model:
openai/gpt-oss-20bis licensed under Apache-2.0 on Hugging Face (also see itsUSAGE_POLICYthere). - Reference implementation: MIT at
https://github.com/logan-robbins/parallel-decoder-transformer.
Model tree for loganrobbins/parallel-decoder-transformer
Base model
openai/gpt-oss-20b