smol-training-playbook

Building on CPU Upgrade

App Files Files Community

Fix RoPE comments: smaller k is faster oscillation

by 6S-bobby - opened 8 days ago

base: refs/heads/main

←

from: refs/pr/5

Discussion Files changed

-3

6S-bobby

8 days ago

Pull Request opened with the huggingface_hub Python library

(Minor/nit) Docs: correct RoPE frequency note, smaller k -> faster oscillations674cf3d7

6S-bobby

8 days ago

Title: (Minor/nit) Fix RoPE comments: smaller k is faster oscillation

Large k means large freq, which means small inv_freq, which means small theta, so large k = slow oscillations; the opposite happens for small k. Not particularly important in the grand scheme of things especially when most code always vectorizes this, but may help the readers that are working through the formula on this part.

P.S. love the post/textbook!

6S-bobby changed pull request status to open 8 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment