Retention Networks.

01

The question

Almost every large language model today is a Transformer. To write each new word, a Transformer looks back at every word before it, so it keeps notes on all of them, and each step gets more expensive as the text gets longer.

In 2023, researchers at Microsoft and Tsinghua University proposed the Retentive Network, or RetNet. It trains in parallel like a Transformer, but when it writes, it carries one fixed-size summary forward instead of rereading everything, so each new word costs the same however long the text gets.

Our research group at BYU asked a plain question: at the same size, trained the same way on the same data, does RetNet keep up with a Transformer?

02

What we built

We built one pipeline that both designs go through, so the only difference between them is the architecture. It downloads the C4 web-text dataset from Hugging Face, splits it, trains a tokenizer, tokenizes the splits in parallel and trains each model, all as jobs on BYU's supercomputer.

Both models come from Microsoft's TorchScale library, wrapped so they also work with Hugging Face tools. We scored them on standard benchmarks such as HellaSwag and WinoGrande with EleutherAI's evaluation harness, timed how fast each one writes, and logged the energy every run used.

03

My part: a fair grid

One comparison at one setting can mislead: a learning rate that suits one design can hold the other back. I made the project's first commit and built the first version of the grid search, which trains both models at every combination of settings and writes each pair's results side by side to one file.

The full grid covered 72 combinations for each design: 3 learning rates, 3 embedding sizes, 2 feed-forward sizes, 2 head counts and 2 sequence lengths. Tap any square to see its settings.