NanoGPT Training Record: Technical Optimizations & Speed

Overview of the NanoGPT Training Record

NanoGPT achieved a new benchmark in language model training speed, completing a full training section in 39.914 seconds with a standard deviation of 0.120 seconds, as recorded in pull request 360. This result is supported by multiple sources, including Hacker News, Hyperstition, and LinkedIn. The previous NanoGPT training record was 73.889 seconds (also referred to as 73.8 seconds). The reduction of 33.98 seconds represents a 46.0 percent decrease in training time and a 1.85x speedup compared to the earlier best. Deven Pietrzak created the NanoGPT training pull request 360. According to Pietrzak, four separate sources individually reviewed the method and reproduced the record on their hardware. Pietrzak also reported testing 18 runs, all of which were under the loss bar with a probability of less than 1e-9.

Hyperstition states that by percentage reduction in training time, this improvement is greater than the previous 45 world records combined. Pietrzak had previously submitted a world record of 64.7 seconds, which he withdrew after achieving the new 39.9-second result. The process of achieving the new record took about five days once additional H100 GPUs became available, according to Pietrzak.

The NanoGPT Benchmark and Speedrun Definition

NanoGPT is described as an open-source benchmark created by Keller Jordan to measure the frontier of pretraining efficiency, specifically testing how efficiently a large language model can be trained to a certain loss bar. The NanoGPT speedrun measures how fast NanoGPT, a 124-million-parameter language model, can reach a fixed validation loss of 3.28 on the FineWeb dataset using eight H100 GPUs. The vocabulary used in the model has 50,304 elements. Approximately one third of the arithmetic during training is accounted for by vocabulary matrix multiplications.

According to Pietrzak, the NanoGPT benchmark started at 45 minutes in 2024 and has been broken 89 times since. Under his new method, training GPT-2 costs only 18 cents of actual compute.

Key Optimizations and Ablation Table Results

The speedup in NanoGPT’s training is attributed to several architectural and algorithmic optimizations, each contributing measurable gains as detailed in the ablation table for pull request 360:

Sampled-softmax training loss: Accounts for 8.40 net seconds in the ablation table. Sampled softmax reduces the number of vocabulary elements the model has to score in both forward and backward passes for most of the run, saving over 8 seconds. The final sampled softmax schedule uses 10,240, then 14,336, then 24,576 candidates, followed by all 50,304 entries for the final 87 steps.
Hashed n-gram table embeddings: Account for 7.05 net seconds. A much larger sparse n-gram table gives the model more capacity to use local token patterns. The final configuration has 84.6 million rows across bigram and trigram channels, compared to the original 377,280. Each GPU owns one eighth of the bigram and trigram table weights and requests only the rows that need to be updated in upcoming batches. Removing the full bigram table replica saved 12.2 GiB per GPU.
ANVIL optimizer: Accounts for 5.08 net seconds. ANVIL stands for Averaged Normalized Velocity with Isotropic Lanes and was released as the flagship optimizer with the previous record. ANVIL maintains two streams of momentum, fast and slow, and applies six polynomial maps to flatten the update’s singular spectrum. The optimizer features a whitening envelope significantly tighter than standard Newton-Schulz using coefficients derived from scratch, dual fast and slow momentum lanes, and a row normalization at fixed Frobenius norm. ANVIL II is an algorithmic improvement to the optimizer and achieves an improved validation loss compared to Muon.
Full-stack FP8 quantization: Accounts for 4.96 net seconds in the ablation table. Narrow QK-attention and fp8 quantization through the backward pass contributed to Pietrzak’s record win.
Mixed-width attention: Accounts for 3.82 net seconds.
Training-step graph capture: Accounts for 2.99 net seconds.
Sparse gradient communications and data loader: Account for 1.20 net seconds.
MUDDformer residual mix: Accounts for 0.48 net seconds.

Each of these optimizations was measured by removing the component from the finished trainer and observing the impact on training time.

Technical Architecture and Implementation Details

The record-setting NanoGPT run combined several technical advances:

A much larger sparse n-gram table was used, giving the model more capacity to use local token patterns. The final configuration has 84.6 million rows across bigram and trigram channels, compared to the original 377,280. Each GPU owns one eighth of the bigram and trigram table weights and requests only the rows that need to be updated in upcoming batches. Removing the full table replica saved 12.2 GiB per GPU. Pietrzak stated that his implementation extends the Bigram and Trigram hashed embedding table.
Sampled softmax was implemented to reduce the number of vocabulary elements scored during training. For most of the run, only a subset of candidates is scored, and only for the final 87 steps is the full vocabulary used. This approach saves over 8 seconds by scoring a subset rather than every token in the vocabulary at training time.
ANVIL II, an improvement over the previous Muon optimizer, maintains two streams of momentum (fast and slow) and applies six polynomial maps to flatten the update’s singular spectrum. Keeping a running average of weights over steps can be better than using the last iterate because the readout, matrix banks, and value embeddings update at different times. For the readout, 35 percent of the final weights are blended with 65 percent of the average. Hyperstition attributes the performance gains to the combination of ANVIL II, sampled softmax, and a larger sparse n-gram model.
Additional optimizations include narrow QK-attention and fp8 quantization through the backward pass, mixed-width attention, training-step graph capture, sparse gradient communications, and the MUDDformer residual mix.

Validation and Reproducibility

The performance of pull request 360 was validated using several key metrics. The final validation cross-entropy achieved was 3.27731 with a standard deviation of 0.00099. Pietrzak reported testing 18 runs, all of which were under the loss bar with a probability of less than 1e-9. Four separate sources individually reviewed the method and reproduced the record on their hardware, according to Pietrzak.

System and Market Context, Disclosure, and Proprietary Methods

Hyperstition states that for reasons proprietary to Hyperstition, it redacted methods that scale to the frontier from its public record, apart from ANVIL II. According to Hyperstition, the percentage reduction in training time achieved by this pull request is greater than the previous 45 world records combined. Pietrzak reports that the NanoGPT benchmark started at 45 minutes in 2024 and has been broken 89 times since. Under the new method, training GPT-2 costs only 18 cents of actual compute, according to Pietrzak. The implementation details and code for pull request 360 are publicly accessible, allowing others in the community to reproduce and build upon these results.