Skip to content

Commit 935d198

Browse files
committed
use permalink for llm.c ref
1 parent 1fdf3a4 commit 935d198

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

torchao/prototype/low_bit_optim/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ Note that our optimizer step calculations are always done in FP32 to ensure accu
6565
- In stochastic rounding, we will round up with the probability of `(x - round_down(x)) / (round_up(x) - round_down(x))`, and round down otherwise.
6666
- It follows that successive weight update with stochastic rounding will correctly approximate high-precision weight update.
6767
- Since BF16 is simply a truncation of FP32, there is an efficient implementation for FP32->BF16 stochastic rounding (the same is not true for FP32->FP16).
68-
- More detailed discussion can be found at https://arxiv.org/abs/2010.06192. [llm.c](https://github.com/karpathy/llm.c/blob/master/llmc/adamw.cuh#L43) also implements this approach.
68+
- More detailed discussion can be found at https://arxiv.org/abs/2010.06192. [llm.c](https://github.com/karpathy/llm.c/blob/7ecd8906afe6ed7a2b2cdb731c042f26d525b820/llmc/adamw.cuh#L43) also implements this approach.
6969

7070
```python
7171
# a clone of torch.optim.AdamW with extra features

0 commit comments

Comments
 (0)