Attention

How Binary Jumps and Local Connections Can Make Attention Faster

A popular-science introduction to sparse attention, binary information propagation, book and code experiments, and the gap between theoretical savings and GPU speed.