ReLU Nets Double as Hash Tables

Interesting idea, but the strongest version here is a modeling lens, not a new architecture yet. The discussion is valuable because it connects ReLU gating, sparse activation patterns, and associative memory into one mental model that fits recent work on neural networks as hash encoders.

• –ReLU activations already partition input space into many linear regions, so the “hash-like” intuition is not hand-wavy
• –The next-layer matrix can be read as context-selected weights, which makes the system resemble fast weights or associative memory more than a standard feedforward block
• –Width matters: if only a tiny fraction of gating patterns ever appear in practice, the network can behave like a sparse codebook over contexts
• –The main gap is formalization and training dynamics; once you have many synthetic matrices sharing weights, identifiability and capacity become the real problems
• –This is closer to a theory thread than a deployable method, but it could still influence how people think about memorization and modularity in deep nets