Timezone: »

 
Poster
Looped Transformers as Programmable Computers
Angeliki Giannou · Shashank Rajput · Jy-yong Sohn · Kangwook Lee · Jason Lee · Dimitris Papailiopoulos

Tue Jul 25 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #733
in Poster Session 2 »

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including lexicographic operations, non-linear functions, function calls, program counters, and conditional branches. Using this framework, we emulate a computer using a simple instruction-set architecture, which allows us to map iterative algorithms to programs that can be executed by a constant depth looped transformer network. We show how a single frozen transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and even a full backpropagation, in-context learning algorithm. Our findings reveal the potential of transformer networks as programmable compute units and offer insight into the mechanics of attention.

[ PDF] 

Author Information

Angeliki Giannou (University of Wisconsin - Madison)
Shashank Rajput (University of Wisconsin-Madison)
Jy-yong Sohn (Yonsei University)
Kangwook Lee (KAIST)
Jason Lee (Princeton University)
Dimitris Papailiopoulos (University of Wisconsin-Madison)

More from the Same Authors