Gated Linear Unit

☆ Save On Wikipedia ↗

In neural networks, the gating mechanism is an architectural motif for controlling the flow of activation and gradient signals. They are most prominently used in recurrent neural networks (RNNs), but have also found applications in other architectures.

RNNs

Gating mechanisms are the centerpiece of long short-term memory (LSTM).[1] They were proposed to mitigate the vanishing gradient problem often encountered by regular RNNs.

An LSTM unit contains three gates:

  • An input gate, which controls the flow of new information into the memory cell
  • A forget gate, which controls how much information is retained from the previous time step
  • An output gate, which controls how much information is passed to the next layer.

The equations for LSTM are:[2]

I t = σ ( X t W x i + H t − 1 W h i + b i ) F t = σ ( X t W x f + H t − 1 W h f + b f ) O t = σ ( X t W x o + H t − 1 W h o + b o ) C ~ t = tanh ⁡ ( X t W x c + H t − 1 W h c + b c ) C t = F t ⊙ C t − 1 + I t ⊙ C ~ t H t = O t ⊙ tanh ⁡ ( C t ) {\displaystyle {\begin{aligned}\mathbf {I} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xi}+\mathbf {H} _{t-1}\mathbf {W} _{hi}+\mathbf {b} _{i})\\\mathbf {F} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xf}+\mathbf {H} _{t-1}\mathbf {W} _{hf}+\mathbf {b} _{f})\\\mathbf {O} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xo}+\mathbf {H} _{t-1}\mathbf {W} _{ho}+\mathbf {b} _{o})\\{\tilde {\mathbf {C} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xc}+\mathbf {H} _{t-1}\mathbf {W} _{hc}+\mathbf {b} _{c})\\\mathbf {C} _{t}&=\mathbf {F} _{t}\odot \mathbf {C} _{t-1}+\mathbf {I} _{t}\odot {\tilde {\mathbf {C} }}_{t}\\\mathbf {H} _{t}&=\mathbf {O} _{t}\odot \tanh(\mathbf {C} _{t})\end{aligned}}} {\displaystyle {\begin{aligned}\mathbf {I} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xi}+\mathbf {H} _{t-1}\mathbf {W} _{hi}+\mathbf {b} _{i})\\\mathbf {F} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xf}+\mathbf {H} _{t-1}\mathbf {W} _{hf}+\mathbf {b} _{f})\\\mathbf {O} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xo}+\mathbf {H} _{t-1}\mathbf {W} _{ho}+\mathbf {b} _{o})\\{\tilde {\mathbf {C} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xc}+\mathbf {H} _{t-1}\mathbf {W} _{hc}+\mathbf {b} _{c})\\\mathbf {C} _{t}&=\mathbf {F} _{t}\odot \mathbf {C} _{t-1}+\mathbf {I} _{t}\odot {\tilde {\mathbf {C} }}_{t}\\\mathbf {H} _{t}&=\mathbf {O} _{t}\odot \tanh(\mathbf {C} _{t})\end{aligned}}}

Here, ⊙ {\displaystyle \odot } {\displaystyle \odot } represents elementwise multiplication.

The gated recurrent unit (GRU) simplifies the LSTM.[3] Compared to the LSTM, the GRU has just two gates: a reset gate and an update gate. GRU also merges the cell state and hidden state. The reset gate roughly corresponds to the forget gate, and the update gate roughly corresponds to the input gate. The output gate is removed.

There are several variants of GRU. One particular variant has these equations:[4]

R t = σ ( X t W x r + H t − 1 W h r + b r ) Z t = σ ( X t W x z + H t − 1 W h z + b z ) H ~ t = tanh ⁡ ( X t W x h + ( R t ⊙ H t − 1 ) W h h + b h ) H t = Z t ⊙ H t − 1 + ( 1 − Z t ) ⊙ H ~ t {\displaystyle {\begin{aligned}\mathbf {R} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xr}+\mathbf {H} _{t-1}\mathbf {W} _{hr}+\mathbf {b} _{r})\\\mathbf {Z} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xz}+\mathbf {H} _{t-1}\mathbf {W} _{hz}+\mathbf {b} _{z})\\{\tilde {\mathbf {H} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xh}+(\mathbf {R} _{t}\odot \mathbf {H} _{t-1})\mathbf {W} _{hh}+\mathbf {b} _{h})\\\mathbf {H} _{t}&=\mathbf {Z} _{t}\odot \mathbf {H} _{t-1}+(1-\mathbf {Z} _{t})\odot {\tilde {\mathbf {H} }}_{t}\end{aligned}}} {\displaystyle {\begin{aligned}\mathbf {R} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xr}+\mathbf {H} _{t-1}\mathbf {W} _{hr}+\mathbf {b} _{r})\\\mathbf {Z} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xz}+\mathbf {H} _{t-1}\mathbf {W} _{hz}+\mathbf {b} _{z})\\{\tilde {\mathbf {H} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xh}+(\mathbf {R} _{t}\odot \mathbf {H} _{t-1})\mathbf {W} _{hh}+\mathbf {b} _{h})\\\mathbf {H} _{t}&=\mathbf {Z} _{t}\odot \mathbf {H} _{t-1}+(1-\mathbf {Z} _{t})\odot {\tilde {\mathbf {H} }}_{t}\end{aligned}}}

Gated Linear Unit

Gated Linear Units (GLUs)[5] adapt the gating mechanism for use in feedforward neural networks, often within transformer-based architectures. They are defined as: G L U ( a , b ) = a ⊙ σ ( b ) {\displaystyle \mathrm {GLU} (a,b)=a\odot \sigma (b)} {\displaystyle \mathrm {GLU} (a,b)=a\odot \sigma (b)}where a , b {\displaystyle a,b} {\displaystyle a,b} are the first and second inputs, respectively. σ {\displaystyle \sigma } {\displaystyle \sigma } represents the sigmoid activation function.

Replacing σ {\displaystyle \sigma } {\displaystyle \sigma } with other activation functions leads to variants of GLU: R e G L U ( a , b ) = a ⊙ ReLU ( b ) G E G L U ( a , b ) = a ⊙ GELU ( b ) S w i G L U ( a , b , β ) = a ⊙ Swish β ( b ) {\displaystyle {\begin{aligned}\mathrm {ReGLU} (a,b)&=a\odot {\text{ReLU}}(b)\\\mathrm {GEGLU} (a,b)&=a\odot {\text{GELU}}(b)\\\mathrm {SwiGLU} (a,b,\beta )&=a\odot {\text{Swish}}_{\beta }(b)\end{aligned}}} {\displaystyle {\begin{aligned}\mathrm {ReGLU} (a,b)&=a\odot {\text{ReLU}}(b)\\\mathrm {GEGLU} (a,b)&=a\odot {\text{GELU}}(b)\\\mathrm {SwiGLU} (a,b,\beta )&=a\odot {\text{Swish}}_{\beta }(b)\end{aligned}}}where ReLU, GELU, and Swish are different activation functions, applied componentwise.

Adding weights and biases, this results in the following functions:[6] GLU ⁡ ( x , W , V , b , c ) = σ ( x W + b ) ⊙ ( x V + c ) Bilinear ⁡ ( x , W , V , b , c ) = ( x W + b ) ⊙ ( x V + c ) ReGLU ⁡ ( x , W , V , b , c ) = max ( 0 , x W + b ) ⊙ ( x V + c ) GEGLU ⁡ ( x , W , V , b , c ) = GELU ⁡ ( x W + b ) ⊙ ( x V + c ) SwiGLU ⁡ ( x , W , V , b , c , β ) = Swish β ⁡ ( x W + b ) ⊙ ( x V + c ) {\displaystyle {\begin{aligned}\operatorname {GLU} (x,W,V,b,c)&=\sigma (xW+b)\odot (xV+c)\\\operatorname {Bilinear} (x,W,V,b,c)&=(xW+b)\odot (xV+c)\\\operatorname {ReGLU} (x,W,V,b,c)&=\max(0,xW+b)\odot (xV+c)\\\operatorname {GEGLU} (x,W,V,b,c)&=\operatorname {GELU} (xW+b)\odot (xV+c)\\\operatorname {SwiGLU} (x,W,V,b,c,\beta )&=\operatorname {Swish} _{\beta }(xW+b)\odot (xV+c)\end{aligned}}} {\displaystyle {\begin{aligned}\operatorname {GLU} (x,W,V,b,c)&=\sigma (xW+b)\odot (xV+c)\\\operatorname {Bilinear} (x,W,V,b,c)&=(xW+b)\odot (xV+c)\\\operatorname {ReGLU} (x,W,V,b,c)&=\max(0,xW+b)\odot (xV+c)\\\operatorname {GEGLU} (x,W,V,b,c)&=\operatorname {GELU} (xW+b)\odot (xV+c)\\\operatorname {SwiGLU} (x,W,V,b,c,\beta )&=\operatorname {Swish} _{\beta }(xW+b)\odot (xV+c)\end{aligned}}}These functions are often used in transformers, in the feedforward modules following an attention block:[6] FFN GLU ⁡ ( x , W , V , b , c , W ′ , b ′ ) = ( σ ( x W + b ) ⊙ ( x V + c ) ) W ′ + b ′ FFN Bilinear ⁡ ( x , W , V , b , c , W ′ , b ′ ) = ( ( x W + b ) ⊙ ( x V + c ) ) W ′ + b ′ FFN ReGLU ⁡ ( x , W , V , b , c , W ′ , b ′ ) = ( max ( 0 , x W + b ) ⊙ ( x V + c ) ) W ′ + b ′ FFN GEGLU ⁡ ( x , W , V , b , c , W ′ , b ′ ) = ( GELU ⁡ ( x W + b ) ⊙ ( x V + c ) ) W ′ + b ′ FFN SwiGLU ⁡ ( x , W , V , b , c , β , W ′ , b ′ ) = ( Swish β ⁡ ( x W + b ) ⊙ ( x V + c ) ) W ′ + b ′ {\displaystyle {\begin{aligned}\operatorname {FFN} _{\operatorname {GLU} }(x,W,V,b,c,W',b')&=(\sigma (xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {Bilinear} }(x,W,V,b,c,W',b')&=((xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {ReGLU} }(x,W,V,b,c,W',b')&=(\max(0,xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {GEGLU} }(x,W,V,b,c,W',b')&=(\operatorname {GELU} (xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {SwiGLU} }(x,W,V,b,c,\beta ,W',b')&=(\operatorname {Swish} _{\beta }(xW+b)\odot (xV+c))W'+b'\end{aligned}}} {\displaystyle {\begin{aligned}\operatorname {FFN} _{\operatorname {GLU} }(x,W,V,b,c,W',b')&=(\sigma (xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {Bilinear} }(x,W,V,b,c,W',b')&=((xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {ReGLU} }(x,W,V,b,c,W',b')&=(\max(0,xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {GEGLU} }(x,W,V,b,c,W',b')&=(\operatorname {GELU} (xW+b)\odot (xV+c))W'+b'\\\operatorname {FFN} _{\operatorname {SwiGLU} }(x,W,V,b,c,\beta ,W',b')&=(\operatorname {Swish} _{\beta }(xW+b)\odot (xV+c))W'+b'\end{aligned}}}

Other architectures

Gating mechanism is used in highway networks, which were designed by unrolling an LSTM.

Channel gating[7] uses a gate to control the flow of information through different channels inside a convolutional neural network (CNN).

Gated Attention Unit (GAU) are used in attention mechanisms in Transformers.[8][9]

See also

References

  1. Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. S2CID 1915014.
  2. Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "10.1. Long Short-Term Memory (LSTM)". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.
  3. Cho, Kyunghyun; van Merrienboer, Bart; Bahdanau, DZmitry; Bougares, Fethi; Schwenk, Holger; Bengio, Yoshua (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation". Association for Computational Linguistics. arXiv:1406.1078.
  4. Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "10.2. Gated Recurrent Units (GRU)". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.
  5. Dauphin, Yann N.; Fan, Angela; Auli, Michael; Grangier, David (2017-07-17). "Language Modeling with Gated Convolutional Networks". Proceedings of the 34th International Conference on Machine Learning. PMLR: 933–941. arXiv:1612.08083.
  6. Shazeer, Noam (February 14, 2020). "GLU Variants Improve Transformer". arXiv:2002.05202 [cs.LG].
  7. Hua, Weizhe; Zhou, Yuan; De Sa, Christopher M; Zhang, Zhiru; Suh, G. Edward (2019). "Channel Gating Neural Networks". Advances in Neural Information Processing Systems. 32. Curran Associates, Inc. arXiv:1805.12549.
  8. Qiu, Zihan; Wang, Zekun; Zheng, Bo; Huang, Zeyu; Wen, Kaiyue; Yang, Songlin; Men, Rui; Yu, Le; Huang, Fei; Huang, Suozhi; Liu, Dayiheng; Zhou, Jingren; Lin, Junyang (2025). "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free". Advances in Neural Information Processing Systems. 38, Main Conference. Curran Associates, Inc.: 100092–100118. doi:10.52202/085713-3345.
  9. Hua, Weizhe; Dai, Zihang; Liu, Hanxiao; Le, Quoc (2022-06-28). "Transformer Quality in Linear Time". Proceedings of the 39th International Conference on Machine Learning. PMLR: 9099–9117. ISSN 2640-3498.

Further reading