Brand-New Attention Chapter

We have added the brand-new Chapter: Attention Mechanisms:

• Attention Cues

  • Attention Cues in Biology
  • Queries, Keys, and Values
  • Visualization of Attention

• Attention Pooling: Nadaraya-Watson Kernel Regression

  • Generating the Dataset
  • Average Pooling
  • Nonparametric Attention Pooling
  • Parametric Attention Pooling

• Attention Scoring Functions

  • Masked Softmax Operation
  • Additive Attention
  • Scaled Dot-Product Attention

• Bahdanau Attention

  • Model
  • Defining the Decoder with Attention
  • Training

• Multi-Head Attention

  • Model
  • Implementation

• Self-Attention and Positional Encoding

  • Self-Attention
  • Comparing CNNs, RNNs, and Self-Attention