Commits

Breadcrumbs

History for

on975db21

User selector

All users

Datepicker

All time

Commit History

Commits on Aug 7, 2022

• add deepnorm, successfully used in a 130B parameter model out of Tsinghua

  lucidrains

  committed
  8774657

  Copy full SHA for 8774657

Commits on Jul 26, 2022

• add a technique that went under my radar, since it was not published anywhere but in a chinese blog post. from Jianlin Su, the creator of rotary embeddings.

  lucidrains

  committed
  709d9da

  Copy full SHA for 709d9da

Commits on Apr 11, 2022

• add noam shazeers "one write head is all you need" one-headed key/values, used in AlphaCode as well as PaLM

  lucidrains

  committed
  40b1ba7

  Copy full SHA for 40b1ba7

Commits on Mar 3, 2022

• add dynamic positional bias, for extrapolating to greater sequence lengths than what is trained on, for autoregressive and bidirectional transformers, thanks to experiments from @bob80333

  lucidrains

  committed
  fed9085

  Copy full SHA for fed9085

Commits on Dec 2, 2021

• add query-key normalization attention

  lucidrains

  committed
  3228fff

  Copy full SHA for 3228fff

Commits on Oct 20, 2021

• add some Normformer proposed changes

  lucidrains

  committed
  a17ebb3

  Copy full SHA for a17ebb3

Commits on Oct 19, 2021

• readme

  lucidrains

  committed
  6ccbfbe

  Copy full SHA for 6ccbfbe

Commits on Oct 3, 2021

• add enhanced recurrence, from Ernie-doc paper

  lucidrains

  committed
  989c2ca

  Copy full SHA for 989c2ca

Commits on Jul 16, 2021

• diagram for alphafold2 attention

  lucidrains

  committed
  c1ddc01

  Copy full SHA for c1ddc01

Commits on May 18, 2021

• Add collaborative head illustration

  jbcdnr

  committed
  72f07b9

  Copy full SHA for 72f07b9

Commits on Apr 21, 2021

• add documentation for rotary embeddings

  lucidrains

  committed
  179f54a

  Copy full SHA for 179f54a

Commits on Jan 16, 2021

• add diagram for gating feature

  lucidrains

  committed
  172985a

  Copy full SHA for 172985a

Commits on Dec 31, 2020

• add position infused attention, from DETR and Shortformer papers

  lucidrains

  committed
  7d46572

  Copy full SHA for 7d46572

Commits on Dec 28, 2020

• add residual attention, from Realformer paper

  lucidrains

  committed
  257fee1

  Copy full SHA for 257fee1

Commits on Nov 25, 2020

• add docs for macarons

  lucidrains

  committed
  375ef1c

  Copy full SHA for 375ef1c

• add docs for sandwiches

  lucidrains

  committed
  a741d7f

  Copy full SHA for a741d7f

• add more docs

  lucidrains

  committed
  a8d6892

  Copy full SHA for a8d6892

• add talking heads

  lucidrains

  committed
  62b7885

  Copy full SHA for 62b7885

• add docs for concentrated attention paper

  lucidrains

  committed
  0db1861

  Copy full SHA for 0db1861

• add docs for rezero is all you need

  lucidrains

  committed
  38961d0

  Copy full SHA for 38961d0

• add docs for feedforward GLU variant

  lucidrains

  committed
  b07dd60

  Copy full SHA for b07dd60

• add docs for scale norm

  lucidrains

  committed
  2bfc4fc

  Copy full SHA for 2bfc4fc

• add docs for memory tokens

  lucidrains

  committed
  1ae0f4c

  Copy full SHA for 1ae0f4c

• add docs for all-attention memory key/value

  lucidrains

  committed
  6e66977

  Copy full SHA for 6e66977

Loading