Commits

Breadcrumbs

History for

onmain

User selector

All users

Datepicker

All time

Commit History

Commits on Jan 10, 2022

• update readme with RMS norm, given news of deepmind using it

  lucidrains

  committed
  b0c0ec9

  Copy full SHA for b0c0ec9

Commits on Dec 13, 2021

• add continuous autoregressive wrapper for continuous transformers training, as done in https://arxiv.org/abs/2112.05329

  lucidrains

  committed
  0c8005b

  Copy full SHA for 0c8005b

Commits on Dec 3, 2021

• make sure feedforward is still pre-norm when qk norm is turned on

  lucidrains

  committed
  6223509

  Copy full SHA for 6223509

Commits on Dec 2, 2021

• add query-key normalization attention

  lucidrains

  committed
  3228fff

  Copy full SHA for 3228fff

Commits on Nov 1, 2021

• fix alphafold2 styled value gating thanks to @imflash217

  lucidrains

  committed
  188cbd7

  Copy full SHA for 188cbd7

Commits on Oct 23, 2021

• complete normformer

  lucidrains

  committed
  d511499

  Copy full SHA for d511499

Commits on Oct 20, 2021

• add some Normformer proposed changes

  lucidrains

  committed
  a17ebb3

  Copy full SHA for a17ebb3

Commits on Oct 19, 2021

• add sandwich norm, from coqview paper

  lucidrains

  committed
  91a3859

  Copy full SHA for 91a3859

Commits on Oct 4, 2021

• do not throw away any memories - instead, route lower level memories to last few layers

  lucidrains

  committed
  c1283da

  Copy full SHA for c1283da

Commits on Oct 3, 2021

• add enhanced recurrence, from Ernie-doc paper

  lucidrains

  committed
  989c2ca

  Copy full SHA for 989c2ca

Commits on Sep 20, 2021

• add relu squared activation, from primer paper

  lucidrains

  committed
  f43d7db

  Copy full SHA for f43d7db

Commits on Sep 6, 2021

• follow better init proposed by shmidhuber, after seeing results on local machine

  lucidrains

  committed
  016b7ae

  Copy full SHA for 016b7ae

Commits on Aug 27, 2021

• rerecommend

  lucidrains

  committed
  b0cadb2

  Copy full SHA for b0cadb2

• add ability to configure number of heads for ALiBi bias

  lucidrains

  committed
  f937d12

  Copy full SHA for f937d12

• readme

  lucidrains

  committed
  a231d07

  Copy full SHA for a231d07

• add ALiBi positional embedding and cite

  lucidrains

  committed
  a99179d

  Copy full SHA for a99179d

Commits on Aug 25, 2021

• readme

  lucidrains

  committed
  3117e5f

  Copy full SHA for 3117e5f

Commits on Aug 22, 2021

• update readme

  lucidrains

  committed
  f791e7e

  Copy full SHA for f791e7e

Commits on Aug 20, 2021

• readme

  lucidrains

  authored
  65f6560

  Copy full SHA for 65f6560

Commits on Aug 13, 2021

• readme

  lucidrains

  authored
  10394ef

  Copy full SHA for 10394ef

• release shifted tokens hyperparameter for faster convergence

  lucidrains

  committed
  3342c10

  Copy full SHA for 3342c10

Commits on Jul 16, 2021

• diagram for alphafold2 attention

  lucidrains

  committed
  c1ddc01

  Copy full SHA for c1ddc01

• add intra-attention gating, used in alphafold2

  lucidrains

  committed
  7f130f5

  Copy full SHA for 7f130f5

Commits on May 18, 2021

• release collaborative heads

  lucidrains

  committed
  8973339

  Copy full SHA for 8973339

• Add collaborative attention

  jbcdnr

  committed
  f89bae7

  Copy full SHA for f89bae7

Commits on Apr 21, 2021

• add documentation for rotary embeddings

  lucidrains

  committed
  179f54a

  Copy full SHA for 179f54a

Commits on Apr 17, 2021

• add continuous transformer wrapper for @guillefix

  lucidrains

  committed
  e239be0

  Copy full SHA for e239be0

Commits on Apr 9, 2021

• readme

  lucidrains

  authored
  585bf54

  Copy full SHA for 585bf54

Commits on Mar 26, 2021

• add rotary positional embedding

  lucidrains

  committed
  e5b0324

  Copy full SHA for e5b0324

Commits on Mar 21, 2021

• add rms norm, given transformer modifications paper out of google

  lucidrains

  committed
  68ce7e7

  Copy full SHA for 68ce7e7

Commits on Jan 16, 2021

• link to txl for RL paper

  lucidrains

  committed
  83fbd35

  Copy full SHA for 83fbd35

• add diagram for gating feature

  lucidrains

  committed
  172985a

  Copy full SHA for 172985a

• add gating at residuals, from deepminds paper for stabilizing txl for RL paper

  lucidrains

  committed
  25dee85

  Copy full SHA for 25dee85

Commits on Jan 1, 2021

• cite shortformer

  lucidrains

  committed
  67877d6

  Copy full SHA for 67877d6

• cleanup

  lucidrains

  committed
  b5c06bc

  Copy full SHA for b5c06bc

Pagination

Previous

Next