Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Jan 10, 2022

• update readme with RMS norm, given news of deepmind using it

  lucidrains

  committed
  b0c0ec9

  Copy full SHA for b0c0ec9

Commits on Dec 13, 2021

• add continuous autoregressive wrapper for continuous transformers training, as done in https://arxiv.org/abs/2112.05329

  lucidrains

  committed
  0c8005b

  Copy full SHA for 0c8005b

Commits on Dec 3, 2021

• make sure feedforward is still pre-norm when qk norm is turned on

  lucidrains

  committed
  6223509

  Copy full SHA for 6223509

Commits on Dec 2, 2021

• add query-key normalization attention

  lucidrains

  committed
  3228fff

  Copy full SHA for 3228fff

Commits on Nov 26, 2021

• allow token shifting and rezero to coexist

  lucidrains

  committed
  a9de3a8

  Copy full SHA for a9de3a8

• fix macaron and rezero

  lucidrains

  committed
  484d555

  Copy full SHA for 484d555

Commits on Nov 25, 2021

• 0.20.7

  lucidrains

  authored
  4a242f2

  Copy full SHA for 4a242f2

• 0.20.6

  lucidrains

  authored
  7cb686e

  Copy full SHA for 7cb686e

• fix rezero

  lucidrains

  authored
  2b1826e

  Copy full SHA for 2b1826e

Commits on Nov 24, 2021

• clearer code

  lucidrains

  authored
  6a7f960

  Copy full SHA for 6a7f960

Commits on Nov 6, 2021

• add ability to define maximum attention length into the past with attn_max_attend_past

  lucidrains

  committed
  55ca5d9

  Copy full SHA for 55ca5d9

Commits on Nov 1, 2021

• fix alphafold2 styled value gating thanks to @imflash217

  lucidrains

  committed
  188cbd7

  Copy full SHA for 188cbd7

Commits on Oct 23, 2021

• complete normformer

  lucidrains

  committed
  d511499

  Copy full SHA for d511499

Commits on Oct 20, 2021

• add some Normformer proposed changes

  lucidrains

  committed
  a17ebb3

  Copy full SHA for a17ebb3

• fix bug

  lucidrains

  committed
  5f37fc0

  Copy full SHA for 5f37fc0

Commits on Oct 19, 2021

• readme

  lucidrains

  committed
  6ccbfbe

  Copy full SHA for 6ccbfbe

• bump for 0.20.0

  lucidrains

  committed
  86e5928

  Copy full SHA for 86e5928

• add sandwich norm, from coqview paper

  lucidrains

  committed
  91a3859

  Copy full SHA for 91a3859

Commits on Oct 4, 2021

• do not throw away any memories - instead, route lower level memories to last few layers

  lucidrains

  committed
  c1283da

  Copy full SHA for c1283da

Commits on Oct 3, 2021

• add enhanced recurrence, from Ernie-doc paper

  lucidrains

  committed
  989c2ca

  Copy full SHA for 989c2ca

Commits on Sep 20, 2021

• add relu squared activation, from primer paper

  lucidrains

  committed
  f43d7db

  Copy full SHA for f43d7db

Commits on Sep 18, 2021

• 0.17.13

  lucidrains

  authored
  0bad811

  Copy full SHA for 0bad811

• Merge pull request lucidrains#62 from gurvindersingh/main

  Show description for 2eb20cf
  lucidrains

  authored
  2eb20cf

  Copy full SHA for 2eb20cf

• Merge branch 'lucidrains:main' into main

  gurvindersingh

  authored
  c60cbc0

  Copy full SHA for c60cbc0

• Support top_a sampling method

  Show description for 0a9437a
  gurvindersingh

  committed
  0a9437a

  Copy full SHA for 0a9437a

Commits on Sep 17, 2021

• Merge pull request lucidrains#61 from gurvindersingh/main

  Show description for 97e66d6
  lucidrains

  authored
  97e66d6

  Copy full SHA for 97e66d6

Commits on Sep 15, 2021

• fixed typo in variable name

  gurvindersingh

  committed
  90f7a7a

  Copy full SHA for 90f7a7a

Commits on Sep 6, 2021

• follow better init proposed by shmidhuber, after seeing results on local machine

  lucidrains

  committed
  016b7ae

  Copy full SHA for 016b7ae

Commits on Sep 3, 2021

• allow for fine control of self attention masking

  lucidrains

  committed
  a8985b5

  Copy full SHA for a8985b5

Commits on Aug 31, 2021

• place an assert

  lucidrains

  committed
  905219f

  Copy full SHA for 905219f

• fix small bug

  lucidrains

  committed
  e5acd11

  Copy full SHA for e5acd11

Commits on Aug 30, 2021

• offer learned alibi as a hidden experimental feature

  lucidrains

  committed
  7e1266b

  Copy full SHA for 7e1266b

Commits on Aug 28, 2021

• remove dead code

  lucidrains

  committed
  cbf0102

  Copy full SHA for cbf0102

• do not persist alibi bias and slopes to state dict

  lucidrains

  committed
  5768739

  Copy full SHA for 5768739

Commits on Aug 27, 2021

• rerecommend

  lucidrains

  committed
  b0cadb2

  Copy full SHA for b0cadb2

Pagination

Previous

Next