Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Apr 25, 2022

• 0.27.1

  Show description for aa2aefb
  lucidrains

  authored
  aa2aefb

  Copy full SHA for aa2aefb

• Merge pull request lucidrains#85 from ilya16/bugfix-alibi

  Show description for 43e565b
  lucidrains

  authored
  43e565b

  Copy full SHA for 43e565b

• fix bug with bias and slopes padding in ALiBi

  ilya16

  committed
  6523746

  Copy full SHA for 6523746

Commits on Apr 23, 2022

• remove rezero, it does not work well and is not "all you need" https://arxiv.org/abs/2102.11972

  lucidrains

  committed
  ca5ddde

  Copy full SHA for ca5ddde

Commits on Apr 22, 2022

• allow for turning off bias in feedforward, given PaLM and results from @borisdayma https://twitter.com/borisdayma/status/1517227191477571585

  lucidrains

  committed
  2428207

  Copy full SHA for 2428207

Commits on Apr 21, 2022

• allow for external absolute positional embeddings to be fed into the transformer, for autoregressive infilling wrapper eventually

  lucidrains

  committed
  4895db5

  Copy full SHA for 4895db5

• 0.26.4 and cleanup some rearrange code

  lucidrains

  committed
  73a99f3

  Copy full SHA for 73a99f3

• Merge pull request lucidrains#84 from cifkao/arbitrary-pos

  Show description for c0d7e57
  lucidrains

  authored
  c0d7e57

  Copy full SHA for c0d7e57

Commits on Apr 20, 2022

• Allow arbitrary shape of pos

  cifkao

  committed
  48c27ed

  Copy full SHA for 48c27ed

• Allow arbitrary positions in absolute PE

  cifkao

  committed
  8a8a7f4

  Copy full SHA for 8a8a7f4

Commits on Apr 15, 2022

• 0.26.3

  lucidrains

  authored
  75afaca

  Copy full SHA for 75afaca

• Merge pull request lucidrains#83 from gurvindersingh/main

  Show description for 3f0e3de
  lucidrains

  authored
  3f0e3de

  Copy full SHA for 3f0e3de

• avoid saving intermediaries unless needed

  gurvindersingh

  committed
  6835738

  Copy full SHA for 6835738

Commits on Apr 14, 2022

• 0.26.2

  lucidrains

  committed
  a358c6f

  Copy full SHA for a358c6f

• Merge pull request lucidrains#82 from gurvindersingh/main

  Show description for 3bb863e
  lucidrains

  authored
  3bb863e

  Copy full SHA for 3bb863e

• allow customizing ignore index and pad value

  gurvindersingh

  committed
  21ab860

  Copy full SHA for 21ab860

Commits on Apr 11, 2022

• allow one to customize the head dimension of the values differently than queries / keys, seen in some papers, but rarely

  lucidrains

  committed
  629e70f

  Copy full SHA for 629e70f

• add noam shazeers "one write head is all you need" one-headed key/values, used in AlphaCode as well as PaLM

  lucidrains

  committed
  40b1ba7

  Copy full SHA for 40b1ba7

Commits on Apr 7, 2022

• sponsor

  lucidrains

  authored
  069fa3e

  Copy full SHA for 069fa3e

Commits on Apr 5, 2022

• patch

  lucidrains

  committed
  d1b7cb3

  Copy full SHA for d1b7cb3

• Merge pull request lucidrains#79 from gurvindersingh/main

  Show description for 51bc00b
  lucidrains

  authored
  51bc00b

  Copy full SHA for 51bc00b

• don't store intermediates and hidden states if not needed

  gurvindersingh

  committed
  082c477

  Copy full SHA for 082c477

• fix bug in use l2norm embedding when pos_emb is instance of always

  gurvindersingh

  committed
  e7233ee

  Copy full SHA for e7233ee

• add ability to turn off absolute positional embeddings, given new paper from Ofir Press

  lucidrains

  committed
  e600d93

  Copy full SHA for e600d93

• offer the swish glu variant, given googles new 540 billion parameter model, PaLM

  lucidrains

  committed
  c2d4efd

  Copy full SHA for c2d4efd

Commits on Mar 30, 2022

• fix bug with GRUGating turned on

  lucidrains

  committed
  11d6983

  Copy full SHA for 11d6983

Commits on Mar 23, 2022

• add ability to scale residuals by a constant factor, for https://arxiv.org/abs/2203.00555

  lucidrains

  committed
  a93bfed

  Copy full SHA for a93bfed

Commits on Mar 5, 2022

• make sure dynamic positional bias is O(n)

  lucidrains

  committed
  ee27e7f

  Copy full SHA for ee27e7f

Commits on Mar 4, 2022

• readme for dynamic pos bias

  lucidrains

  authored
  e94cc4b

  Copy full SHA for e94cc4b

Commits on Mar 3, 2022

• add dynamic positional bias, for extrapolating to greater sequence lengths than what is trained on, for autoregressive and bidirectional transformers, thanks to experiments from @bob80333

  lucidrains

  committed
  fed9085

  Copy full SHA for fed9085

Commits on Feb 13, 2022

• add l2norm embeddings (part of fixnorm), from Transformer Without Tears paper

  lucidrains

  committed
  b31ed8f

  Copy full SHA for b31ed8f

Commits on Feb 10, 2022

• cleanup talking heads

  lucidrains

  committed
  75d723b

  Copy full SHA for 75d723b

Commits on Jan 31, 2022

• fix bug with bidirectional alibi

  lucidrains

  committed
  4ed4d0a

  Copy full SHA for 4ed4d0a

Commits on Jan 26, 2022

• offer bidirectional ALiBi (nonsymmetric), proposed by @ofirpress

  lucidrains

  committed
  82bd182

  Copy full SHA for 82bd182

Commits on Jan 10, 2022

• update readme with RMS norm, given news of deepmind using it

  lucidrains

  committed
  b0c0ec9

  Copy full SHA for b0c0ec9

Pagination

Previous

Next