Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Dec 22, 2022

• add an extra warning

  lucidrains

  committed
  b54891a

  Copy full SHA for b54891a

• add a note for alibi

  lucidrains

  committed
  ce93b15

  Copy full SHA for ce93b15

• Merge pull request lucidrains#117 from lucidrains/xpos

  Show description for 64b14f2
  lucidrains

  authored
  64b14f2

  Copy full SHA for 64b14f2

• it works but dynamic pos bias is probably still better (just more complicated). alibi also seems to be around the same results in my tests

  lucidrains

  committed
  4dbe6b4

  Copy full SHA for 4dbe6b4

• keep it simple

  lucidrains

  committed
  d9fd14e

  Copy full SHA for d9fd14e

• prepare to potentially fix rotary embeddings

  lucidrains

  committed
  33231b4

  Copy full SHA for 33231b4

Commits on Dec 14, 2022

• be able to return all intermediates from transformer wrapper

  lucidrains

  committed
  15cef57

  Copy full SHA for 15cef57

Commits on Dec 5, 2022

• add cross attention context token dropout. numerous papers have used this by now (perceiver ar), although never the main focus on the paper, so not sure who to cite

  lucidrains

  committed
  4b46feb

  Copy full SHA for 4b46feb

• start embracing structured dropout, starting with entire layer dropouts, customizable for each layer by passing in a tuple as well

  lucidrains

  committed
  eeadd43

  Copy full SHA for eeadd43

Commits on Nov 27, 2022

• more cleanup

  lucidrains

  committed
  bcea583

  Copy full SHA for bcea583

• cleanup some masking related code

  lucidrains

  committed
  712cb03

  Copy full SHA for 712cb03

Commits on Nov 22, 2022

• 1.2.4

  lucidrains

  committed
  b20db6d

  Copy full SHA for b20db6d

• Merge pull request lucidrains#112 from Alfredvc/continous_wrapper_prepend_embeds

  Show description for 334252a
  lucidrains

  authored
  334252a

  Copy full SHA for 334252a

Commits on Nov 21, 2022

• Added prepend_embeds to ContinousTransformerWrapper, same as in TransformerWrapper. Removed unused line in ContinousTransformerWrapper.forward

  

  Alfredo Clemente

  committed
  5ff688e

  Copy full SHA for 5ff688e

Commits on Nov 4, 2022

• address lucidrains#108

  lucidrains

  committed
  526f675

  Copy full SHA for 526f675

Commits on Nov 2, 2022

• add diagram

  lucidrains

  committed
  2eae4a3

  Copy full SHA for 2eae4a3

• need to differentiate between context mask between self and cross attention, since attention layers may have both

  lucidrains

  committed
  9dbbcd3

  Copy full SHA for 9dbbcd3

• add forgetful causal mask implementation to AutoregressiveWrapper

  lucidrains

  committed
  595a474

  Copy full SHA for 595a474

Commits on Nov 1, 2022

• Revert "document a new layernorm placement, this time before outward projection in multi-head attention"

  Show description for d056af8
  lucidrains

  committed
  d056af8

  Copy full SHA for d056af8

Commits on Oct 16, 2022

• document a new layernorm placement, this time before outward projection in multi-head attention

  lucidrains

  committed
  a848433

  Copy full SHA for a848433

Commits on Sep 17, 2022

• add ability to append extra embeddings to be attended to in transformer, show how SOTA language-vision PaLI model can be built

  lucidrains

  committed
  fb16713

  Copy full SHA for fb16713

Commits on Sep 13, 2022

• remove entmax to only have einops and torch as dependency, also lower cosine sim attention scale to 10, given @borisdayma cosine sim attention experiment. 16 may have numerical precision issues in f16

  lucidrains

  committed
  975db21

  Copy full SHA for 975db21

Commits on Aug 19, 2022

• fix rel pos bias bug in presence of transformer-xl, thanks to @taemincho

  lucidrains

  committed
  cc107eb

  Copy full SHA for cc107eb

• Merge pull request lucidrains#103 from taemincho/main

  Show description for 7fb85f2
  lucidrains

  authored
  7fb85f2

  Copy full SHA for 7fb85f2

• Fix wrong positional biases

  taemincho

  committed
  dcae403

  Copy full SHA for dcae403

Commits on Aug 15, 2022

• fix for cross attention

  lucidrains

  committed
  aabfb75

  Copy full SHA for aabfb75

• add the tensor product attention, make 1.0 since why not

  lucidrains

  committed
  9866ca5

  Copy full SHA for 9866ca5

Commits on Aug 10, 2022

• tweak scalenorm

  lucidrains

  committed
  7df4720

  Copy full SHA for 7df4720

Commits on Aug 7, 2022

• add deepnorm, successfully used in a 130B parameter model out of Tsinghua

  lucidrains

  committed
  8774657

  Copy full SHA for 8774657

• softmax is already stable, also start enforcing it to be always float32

  lucidrains

  committed
  1adb2ea

  Copy full SHA for 1adb2ea

• add gradient shrinking technique from cogview and glm-130B

  lucidrains

  committed
  7212069

  Copy full SHA for 7212069

Commits on Aug 5, 2022

• remove what does not work

  lucidrains

  committed
  44f30eb

  Copy full SHA for 44f30eb

Commits on Jul 28, 2022

• more informative error message

  lucidrains

  committed
  655ff30

  Copy full SHA for 655ff30

• update with findings

  lucidrains

  committed
  ded98e3

  Copy full SHA for ded98e3

Commits on Jul 26, 2022

• add a technique that went under my radar, since it was not published anywhere but in a chinese blog post. from Jianlin Su, the creator of rotary embeddings.

  lucidrains

  committed
  709d9da

  Copy full SHA for 709d9da

Pagination

Previous

Next