Commits

Breadcrumbs

History for

onmain

User selector

All users

Datepicker

All time

Commit History

Commits on Dec 14, 2022

• be able to return all intermediates from transformer wrapper

  lucidrains

  committed
  15cef57

  Copy full SHA for 15cef57

Commits on Dec 5, 2022

• add cross attention context token dropout. numerous papers have used this by now (perceiver ar), although never the main focus on the paper, so not sure who to cite

  lucidrains

  committed
  4b46feb

  Copy full SHA for 4b46feb

• start embracing structured dropout, starting with entire layer dropouts, customizable for each layer by passing in a tuple as well

  lucidrains

  committed
  eeadd43

  Copy full SHA for eeadd43

Commits on Nov 27, 2022

• more cleanup

  lucidrains

  committed
  bcea583

  Copy full SHA for bcea583

• cleanup some masking related code

  lucidrains

  committed
  712cb03

  Copy full SHA for 712cb03

Commits on Nov 21, 2022

• Added prepend_embeds to ContinousTransformerWrapper, same as in TransformerWrapper. Removed unused line in ContinousTransformerWrapper.forward

  

  Alfredo Clemente

  committed
  5ff688e

  Copy full SHA for 5ff688e

Commits on Nov 4, 2022

• address lucidrains#108

  lucidrains

  committed
  526f675

  Copy full SHA for 526f675

Commits on Nov 2, 2022

• need to differentiate between context mask between self and cross attention, since attention layers may have both

  lucidrains

  committed
  9dbbcd3

  Copy full SHA for 9dbbcd3

• add forgetful causal mask implementation to AutoregressiveWrapper

  lucidrains

  committed
  595a474

  Copy full SHA for 595a474

Commits on Nov 1, 2022

• Revert "document a new layernorm placement, this time before outward projection in multi-head attention"

  Show description for d056af8
  lucidrains

  committed
  d056af8

  Copy full SHA for d056af8

Commits on Oct 16, 2022

• document a new layernorm placement, this time before outward projection in multi-head attention

  lucidrains

  committed
  a848433

  Copy full SHA for a848433

Commits on Sep 17, 2022

• add ability to append extra embeddings to be attended to in transformer, show how SOTA language-vision PaLI model can be built

  lucidrains

  committed
  fb16713

  Copy full SHA for fb16713

Commits on Sep 13, 2022

• remove entmax to only have einops and torch as dependency, also lower cosine sim attention scale to 10, given @borisdayma cosine sim attention experiment. 16 may have numerical precision issues in f16

  lucidrains

  committed
  975db21

  Copy full SHA for 975db21

Commits on Aug 19, 2022

• fix rel pos bias bug in presence of transformer-xl, thanks to @taemincho

  lucidrains

  committed
  cc107eb

  Copy full SHA for cc107eb

• Fix wrong positional biases

  taemincho

  committed
  dcae403

  Copy full SHA for dcae403

Commits on Aug 15, 2022

• fix for cross attention

  lucidrains

  committed
  aabfb75

  Copy full SHA for aabfb75

• add the tensor product attention, make 1.0 since why not

  lucidrains

  committed
  9866ca5

  Copy full SHA for 9866ca5

Commits on Aug 10, 2022

• tweak scalenorm

  lucidrains

  committed
  7df4720

  Copy full SHA for 7df4720

Commits on Aug 7, 2022

• add deepnorm, successfully used in a 130B parameter model out of Tsinghua

  lucidrains

  committed
  8774657

  Copy full SHA for 8774657

• softmax is already stable, also start enforcing it to be always float32

  lucidrains

  committed
  1adb2ea

  Copy full SHA for 1adb2ea

• add gradient shrinking technique from cogview and glm-130B

  lucidrains

  committed
  7212069

  Copy full SHA for 7212069

Commits on Aug 5, 2022

• remove what does not work

  lucidrains

  committed
  44f30eb

  Copy full SHA for 44f30eb

Commits on Jul 28, 2022

• more informative error message

  lucidrains

  committed
  655ff30

  Copy full SHA for 655ff30

Commits on Jul 26, 2022

• add a technique that went under my radar, since it was not published anywhere but in a chinese blog post. from Jianlin Su, the creator of rotary embeddings.

  lucidrains

  committed
  709d9da

  Copy full SHA for 709d9da

Commits on Jul 17, 2022

• maybe group l2norm is not needed

  lucidrains

  committed
  92a28fd

  Copy full SHA for 92a28fd

Commits on Jul 15, 2022

• release grouped qk-normalization for researchers to play around with

  lucidrains

  committed
  52a4494

  Copy full SHA for 52a4494

Commits on Jul 11, 2022

• simplify cosine sim attention based on my own explorations and experimentations

  lucidrains

  committed
  e1a9bfc

  Copy full SHA for e1a9bfc

Commits on Jul 4, 2022

• Revert "fix readme, thanks to @KisuYang"

  Show description for 8f886d7
  lucidrains

  committed
  8f886d7

  Copy full SHA for 8f886d7

Commits on Jun 23, 2022

• give post embedding norm feature to continuous transformer wrapper as well as vision transformer wrapper

  lucidrains

  committed
  197b582

  Copy full SHA for 197b582

• update with new emerging technique for stabilizing training further

  lucidrains

  committed
  b7f91c6

  Copy full SHA for b7f91c6

Commits on Jun 17, 2022

• fix readme, thanks to @KisuYang

  lucidrains

  committed
  0a8fc9b

  Copy full SHA for 0a8fc9b

Commits on Jun 5, 2022

• add ability to share key / values, with attn_shared_kv = True

  lucidrains

  committed
  32fdd1d

  Copy full SHA for 32fdd1d

Commits on May 18, 2022

• fix AliBi bias caching for varying input shapes

  ilya16

  committed
  74f241d

  Copy full SHA for 74f241d

Commits on May 15, 2022

• Fix arguments related to bidirection which no longer exist, and handle param registers.

  jstjohn

  committed
  b489fd8

  Copy full SHA for b489fd8

• slight modifications to alibi which allow for bidirectional position encoding

  jstjohn

  committed
  2f940dc

  Copy full SHA for 2f940dc

Pagination

Previous

Next