Commits

Breadcrumbs

History for

onmain

User selector

All users

Datepicker

All time

Commit History

Commits on Jun 28, 2024

• cleanup gamma unit offset in norms

  lucidrains

  committed
  af345a3

  Copy full SHA for af345a3

Commits on Jun 27, 2024

• add the cool trick that makes norm gammas weight decayable, from @OhadRubin . also throw in another optional soft clamp on the final logits, used in gemma2

  lucidrains

  committed
  cc3b663

  Copy full SHA for cc3b663

Commits on Jun 19, 2024

• complete the ada-ln zero conditioning used in DiT

  lucidrains

  committed
  830db47

  Copy full SHA for 830db47

Commits on Jun 4, 2024

• add support for CoPE

  lucidrains

  committed
  306c1d9

  Copy full SHA for 306c1d9

Commits on May 29, 2024

• add ability to softclamp the pre-softmax attention logits, using attn_logits_softclamp_value: float

  lucidrains

  committed
  b9bec04

  Copy full SHA for b9bec04

Commits on May 28, 2024

• show key padding mask for continous transformer example

  lucidrains

  committed
  e9a6b18

  Copy full SHA for e9a6b18

Commits on May 27, 2024

• link

  lucidrains

  authored
  f2bf753

  Copy full SHA for f2bf753

Commits on May 11, 2024

• add layerscale, researcher friend says it works well, also bump torch and einops dep

  lucidrains

  committed
  a11e039

  Copy full SHA for a11e039

Commits on May 10, 2024

• default the depth to length of computed execution order of layers

  lucidrains

  committed
  58fb026

  Copy full SHA for 58fb026

• make sure a sort of hidden feature is documented, to encourage exploration

  lucidrains

  committed
  3abed69

  Copy full SHA for 3abed69

Commits on Apr 26, 2024

• address lucidrains#248

  lucidrains

  committed
  17ca68b

  Copy full SHA for 17ca68b

Commits on Jan 16, 2024

• add dpo

  lucidrains

  committed
  f185da8

  Copy full SHA for f185da8

Commits on Jan 12, 2024

• update

  lucidrains

  authored
  c7c7a9e

  Copy full SHA for c7c7a9e

Commits on Dec 8, 2023

• corroboration for dynamic pos bias

  lucidrains

  authored
  aabee05

  Copy full SHA for aabee05

Commits on Nov 22, 2023

• address lucidrains#205

  lucidrains

  committed
  e585d33

  Copy full SHA for e585d33

Commits on Oct 20, 2023

• xval readme

  lucidrains

  authored
  5067dc1

  Copy full SHA for 5067dc1

• readme

  lucidrains

  authored
  446cdf8

  Copy full SHA for 446cdf8

Commits on Oct 19, 2023

• xval document (lucidrains#196)

  Show description for 61e8a21
  lucidrains

  committed
  61e8a21

  Copy full SHA for 61e8a21

• add xval wrapper and autoregressive wrapper (lucidrains#195)

  lucidrains

  authored
  d4cc232

  Copy full SHA for d4cc232

Commits on Oct 7, 2023

• start buying the whole attend to nothing line of research

  lucidrains

  committed
  c4fd516

  Copy full SHA for c4fd516

Commits on Oct 3, 2023

• remove cascading heads

  lucidrains

  committed
  4f9775b

  Copy full SHA for 4f9775b

• add ability to intersperse register / memory tokens for decoders. also add memory tokens to vision transformer wrapper

  lucidrains

  committed
  37b351b

  Copy full SHA for 37b351b

Commits on Oct 2, 2023

• just bait some researcher into doing a paper for gpt

  lucidrains

  committed
  9fc3023

  Copy full SHA for 9fc3023

Commits on Oct 1, 2023

• update

  lucidrains

  committed
  7e0cd2d

  Copy full SHA for 7e0cd2d

Commits on Sep 19, 2023

• add ability to do contrastive decoding by passing in amateur_model kwarg through .generate function on AutoregressiveWrapper. using the new formulation from O'Brien et al

  lucidrains

  committed
  b38cca6

  Copy full SHA for b38cca6

• found an elegant way to generalize weight tying

  lucidrains

  committed
  9075074

  Copy full SHA for 9075074

Commits on Sep 14, 2023

• remove deepnorm

  lucidrains

  committed
  4ba5087

  Copy full SHA for 4ba5087

Commits on Sep 7, 2023

• document usage of qk norm

  lucidrains

  authored
  3c728fb

  Copy full SHA for 3c728fb

Commits on Aug 27, 2023

• add grouped multi-query attention, from a recent Brain paper

  lucidrains

  committed
  1b9aea8

  Copy full SHA for 1b9aea8

Commits on Aug 23, 2023

• add the auxiliary z-loss for attention logits; reading mixed results

  lucidrains

  committed
  23f913a

  Copy full SHA for 23f913a

Commits on Aug 18, 2023

• start removing some stuff that is unpromising

  lucidrains

  committed
  6867e9a

  Copy full SHA for 6867e9a

• add ability to use the multiplicative bias used on conjunction with ff geglu in the ST-MoE paper, appendix C

  lucidrains

  committed
  bba1959

  Copy full SHA for bba1959

Commits on Aug 4, 2023

• 1.17.1

  lucidrains

  committed
  7395ebd

  Copy full SHA for 7395ebd

• address lucidrains#171

  lucidrains

  committed
  273d4e8

  Copy full SHA for 273d4e8

Commits on Jul 30, 2023

• add the "simple" rmsnorm from the new transnormer paper

  lucidrains

  committed
  1cfead3

  Copy full SHA for 1cfead3

Pagination

Previous

Next