Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Jan 5, 2025

• if the hybrid module is an RNN, allow for folding it across the sequence for efficiency, and to answer the question just how much recurrence is needed

  lucidrains

  committed
  b28d82a

  Copy full SHA for b28d82a

• flexibly handle hybrid module outputs

  lucidrains

  committed
  c51ecd3

  Copy full SHA for c51ecd3

• readme

  lucidrains

  committed
  b81646f

  Copy full SHA for b81646f

Commits on Jan 3, 2025

• add ability to hybridize attention with external module, for aiming to resolve state tracking issue by next week end

  lucidrains

  committed
  39bbb08

  Copy full SHA for 39bbb08

Commits on Dec 27, 2024

• able to return per token logit entropy

  lucidrains

  committed
  f944dd7

  Copy full SHA for f944dd7

Commits on Dec 24, 2024

• merge hyper connection streams before final norm, to avoid edge case with adaptive layernorm

  lucidrains

  committed
  e1be411

  Copy full SHA for e1be411

Commits on Dec 23, 2024

• address lucidrains#305

  lucidrains

  committed
  003275c

  Copy full SHA for 003275c

Commits on Dec 22, 2024

• using tanh in hyperconnection was not clear cut

  lucidrains

  committed
  cdf51f7

  Copy full SHA for cdf51f7

Commits on Dec 21, 2024

• note

  lucidrains

  committed
  8880051

  Copy full SHA for 8880051

• add the proposed hyper-connections (multiple residual streams) proposed by bytedance ai labs

  lucidrains

  committed
  8b367e6

  Copy full SHA for 8b367e6

Commits on Dec 20, 2024

• default learned value residual mix to true

  lucidrains

  committed
  0fd37f5

  Copy full SHA for 0fd37f5

Commits on Dec 10, 2024

• address a bug

  lucidrains

  committed
  5231e5d

  Copy full SHA for 5231e5d

• one more test

  lucidrains

  committed
  f5d3907

  Copy full SHA for f5d3907

• address lucidrains#303, allow for rotary embedding for cross attention if context is in shared positional space as input

  lucidrains

  committed
  70c59b6

  Copy full SHA for 70c59b6

Commits on Dec 7, 2024

• readme

  lucidrains

  authored
  66be236

  Copy full SHA for 66be236

Commits on Dec 4, 2024

• oops learned value residual mix does not apply to cross attention

  lucidrains

  committed
  90a3b11

  Copy full SHA for 90a3b11

Commits on Dec 3, 2024

• deviate from paper and use softclamping instead for laser

  lucidrains

  committed
  544c699

  Copy full SHA for 544c699

Commits on Dec 2, 2024

• it performs worse than baseline. maybe there is something else going on with laser attn

  lucidrains

  committed
  5b4ddef

  Copy full SHA for 5b4ddef

• safe log

  lucidrains

  committed
  48afc51

  Copy full SHA for 48afc51

Commits on Nov 30, 2024

• 1.42.21

  lucidrains

  committed
  35452f4

  Copy full SHA for 35452f4

• Support custom positions for rotary positional embedding (lucidrains#301)

  Show description for c493eb4
  Aceticia

  authored
  c493eb4

  Copy full SHA for c493eb4

Commits on Nov 29, 2024

• 1.42.20

  lucidrains

  authored
  8c993f0

  Copy full SHA for 8c993f0

• Custom pos alibi flash attn fix (lucidrains#300)

  Show description for 4c3e62a
  Aceticia

  authored
  4c3e62a

  Copy full SHA for 4c3e62a

• seeing a small but noticeable improvement from proposed Laser attention out of UT Austin

  lucidrains

  committed
  57efd77

  Copy full SHA for 57efd77

Commits on Nov 26, 2024

• address lucidrains#296

  lucidrains

  committed
  b720245

  Copy full SHA for b720245

Commits on Nov 24, 2024

• make sure forgettable transformer works with memory key values

  lucidrains

  committed
  5791a85

  Copy full SHA for 5791a85

Commits on Nov 23, 2024

• even stronger

  lucidrains

  committed
  28b693d

  Copy full SHA for 28b693d

Commits on Nov 22, 2024

• seeing really big improvements with per token learned value residual mixing values

  lucidrains

  committed
  3d61134

  Copy full SHA for 3d61134

Commits on Nov 20, 2024

• 1.42.12

  lucidrains

  committed
  9a5f78d

  Copy full SHA for 9a5f78d

• add packaging (lucidrains#295)

  iamthebot

  authored
  28a8897

  Copy full SHA for 28a8897

Commits on Nov 11, 2024

• address lucidrains#291

  lucidrains

  committed
  abeedc8

  Copy full SHA for abeedc8

• address lucidrains#290

  lucidrains

  committed
  15dad55

  Copy full SHA for 15dad55

Commits on Nov 8, 2024

• add ability to set slopes on alibi as well as custom position across heads

  lucidrains

  committed
  144d9ba

  Copy full SHA for 144d9ba

Commits on Nov 7, 2024

• release custom embedder and unembedder, contributed by @pradeep-pyro

  lucidrains

  committed
  55148ba

  Copy full SHA for 55148ba

• Customizable Embedder and Logit Mapper (lucidrains#288)

  Show description for 409ba0f
  pradeep-pyro

  authored
  409ba0f

  Copy full SHA for 409ba0f

Pagination

Previous

Next