Commits

Branch selector

 main

User selector

lucidrains

Datepicker

All time

Commit History

Commits on Jan 21, 2024

• patch

  lucidrains

  authored
  90cef69

  Copy full SHA for 90cef69

Commits on Jan 16, 2024

• fix flash attention with num memory key / values + 1 headed kv

  lucidrains

  committed
  b297919

  Copy full SHA for b297919

• fix memory key values with one write-head paper, lucidrains#228

  lucidrains

  committed
  029ec31

  Copy full SHA for 029ec31

• mem masks should be on the continuous transformer wrapper

  lucidrains

  committed
  aa380f1

  Copy full SHA for aa380f1

• add dpo

  lucidrains

  committed
  f185da8

  Copy full SHA for f185da8

Commits on Jan 12, 2024

• fix xpos lucidrains#226 (comment)

  lucidrains

  committed
  22f9b41

  Copy full SHA for 22f9b41

• apply pre-normalization to memories entering self attention block as well

  lucidrains

  committed
  49b196e

  Copy full SHA for 49b196e

• update

  lucidrains

  authored
  c7c7a9e

  Copy full SHA for c7c7a9e

• address lucidrains#223 with memory masking and offset for rotary positions

  lucidrains

  committed
  1d4e887

  Copy full SHA for 1d4e887

Commits on Jan 6, 2024

• address lucidrains#38

  lucidrains

  committed
  583c19d

  Copy full SHA for 583c19d

Commits on Jan 5, 2024

• use the latest layernorm without bias in pytorch if available

  lucidrains

  committed
  bb4d8fb

  Copy full SHA for bb4d8fb

• remove a bunch of unnecessary biases

  lucidrains

  committed
  7916859

  Copy full SHA for 7916859

Commits on Dec 30, 2023

• turn off kv caching for absolute positional embedding, when exceeding the context window during decoding, addressing lucidrains#219

  lucidrains

  committed
  a549c55

  Copy full SHA for a549c55

• 1.26.4

  lucidrains

  committed
  324cfac

  Copy full SHA for 324cfac

• just use argmax when temperature is 0 for sampling

  lucidrains

  committed
  ce42b57

  Copy full SHA for ce42b57

Commits on Dec 21, 2023

• fix masking out tokens after eos during generation

  lucidrains

  committed
  496e41d

  Copy full SHA for 496e41d

Commits on Dec 8, 2023

• corroboration for dynamic pos bias

  lucidrains

  authored
  aabee05

  Copy full SHA for aabee05

Commits on Dec 2, 2023

• support any number of additional embeddings, addressing lucidrains#213

  lucidrains

  committed
  3ebaa3f

  Copy full SHA for 3ebaa3f

Commits on Nov 27, 2023

• address some onnx issues lucidrains#212

  lucidrains

  committed
  59d6e00

  Copy full SHA for 59d6e00

• allow continuous transformer wrapper to have prepend embedding mask too

  lucidrains

  committed
  1f0232f

  Copy full SHA for 1f0232f

Commits on Nov 26, 2023

• allow for one to pass in prepend mask (for prepended embeddings), addressing lucidrains#211

  lucidrains

  committed
  13ac0cb

  Copy full SHA for 13ac0cb

Commits on Nov 25, 2023

• also return the final hidden layer after the last layernorm in the attention layers

  lucidrains

  committed
  99051c5

  Copy full SHA for 99051c5

• ability to return outputs from autoregressive wrapper forward

  lucidrains

  committed
  8795108

  Copy full SHA for 8795108

• patch

  lucidrains

  committed
  a00f542

  Copy full SHA for a00f542

Commits on Nov 24, 2023

• disable mixed precision for rotary embeddings, addressing lucidrains#210

  lucidrains

  committed
  4adbfc2

  Copy full SHA for 4adbfc2

• pad kv mask if memory is passed in, addressing lucidrains#209, even though it should typically only be used in autoregressive setting without mask

  lucidrains

  committed
  eb93a8b

  Copy full SHA for eb93a8b

Commits on Nov 23, 2023

• 1.25.5

  lucidrains

  authored
  5ce82c9

  Copy full SHA for 5ce82c9

Commits on Nov 22, 2023

• able to pass in custom positions for rotary embedding

  lucidrains

  committed
  37e82cd

  Copy full SHA for 37e82cd

• allow for rotary embeddings to be constructed outside the attention layers and passed in

  lucidrains

  committed
  6220d86

  Copy full SHA for 6220d86

• allow for cross attending to embeddings of a different dimension than the model dimensions of the decoder

  lucidrains

  committed
  9de5a5c

  Copy full SHA for 9de5a5c

• address lucidrains#205

  lucidrains

  committed
  e585d33

  Copy full SHA for e585d33

Commits on Nov 21, 2023

• also allow a constant prefix attention length given by integer, in addition to variable across batch samples

  lucidrains

  committed
  6acbf38

  Copy full SHA for 6acbf38

• add prefix causal attention, given some paper claims this improves icl

  lucidrains

  committed
  f65d186

  Copy full SHA for f65d186

• add ability to customize loss for continuous transformer wrapper

  lucidrains

  committed
  25b7639

  Copy full SHA for 25b7639

Commits on Nov 17, 2023

• 1.24.6

  lucidrains

  authored
  b348f76

  Copy full SHA for b348f76

Pagination

Previous

Next