tanh(C_t) is not in the final paper

[twoletters]

xLSTM/xlstm_torch/main.py

Line 96 in 020209f

h_t = self.sigmoid(o_tilda) * torch.tanh(c_hat)

On this line, c_hat is passed through tanh, but in the final paper, h_tilda (a.k.a. c_hat here) is only c_t / n_t, so h_t should be:

h_t = self.sigmoid(o_tilda) * c_hat

Excerpt from Section A.2 of the final paper:

Here, the cell input activation function φ is tanh, the hidden state activation function is the identity. φ helps stabilizing the recurrence.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[github-actions]

Stale issue message