Skip to content

Commit

Permalink
Update transformer mask comment
Browse files Browse the repository at this point in the history
Update names for consistency with code

Co-authored-by: ruanslv <[email protected]>
  • Loading branch information
flu0r1ne and ruanslv authored Nov 13, 2023
1 parent e9077bd commit 6b3154b
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions llama/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -481,8 +481,8 @@ def forward(self, tokens: torch.Tensor, start_pos: int):

# When performing key-value caching, we compute the attention scores
# only for the new sequence. Thus, the matrix of scores is of size
# (seq_len, total_len), and the only masked entries are (i, j) for
# j > cached_len + i, since row i corresponds to token cached_len + i.
# (seqlen, cache_len + seqlen), and the only masked entries are (i, j) for
# j > cache_len + i, since row i corresponds to token cache_len + i.
mask = torch.hstack([
torch.zeros((seqlen, start_pos), device=tokens.device),
mask
Expand Down

0 comments on commit 6b3154b

Please sign in to comment.