Skip to content

Commit

Permalink
llama : remember and restore kv cache data pointers (ggerganov#1104)
Browse files Browse the repository at this point in the history
because their value is stored in buf and overwritten by memcpy
  • Loading branch information
xaedes authored Apr 21, 2023
1 parent 1bfc153 commit 8687c1f
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions llama.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2092,7 +2092,11 @@ void llama_set_kv_cache(
int n_token_count) {
// Make sure we have the same kv cache setup
LLAMA_ASSERT(ctx->model.kv_self.buf.size == n_size);
void * k_data = ctx->model.kv_self.k->data; // remember data pointers
void * v_data = ctx->model.kv_self.v->data; // because their value is stored in buf and overwritten by memcpy
memcpy(ctx->model.kv_self.buf.addr, kv_cache, n_size);
ctx->model.kv_self.k->data = k_data; // restore correct data pointers
ctx->model.kv_self.v->data = v_data;
ctx->model.kv_self.n = n_token_count;
}

Expand Down

0 comments on commit 8687c1f

Please sign in to comment.