Skip to content

Commit

Permalink
fix index out of range on zero layer metal load (ollama#7696)
Browse files Browse the repository at this point in the history
If the model doesn't fit any layers on metal, and we load zero layers
we would panic trying to look up the GPU size during scheduling ops
  • Loading branch information
dhiltgen authored Nov 18, 2024
1 parent a14f764 commit 81d55d3
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion llm/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -1092,7 +1092,9 @@ func (s *llmServer) EstimatedTotal() uint64 {
func (s *llmServer) EstimatedVRAMByGPU(gpuID string) uint64 {
for i, gpu := range s.gpus {
if gpu.ID == gpuID {
return s.estimate.GPUSizes[i]
if i < len(s.estimate.GPUSizes) {
return s.estimate.GPUSizes[i]
}
}
}
return 0
Expand Down

0 comments on commit 81d55d3

Please sign in to comment.