Skip to content

Commit

Permalink
mm/percpu: micro-optimize pcpu_is_populated()
Browse files Browse the repository at this point in the history
bitmap_next_clear_region() calls find_next_zero_bit() and find_next_bit()
sequentially to find a range of clear bits. In case of pcpu_is_populated()
there's a chance to return earlier if bitmap has all bits set.

Signed-off-by: Yury Norov <[email protected]>
Tested-by: Wolfram Sang <[email protected]>
Acked-by: Dennis Zhou <[email protected]>
  • Loading branch information
YuryNorov committed Jan 15, 2022
1 parent 749443d commit 801a573
Showing 1 changed file with 8 additions and 7 deletions.
15 changes: 8 additions & 7 deletions mm/percpu.c
Original file line number Diff line number Diff line change
Expand Up @@ -1070,17 +1070,18 @@ static void pcpu_block_update_hint_free(struct pcpu_chunk *chunk, int bit_off,
static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
int *next_off)
{
unsigned int page_start, page_end, rs, re;
unsigned int start, end;

page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);

rs = page_start;
bitmap_next_clear_region(chunk->populated, &rs, &re, page_end);
if (rs >= page_end)
start = find_next_zero_bit(chunk->populated, end, start);
if (start >= end)
return true;

*next_off = re * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
end = find_next_bit(chunk->populated, end, start + 1);

*next_off = end * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
return false;
}

Expand Down

0 comments on commit 801a573

Please sign in to comment.