forked from torvalds/linux
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
KVM: arm64: Optimise FPSIMD handling to reduce guest/host thrashing
This patch refactors KVM to align the host and guest FPSIMD
save/restore logic with each other for arm64. This reduces the
number of redundant save/restore operations that must occur, and
reduces the common-case IRQ blackout time during guest exit storms
by saving the host state lazily and optimising away the need to
restore the host state before returning to the run loop.
Four hooks are defined in order to enable this:
* kvm_arch_vcpu_run_map_fp():
Called on PID change to map necessary bits of current to Hyp.
* kvm_arch_vcpu_load_fp():
Set up FP/SIMD for entering the KVM run loop (parse as
"vcpu_load fp").
* kvm_arch_vcpu_ctxsync_fp():
Get FP/SIMD into a safe state for re-enabling interrupts after a
guest exit back to the run loop.
For arm64 specifically, this involves updating the host kernel's
FPSIMD context tracking metadata so that kernel-mode NEON use
will cause the vcpu's FPSIMD state to be saved back correctly
into the vcpu struct. This must be done before re-enabling
interrupts because kernel-mode NEON may be used by softirqs.
* kvm_arch_vcpu_put_fp():
Save guest FP/SIMD state back to memory and dissociate from the
CPU ("vcpu_put fp").
Also, the arm64 FPSIMD context switch code is updated to enable it
to save back FPSIMD state for a vcpu, not just current. A few
helpers drive this:
* fpsimd_bind_state_to_cpu(struct user_fpsimd_state *fp):
mark this CPU as having context fp (which may belong to a vcpu)
currently loaded in its registers. This is the non-task
equivalent of the static function fpsimd_bind_to_cpu() in
fpsimd.c.
* task_fpsimd_save():
exported to allow KVM to save the guest's FPSIMD state back to
memory on exit from the run loop.
* fpsimd_flush_state():
invalidate any context's FPSIMD state that is currently loaded.
Used to disassociate the vcpu from the CPU regs on run loop exit.
These changes allow the run loop to enable interrupts (and thus
softirqs that may use kernel-mode NEON) without having to save the
guest's FPSIMD state eagerly.
Some new vcpu_arch fields are added to make all this work. Because
host FPSIMD state can now be saved back directly into current's
thread_struct as appropriate, host_cpu_context is no longer used
for preserving the FPSIMD state. However, it is still needed for
preserving other things such as the host's system registers. To
avoid ABI churn, the redundant storage space in host_cpu_context is
not removed for now.
arch/arm is not addressed by this patch and continues to use its
current save/restore logic. It could provide implementations of
the helpers later if desired.
Signed-off-by: Dave Martin <[email protected]>
Reviewed-by: Marc Zyngier <[email protected]>
Reviewed-by: Christoffer Dall <[email protected]>
Reviewed-by: Alex Bennée <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>- Loading branch information
Dave Martin
authored and
Marc Zyngier
committed
May 25, 2018
1 parent
fa89d31
commit e6b673b
Showing
9 changed files
with
192 additions
and
31 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| // SPDX-License-Identifier: GPL-2.0 | ||
| /* | ||
| * arch/arm64/kvm/fpsimd.c: Guest/host FPSIMD context coordination helpers | ||
| * | ||
| * Copyright 2018 Arm Limited | ||
| * Author: Dave Martin <[email protected]> | ||
| */ | ||
| #include <linux/bottom_half.h> | ||
| #include <linux/sched.h> | ||
| #include <linux/thread_info.h> | ||
| #include <linux/kvm_host.h> | ||
| #include <asm/kvm_asm.h> | ||
| #include <asm/kvm_host.h> | ||
| #include <asm/kvm_mmu.h> | ||
|
|
||
| /* | ||
| * Called on entry to KVM_RUN unless this vcpu previously ran at least | ||
| * once and the most recent prior KVM_RUN for this vcpu was called from | ||
| * the same task as current (highly likely). | ||
| * | ||
| * This is guaranteed to execute before kvm_arch_vcpu_load_fp(vcpu), | ||
| * such that on entering hyp the relevant parts of current are already | ||
| * mapped. | ||
| */ | ||
| int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) | ||
| { | ||
| int ret; | ||
|
|
||
| struct thread_info *ti = ¤t->thread_info; | ||
| struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state; | ||
|
|
||
| /* | ||
| * Make sure the host task thread flags and fpsimd state are | ||
| * visible to hyp: | ||
| */ | ||
| ret = create_hyp_mappings(ti, ti + 1, PAGE_HYP); | ||
| if (ret) | ||
| goto error; | ||
|
|
||
| ret = create_hyp_mappings(fpsimd, fpsimd + 1, PAGE_HYP); | ||
| if (ret) | ||
| goto error; | ||
|
|
||
| vcpu->arch.host_thread_info = kern_hyp_va(ti); | ||
| vcpu->arch.host_fpsimd_state = kern_hyp_va(fpsimd); | ||
| error: | ||
| return ret; | ||
| } | ||
|
|
||
| /* | ||
| * Prepare vcpu for saving the host's FPSIMD state and loading the guest's. | ||
| * The actual loading is done by the FPSIMD access trap taken to hyp. | ||
| * | ||
| * Here, we just set the correct metadata to indicate that the FPSIMD | ||
| * state in the cpu regs (if any) belongs to current on the host. | ||
| * | ||
| * TIF_SVE is backed up here, since it may get clobbered with guest state. | ||
| * This flag is restored by kvm_arch_vcpu_put_fp(vcpu). | ||
| */ | ||
| void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) | ||
| { | ||
| BUG_ON(system_supports_sve()); | ||
| BUG_ON(!current->mm); | ||
|
|
||
| vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED | KVM_ARM64_HOST_SVE_IN_USE); | ||
| vcpu->arch.flags |= KVM_ARM64_FP_HOST; | ||
| if (test_thread_flag(TIF_SVE)) | ||
| vcpu->arch.flags |= KVM_ARM64_HOST_SVE_IN_USE; | ||
| } | ||
|
|
||
| /* | ||
| * If the guest FPSIMD state was loaded, update the host's context | ||
| * tracking data mark the CPU FPSIMD regs as dirty and belonging to vcpu | ||
| * so that they will be written back if the kernel clobbers them due to | ||
| * kernel-mode NEON before re-entry into the guest. | ||
| */ | ||
| void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) | ||
| { | ||
| WARN_ON_ONCE(!irqs_disabled()); | ||
|
|
||
| if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { | ||
| fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.gp_regs.fp_regs); | ||
| clear_thread_flag(TIF_FOREIGN_FPSTATE); | ||
| clear_thread_flag(TIF_SVE); | ||
| } | ||
| } | ||
|
|
||
| /* | ||
| * Write back the vcpu FPSIMD regs if they are dirty, and invalidate the | ||
| * cpu FPSIMD regs so that they can't be spuriously reused if this vcpu | ||
| * disappears and another task or vcpu appears that recycles the same | ||
| * struct fpsimd_state. | ||
| */ | ||
| void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) | ||
| { | ||
| local_bh_disable(); | ||
|
|
||
| update_thread_flag(TIF_SVE, | ||
| vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE); | ||
|
|
||
| if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { | ||
| /* Clean guest FP state to memory and invalidate cpu view */ | ||
| fpsimd_save(); | ||
| fpsimd_flush_cpu_state(); | ||
| } else if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { | ||
| /* Ensure user trap controls are correctly restored */ | ||
| fpsimd_bind_task_to_cpu(); | ||
| } | ||
|
|
||
| local_bh_enable(); | ||
| } |
Oops, something went wrong.