Skip to content

Commit

Permalink
Merge branch 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/lin…
Browse files Browse the repository at this point in the history 
…ux/kernel/git/tip/tip

Pull x86 APIC updates from Thomas Gleixner:
 "This update provides a major overhaul of the APIC initialization and
  vector allocation code:

   - Unification of the APIC and interrupt mode setup which was
     scattered all over the place and was hard to follow. This also
     distangles the timer setup from the APIC initialization which
     brings a clear separation of functionality.

     Great detective work from Dou Lyiang!

   - Refactoring of the x86 vector allocation mechanism. The existing
     code was based on nested loops and rather convoluted APIC callbacks
     which had a horrible worst case behaviour and tried to serve all
     different use cases in one go. This led to quite odd hacks when
     supporting the new managed interupt facility for multiqueue devices
     and made it more or less impossible to deal with the vector space
     exhaustion which was a major roadblock for server hibernation.

     Aside of that the code dealing with cpu hotplug and the system
     vectors was disconnected from the actual vector management and
     allocation code, which made it hard to follow and maintain.

     Utilizing the new bitmap matrix allocator core mechanism, the new
     allocator and management code consolidates the handling of system
     vectors, legacy vectors, cpu hotplug mechanisms and the actual
     allocation which needs to be aware of system and legacy vectors and
     hotplug constraints into a single consistent entity.

     This has one visible change: The support for multi CPU targets of
     interrupts, which is only available on a certain subset of
     CPUs/APIC variants has been removed in favour of single interrupt
     targets. A proper analysis of the multi CPU target feature revealed
     that there is no real advantage as the vast majority of interrupts
     end up on the CPU with the lowest APIC id in the set of target CPUs
     anyway. That change was agreed on by the relevant folks and allowed
     to simplify the implementation significantly and to replace rather
     fragile constructs like the vector cleanup IPI with straight
     forward and solid code.

     Furthermore this allowed to cleanly separate the allocation details
     for legacy, normal and managed interrupts:

      * Legacy interrupts are not longer wasting 16 vectors
        unconditionally

      * Managed interrupts have now a guaranteed vector reservation, but
        the actual vector assignment happens when the interrupt is
        requested. It's guaranteed not to fail.

      * Normal interrupts no longer allocate vectors unconditionally
        when the interrupt is set up (IO/APIC init or MSI(X) enable).
        The mechanism has been switched to a best effort reservation
        mode. The actual allocation happens when the interrupt is
        requested. Contrary to managed interrupts the request can fail
        due to vector space exhaustion, but drivers must handle a fail
        of request_irq() anyway. When the interrupt is freed, the vector
        is handed back as well.

        This solves a long standing problem with large unconditional
        vector allocations for a certain class of enterprise devices
        which prevented server hibernation due to vector space
        exhaustion when the unused allocated vectors had to be migrated
        to CPU0 while unplugging all non boot CPUs.

     The code has been equipped with trace points and detailed debugfs
     information to aid analysis of the vector space"

* 'x86-apic-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (60 commits)
  x86/vector/msi: Select CONFIG_GENERIC_IRQ_RESERVATION_MODE
  PCI/MSI: Set MSI_FLAG_MUST_REACTIVATE in core code
  genirq: Add config option for reservation mode
  x86/vector: Use correct per cpu variable in free_moved_vector()
  x86/apic/vector: Ignore set_affinity call for inactive interrupts
  x86/apic: Fix spelling mistake: "symmectic" -> "symmetric"
  x86/apic: Use dead_cpu instead of current CPU when cleaning up
  ACPI/init: Invoke early ACPI initialization earlier
  x86/vector: Respect affinity mask in irq descriptor
  x86/irq: Simplify hotplug vector accounting
  x86/vector: Switch IOAPIC to global reservation mode
  x86/vector/msi: Switch to global reservation mode
  x86/vector: Handle managed interrupts proper
  x86/io_apic: Reevaluate vector configuration on activate()
  iommu/amd: Reevaluate vector configuration on activate()
  iommu/vt-d: Reevaluate vector configuration on activate()
  x86/apic/msi: Force reactivation of interrupts at startup time
  x86/vector: Untangle internal state from irq_cfg
  x86/vector: Compile SMP only code conditionally
  x86/apic: Remove unused callbacks
  ...
  • Loading branch information
@torvalds
torvalds committed Nov 14, 2017
2 parents 7d58e1c + 141d3b1 commit b18d628
Show file tree
Hide file tree
Showing 43 changed files with 1,493 additions and 1,317 deletions.
2 changes: 2 additions & 0 deletions arch/x86/Kconfig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,10 @@ config X86
select GENERIC_FIND_FIRST_BIT
select GENERIC_IOMAP
select GENERIC_IRQ_EFFECTIVE_AFF_MASK if SMP
select GENERIC_IRQ_MATRIX_ALLOCATOR if X86_LOCAL_APIC
select GENERIC_IRQ_MIGRATION if SMP
select GENERIC_IRQ_PROBE
select GENERIC_IRQ_RESERVATION_MODE
select GENERIC_IRQ_SHOW
select GENERIC_PENDING_IRQ if SMP
select GENERIC_SMP_IDLE_THREAD
Expand Down
270 changes: 86 additions & 184 deletions arch/x86/include/asm/apic.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,15 @@ extern int local_apic_timer_c2_ok;
extern int disable_apic;
extern unsigned int lapic_timer_frequency;

extern enum apic_intr_mode_id apic_intr_mode;
enum apic_intr_mode_id {
APIC_PIC,
APIC_VIRTUAL_WIRE,
APIC_VIRTUAL_WIRE_NO_CONFIG,
APIC_SYMMETRIC_IO,
APIC_SYMMETRIC_IO_NO_ROUTING
};

#ifdef CONFIG_SMP
extern void __inquire_remote_apic(int apicid);
#else /* CONFIG_SMP */
Expand Down Expand Up @@ -127,14 +136,13 @@ extern void disconnect_bsp_APIC(int virt_wire_setup);
extern void disable_local_APIC(void);
extern void lapic_shutdown(void);
extern void sync_Arb_IDs(void);
extern void init_bsp_APIC(void);
extern void apic_intr_mode_init(void);
extern void setup_local_APIC(void);
extern void init_apic_mappings(void);
void register_lapic_address(unsigned long address);
extern void setup_boot_APIC_clock(void);
extern void setup_secondary_APIC_clock(void);
extern void lapic_update_tsc_freq(void);
extern int APIC_init_uniprocessor(void);

#ifdef CONFIG_X86_64
static inline int apic_force_enable(unsigned long addr)
Expand All @@ -145,7 +153,7 @@ static inline int apic_force_enable(unsigned long addr)
extern int apic_force_enable(unsigned long addr);
#endif

extern int apic_bsp_setup(bool upmode);
extern void apic_bsp_setup(bool upmode);
extern void apic_ap_setup(void);

/*
Expand All @@ -161,6 +169,10 @@ static inline int apic_is_clustered_box(void)
#endif

extern int setup_APIC_eilvt(u8 lvt_off, u8 vector, u8 msg_type, u8 mask);
extern void lapic_assign_system_vectors(void);
extern void lapic_assign_legacy_vector(unsigned int isairq, bool replace);
extern void lapic_online(void);
extern void lapic_offline(void);

#else /* !CONFIG_X86_LOCAL_APIC */
static inline void lapic_shutdown(void) { }
Expand All @@ -170,6 +182,9 @@ static inline void disable_local_APIC(void) { }
# define setup_boot_APIC_clock x86_init_noop
# define setup_secondary_APIC_clock x86_init_noop
static inline void lapic_update_tsc_freq(void) { }
static inline void apic_intr_mode_init(void) { }
static inline void lapic_assign_system_vectors(void) { }
static inline void lapic_assign_legacy_vector(unsigned int i, bool r) { }
#endif /* !CONFIG_X86_LOCAL_APIC */

#ifdef CONFIG_X86_X2APIC
Expand Down Expand Up @@ -265,73 +280,63 @@ struct irq_data;
* James Cleverdon.
*/
struct apic {
char *name;

int (*probe)(void);
int (*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
int (*apic_id_valid)(int apicid);
int (*apic_id_registered)(void);

u32 irq_delivery_mode;
u32 irq_dest_mode;

const struct cpumask *(*target_cpus)(void);

int disable_esr;

int dest_logical;
unsigned long (*check_apicid_used)(physid_mask_t *map, int apicid);

void (*vector_allocation_domain)(int cpu, struct cpumask *retmask,
const struct cpumask *mask);
void (*init_apic_ldr)(void);

void (*ioapic_phys_id_map)(physid_mask_t *phys_map, physid_mask_t *retmap);

void (*setup_apic_routing)(void);
int (*cpu_present_to_apicid)(int mps_cpu);
void (*apicid_to_cpu_present)(int phys_apicid, physid_mask_t *retmap);
int (*check_phys_apicid_present)(int phys_apicid);
int (*phys_pkg_id)(int cpuid_apic, int index_msb);

unsigned int (*get_apic_id)(unsigned long x);
/* Can't be NULL on 64-bit */
unsigned long (*set_apic_id)(unsigned int id);

int (*cpu_mask_to_apicid)(const struct cpumask *cpumask,
struct irq_data *irqdata,
unsigned int *apicid);

/* ipi */
void (*send_IPI)(int cpu, int vector);
void (*send_IPI_mask)(const struct cpumask *mask, int vector);
void (*send_IPI_mask_allbutself)(const struct cpumask *mask,
int vector);
void (*send_IPI_allbutself)(int vector);
void (*send_IPI_all)(int vector);
void (*send_IPI_self)(int vector);
/* Hotpath functions first */
void (*eoi_write)(u32 reg, u32 v);
void (*native_eoi_write)(u32 reg, u32 v);
void (*write)(u32 reg, u32 v);
u32 (*read)(u32 reg);

/* IPI related functions */
void (*wait_icr_idle)(void);
u32 (*safe_wait_icr_idle)(void);

void (*send_IPI)(int cpu, int vector);
void (*send_IPI_mask)(const struct cpumask *mask, int vector);
void (*send_IPI_mask_allbutself)(const struct cpumask *msk, int vec);
void (*send_IPI_allbutself)(int vector);
void (*send_IPI_all)(int vector);
void (*send_IPI_self)(int vector);

/* dest_logical is used by the IPI functions */
u32 dest_logical;
u32 disable_esr;
u32 irq_delivery_mode;
u32 irq_dest_mode;

/* Functions and data related to vector allocation */
void (*vector_allocation_domain)(int cpu, struct cpumask *retmask,
const struct cpumask *mask);
int (*cpu_mask_to_apicid)(const struct cpumask *cpumask,
struct irq_data *irqdata,
unsigned int *apicid);
u32 (*calc_dest_apicid)(unsigned int cpu);

/* ICR related functions */
u64 (*icr_read)(void);
void (*icr_write)(u32 low, u32 high);

/* Probe, setup and smpboot functions */
int (*probe)(void);
int (*acpi_madt_oem_check)(char *oem_id, char *oem_table_id);
int (*apic_id_valid)(int apicid);
int (*apic_id_registered)(void);

bool (*check_apicid_used)(physid_mask_t *map, int apicid);
void (*init_apic_ldr)(void);
void (*ioapic_phys_id_map)(physid_mask_t *phys_map, physid_mask_t *retmap);
void (*setup_apic_routing)(void);
int (*cpu_present_to_apicid)(int mps_cpu);
void (*apicid_to_cpu_present)(int phys_apicid, physid_mask_t *retmap);
int (*check_phys_apicid_present)(int phys_apicid);
int (*phys_pkg_id)(int cpuid_apic, int index_msb);

u32 (*get_apic_id)(unsigned long x);
u32 (*set_apic_id)(unsigned int id);

/* wakeup_secondary_cpu */
int (*wakeup_secondary_cpu)(int apicid, unsigned long start_eip);
int (*wakeup_secondary_cpu)(int apicid, unsigned long start_eip);

void (*inquire_remote_apic)(int apicid);

/* apic ops */
u32 (*read)(u32 reg);
void (*write)(u32 reg, u32 v);
/*
* ->eoi_write() has the same signature as ->write().
*
* Drivers can support both ->eoi_write() and ->write() by passing the same
* callback value. Kernel can override ->eoi_write() and fall back
* on write for EOI.
*/
void (*eoi_write)(u32 reg, u32 v);
void (*native_eoi_write)(u32 reg, u32 v);
u64 (*icr_read)(void);
void (*icr_write)(u32 low, u32 high);
void (*wait_icr_idle)(void);
u32 (*safe_wait_icr_idle)(void);
void (*inquire_remote_apic)(int apicid);

#ifdef CONFIG_X86_32
/*
Expand All @@ -346,6 +351,7 @@ struct apic {
*/
int (*x86_32_early_logical_apicid)(int cpu);
#endif
char *name;
};

/*
Expand Down Expand Up @@ -380,6 +386,7 @@ extern struct apic *__apicdrivers[], *__apicdrivers_end[];
*/
#ifdef CONFIG_SMP
extern int wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip);
extern int lapic_can_unplug_cpu(void);
#endif

#ifdef CONFIG_X86_LOCAL_APIC
Expand Down Expand Up @@ -463,156 +470,51 @@ static inline unsigned default_get_apic_id(unsigned long x)
extern void apic_send_IPI_self(int vector);

DECLARE_PER_CPU(int, x2apic_extra_bits);

extern int default_cpu_present_to_apicid(int mps_cpu);
extern int default_check_phys_apicid_present(int phys_apicid);
#endif

extern void generic_bigsmp_probe(void);


#ifdef CONFIG_X86_LOCAL_APIC

#include <asm/smp.h>

#define APIC_DFR_VALUE (APIC_DFR_FLAT)

static inline const struct cpumask *default_target_cpus(void)
{
#ifdef CONFIG_SMP
return cpu_online_mask;
#else
return cpumask_of(0);
#endif
}

static inline const struct cpumask *online_target_cpus(void)
{
return cpu_online_mask;
}

DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_bios_cpu_apicid);

extern struct apic apic_noop;

static inline unsigned int read_apic_id(void)
{
unsigned int reg;

reg = apic_read(APIC_ID);
unsigned int reg = apic_read(APIC_ID);

return apic->get_apic_id(reg);
}

static inline int default_apic_id_valid(int apicid)
{
return (apicid < 255);
}

extern int default_apic_id_valid(int apicid);
extern int default_acpi_madt_oem_check(char *, char *);

extern void default_setup_apic_routing(void);

extern struct apic apic_noop;

#ifdef CONFIG_X86_32

static inline int noop_x86_32_early_logical_apicid(int cpu)
{
return BAD_APICID;
}

/*
* Set up the logical destination ID.
*
* Intel recommends to set DFR, LDR and TPR before enabling
* an APIC. See e.g. "AP-388 82489DX User's Manual" (Intel
* document number 292116). So here it goes...
*/
extern void default_init_apic_ldr(void);

static inline int default_apic_id_registered(void)
{
return physid_isset(read_apic_id(), phys_cpu_present_map);
}

static inline int default_phys_pkg_id(int cpuid_apic, int index_msb)
{
return cpuid_apic >> index_msb;
}

#endif
extern u32 apic_default_calc_apicid(unsigned int cpu);
extern u32 apic_flat_calc_apicid(unsigned int cpu);

extern int flat_cpu_mask_to_apicid(const struct cpumask *cpumask,
struct irq_data *irqdata,
unsigned int *apicid);
extern int default_cpu_mask_to_apicid(const struct cpumask *cpumask,
struct irq_data *irqdata,
unsigned int *apicid);

static inline void
flat_vector_allocation_domain(int cpu, struct cpumask *retmask,
const struct cpumask *mask)
{
/* Careful. Some cpus do not strictly honor the set of cpus
* specified in the interrupt destination when using lowest
* priority interrupt delivery mode.
*
* In particular there was a hyperthreading cpu observed to
* deliver interrupts to the wrong hyperthread when only one
* hyperthread was specified in the interrupt desitination.
*/
cpumask_clear(retmask);
cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}

static inline void
default_vector_allocation_domain(int cpu, struct cpumask *retmask,
const struct cpumask *mask)
{
cpumask_copy(retmask, cpumask_of(cpu));
}

static inline unsigned long default_check_apicid_used(physid_mask_t *map, int apicid)
{
return physid_isset(apicid, *map);
}

static inline void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap)
{
*retmap = *phys_map;
}

static inline int __default_cpu_present_to_apicid(int mps_cpu)
{
if (mps_cpu < nr_cpu_ids && cpu_present(mps_cpu))
return (int)per_cpu(x86_bios_cpu_apicid, mps_cpu);
else
return BAD_APICID;
}

static inline int
__default_check_phys_apicid_present(int phys_apicid)
{
return physid_isset(phys_apicid, phys_cpu_present_map);
}

#ifdef CONFIG_X86_32
static inline int default_cpu_present_to_apicid(int mps_cpu)
{
return __default_cpu_present_to_apicid(mps_cpu);
}

static inline int
default_check_phys_apicid_present(int phys_apicid)
{
return __default_check_phys_apicid_present(phys_apicid);
}
#else
extern bool default_check_apicid_used(physid_mask_t *map, int apicid);
extern void flat_vector_allocation_domain(int cpu, struct cpumask *retmask,
const struct cpumask *mask);
extern void default_vector_allocation_domain(int cpu, struct cpumask *retmask,
const struct cpumask *mask);
extern void default_ioapic_phys_id_map(physid_mask_t *phys_map, physid_mask_t *retmap);
extern int default_cpu_present_to_apicid(int mps_cpu);
extern int default_check_phys_apicid_present(int phys_apicid);
#endif

#endif /* CONFIG_X86_LOCAL_APIC */

extern void irq_enter(void);
extern void irq_exit(void);

Expand Down
2 changes: 1 addition & 1 deletion arch/x86/include/asm/desc.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)
void update_intr_gate(unsigned int n, const void *addr);
void alloc_intr_gate(unsigned int n, const void *addr);

extern unsigned long used_vectors[];
extern unsigned long system_vectors[];

#ifdef CONFIG_X86_64
DECLARE_PER_CPU(u32, debug_idt_ctr);
Expand Down
Loading

0 comments on commit b18d628

Please sign in to comment.