implement x2APIC support

[michaeljclark]

this patch implements x2APIC support. it changes the order of processor initialization bringing up multiprocessor and calling mpmain much earlier and moving some peripheral initialization into postinit which still runs on CPU 0. it's more of an FYI at this point. there are a couple of ancillaries changes included in the branch such as adding sudo to the loop filesystem setup and adding -p to mkdir as they were required to get make binaries to work locally. there is also a change to make use typedef long long uint64 otherwise long compiles as a 32-bit type with -m32.

irq_register_handler and get_registered_handler were made to subtract T_IRQ0 so that they are consistent with the use of IRQ numbers in the other interrupt APIs. it was necessary to change get_registered_handler as otherwise, it would not find traps, noting that dynamic IRQs were not in use. this small patch can be added on top of this branch to test IPI support by creating a dynamic IRQ function and sending some IPIs in mpmain.

diff --git a/kernel/main.c b/kernel/main.c
index cbce2c9a7a6c..a583e3529dd1 100644
--- a/kernel/main.c
+++ b/kernel/main.c
@@ -78,6 +78,11 @@ void mpenter(void){
        mpmain();
 }
 
+static void testirq(uint16 irq)
+{
+       cprintf("cpu-%d: received interrupt: %d\n", cpu->id, irq);
+}
+
 // Common CPU setup code.
 static void mpmain(void){
        idtinit(); // load idt register
@@ -87,6 +92,9 @@ static void mpmain(void){
                cprintf("%d-way SMP kernel online.\n", ncpu);
                postinit();
                cprintf("starting scheduler\n");
+               irq_register_handler(T_IRQ0 + 5, testirq);
+               for (int i = 0; i < ncpu; i++)
+                       lapicsendipi(T_IRQ0 + 5, i);
        }
        amd64_xchg(&cpu->started, 1); // tell startothers() we're up
        scheduler(); // start running processes

identcpu and lapicinit have been modified to log CPU model and APIC details together during processor initialization. TSC frequency is detected on bare metal, in VMWare and KVM but it may be necessary to manually provide the TSC frequency with KVM. microdelay has been updated to use the TSC base frequency.

note run.sh has been changed to add a -x flag which passes -machine pc,accel=kvm,kernel-irqchip=on -cpu host,vmware-cpuid-freq=on,tsc-frequency=2600000000 to qemu which will enable the KVM kernel module x2APIC support.

here is the boot log after applying the patches in this branch (tested with and without x2APIC):

SeaBIOS (version 1.13.0-1ubuntu1.1)
Booting from Hard Disk..Xv64 UART Console 1
Freeing mem from 50 MB to 54 MB...
acpi: cpu#0 apicid 0
acpi: cpu#1 apicid 1
acpi: cpu#2 apicid 2
acpi: cpu#3 apicid 3
acpi: ioapic#0 @fec00000 id=0 base=0
cpu-0: name: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz vendor: GenuineIntel model: 22 tscfreq: 2600
cpu-0: lapicinit: lapic#0 @fee00000, version=0x14, maxlvt=5, xapic=1, x2apic=1, bsp=1
cpu-1: name: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz vendor: GenuineIntel model: 22 tscfreq: 2600
cpu-1: lapicinit: lapic#1 @fee00000, version=0x14, maxlvt=5, xapic=1, x2apic=1, bsp=0
cpu-2: name: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz vendor: GenuineIntel model: 22 tscfreq: 2600
cpu-2: lapicinit: lapic#2 @fee00000, version=0x14, maxlvt=5, xapic=1, x2apic=1, bsp=0
cpu-3: name: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz vendor: GenuineIntel model: 22 tscfreq: 2600
cpu-3: lapicinit: lapic#3 @fee00000, version=0x14, maxlvt=5, xapic=1, x2apic=1, bsp=0
Freeing mem from 54 MB to 224 MB...
4-way SMP kernel online.
CGA COLOR Console

cpu0: starting Xv64

Xv64 is Copyright (c) 1997-2021, contributors.
See LICENSE for details.
init: process loop device
probing PCI...
PCI: 0:0.0: 8086:1237: class: 6.0 (Bridge device) irq: 0
PCI: 0:1.0: 8086:7000: class: 6.1 (Bridge device) irq: 0
PCI: 0:1.1: 8086:7010: class: 1.1 (Storage controller) irq: 0
PCI: 0:1.3: 8086:7113: class: 6.80 (Bridge device) irq: 9
PCI: 0:2.0: 1234:1111: class: 3.0 (Display controller) irq: 0
PCI: 0:3.0: 8086:2922: class: 1.6 (Storage controller) irq: 11 bar5: febf1000
Intel ICH9 controller found (bus=0, slot=3, func=0, abar=0xfebf1000)
   HBA in AHCI-only mode
SATA device detected:
   port[0].sig = 101
   ipm=1, spd=1, det=3
   rebasing port...DONE
   Init success: disk(1, 0)
Detecting IDE devices:
   no IDE devices detected
Root dev: disk(1, 0)
xv6 filesystem assumed on disk(1,0)
starting scheduler
init: starting...
init: starting kidle
init: starting sh

Welcome to Xv64
~> halt
Xv64 is shutting down now. Goodbye...

[jwhitehorn]

oh, wow!

Thank you for all this hard work. Sorry for not replying sooner - I just now saw this PR.

Let me test this and I'll get back with you.

[michaeljclark]

On 8/03/2022, at 9:10 AM, Jason Whitehorn ***@***.***> wrote:  oh, wow! Thank you for all this hard work. Sorry for not replying sooner - I just now saw this PR. Let me test this and I'll get back with you.

You’re welcome. Thanks. Take your time. I thought Xv64 would be useful for timing IPIs which is something I’ve wanted to do for a while which is why I worked on this. x2APIC should result in slightly faster interrupts within a VM because MSRs are easier to intercept than MMIO accesses.