|
Description
|
Category
kernel
Sub-Category
arch-x86
Description
When trying to boot any of these MP systems as a diskless client (root
filesystem on NFS), the boot process stops when the first non-boot-cpu is
started. System is stuck and never completes the boot, and has to be
rebooted.
See also this www.opensolaris.org thrad:
http://www.opensolaris.org/jive/thread.jspa?threadID=50608&tstart=0
Apparently interrupts are re-distributed to the non-boot-cpu
early during mp_startup, the non-boot-cpu has it's SPL still
raised to LOCK_LEVEL, and the microcode update driver tries to
load files from the root filesystem or cpu driver kernel modules
are loaded for the new non-boot-cpu. The file I/O triggers NIC
device interrupts. The NIC device interrupts are never serviced
because of SPL == LOCK_LEVEL.
Config file for the PV domU is something like this:
# cat /etc/xen/solaris-nfs
name = "solaris"
memory = 512
vcpus = 1
localtime = 1
kernel = '/export/root/xen/platform/i86xpv/kernel/amd64/unix'
ramdisk = '/export/root/xen/platform/i86pc/amd64/boot_archive'
extra = '/platform/i86xpv/kernel/amd64/unix -kv'
nfs_server = '192.168.2.20'
nfs_root = 'moritz:/export/root/xen'
vif = [ 'mac=0:8:c7:d2:82:63' ]
on_shutdown = 'destroy'
on_reboot = 'destroy'
on_crash = 'destroy'
With this configuration (note: vcpus=1) the snv_83 domU boots ok.
But with vcpus=2, the domU hangs during the boot, after / while
cpu#1 is initialized:
# xm create solaris-nfs -c vcpus=2
Using config file "/etc/xen/solaris-nfs".
Started domain solaris
module /platform/i86xpv/kernel/amd64/unix: text at [0xfffffffffb800000, 0xfffffffffb9205f3] data at 0xfffffffffbc00000
module /kernel/amd64/genunix: text at [0xfffffffffb920600, 0xfffffffffbb55047] data at 0xfffffffffbca2000
Loading kmdb...
module /kernel/misc/amd64/kmdbmod: text at [0xfffffffffbb55050, 0xfffffffffbbe42bf] data at 0xfffffffffbd0aa70
module /kernel/misc/amd64/ctf: text at [0xfffffffffbbe42c0, 0xfffffffffbbee1bf] data at 0xfffffffffbd25d40
v3.0.4-1-xvm chgset 'Mon Oct 29 23:19:32 2007 -0700 13226:b0c4d9d28fee'
SunOS Release 5.11 Version wos_b83 64-bit
Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
features: 10e66c6<cpuid,cx16,sse3,nx,sse2,sse,cx8,pae,mmx,cmov,msr,tsc>
mem = 524288K (0x20000000)
root nexus = i86xpv
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
pseudo-device: dld0
dld0 is /pseudo/dld at 0
xpvd0 at root
/xpvd/xnf at 0 (xnf0) online
Requesting Internet address for 0:8:c7:d2:82:63
Found my IP address: 2802a8c0 (192.168.2.40)
hostname: xen
domainname: gandalf.net
root on moritz:/export/root/xen fstype nfsdyn
/cpus (cpunex0) online
xencons at 0, xencons0
xencons0 is /xpvd/xencons at 0
cpu0: x86 (AuthenticAMD 40FB2 family 15 model 75 step 2 clock 2210 MHz)
cpu0: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
cpu1: x86 (AuthenticAMD 40FB2 family 15 model 75 step 2 clock 2210 MHz)
cpu1: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
NFS server moritz not responding still trying
<<<<< hangs here / no more progress >>>>>
Under xVM, this happens:
- mp_startup() is called to startup cpu#1
- in mp_startup(), "(*ap_mlsetup)()" is called,
which calls xen's xen_psm_post_cpu_start()
- in xen_psm_post_cpu_start() we have this:
/*
* Re-distribute interrupts to include the newly added cpu.
*/
xen_psm_enable_intr(cpun);
In my setup, this re-binds netfront's interrupt handler
xnf`xnf_intr() from cpu0 to the new cpu1.
(This might have changed in snv_77, with the fix for
6611846 "after boot, all dom0 interrupts are targeting
CPU 0 in a MP system" - this could explain why it
did work for you ~6 month ago).
- later on, in mp_setup() it raises the spl for the new cpu1
to LOCK_LEVEL, and enables interrupts. But at
spl == LOCK_LEVEL, xnf_intr should be masked.
add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi)
is called. This tries to load & attach the "cpudrv" kernel
module (while we're still at spl == LOCK_LEVEL on cpu1).
It sends packes out of the domU, but the replies from
the NFS server are never seen by xnf`xnf_intr, which is
masked.
When the domU is hung, it see this:
[1]> ::cpuinfo -v
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc3fff0 1b 0 0 -1 no no t-0 ffffff0001005c80
(idle)
|
RUNNING <--+
READY
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
1 ffffff0086199ac0 1b 0 10 60 no no t-0 ffffff00010cbc80
|
RUNNING <--+
READY
EXISTS
ENABLE
[1]> ::interrupts
IRQ Vect Evtchn IPL Bus Trg Type CPU Share APIC/INT# ISR(s)
256 - I 15 - Edg ipi all - - xc_serv
257 - I 13 - Edg ipi all - - xc_serv
258 - I 11 - Edg ipi all - - poke_cpu
259 - 1 15 - Edg virq all - - xen_debug_handler
260 - 1 1 - Edg evtchn 0 - - xenbus_intr
261 - T 14 - Edg virq all - - cbe_fire
262 - I 14 - Edg ipi all - - cbe_fire
263 - 9 6 xpvd Edg evtchn 1 - - xnf`xnf_intr
264 - 2 9 xpvd Edg evtchn 0 - - xencons`xenconsintr
[1]> ::evtchns
Type Evtchn IRQ IPL CPU Masked Pending ISR(s)
evtchn 1 260 1 0 0 0 xenbus_intr
evtchn 2 264 9 0 0 1 xencons`xenconsintr
ipi 3 256 15 0 1 0 xc_serv
ipi 4 257 13 0 0 0 xc_serv
ipi 5 258 11 0 0 0 poke_cpu
virq:debug 6 259 15 0 0 0 xen_debug_handler
virq:timer 7 261 14 0 1 1 cbe_fire
ipi 8 262 14 0 0 0 cbe_fire
evtchn 9 263 6 1 1 1 xnf`xnf_intr
ipi 10 258 11 1 0 0 poke_cpu
ipi 11 257 13 1 0 0 xc_serv
ipi 12 262 14 1 0 0 cbe_fire
ipi 13 256 15 1 0 0 xc_serv
virq:timer 14 261 14 1 1 1 cbe_fire
A possible fix could be to move the add_cpunode2devtree()
call down a few lines in mp_startup(), after the spl0():
diff -r f6814e9b7def usr/src/uts/i86pc/os/mp_startup.c
--- a/usr/src/uts/i86pc/os/mp_startup.c Wed Jan 30 09:01:17 2008 -0800
+++ b/usr/src/uts/i86pc/os/mp_startup.c Thu Jan 31 01:00:58 2008 +0100
@@ -1518,13 +1518,15 @@ mp_startup(void)
*/
curthread->t_preempt = 0;
- add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi);
+ /* add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi); */
/* The base spl should still be at LOCK LEVEL here */
ASSERT(cp->cpu_base_spl == ipltospl(LOCK_LEVEL));
set_base_spl(); /* Restore the spl to its proper value */
(void) spl0(); /* enable interrupts */
+
+ add_cpunode2devtree(cp->cpu_id, cp->cpu_m.mcpu_cpi);
#ifndef __xpv
{
Frequency
Always
Regression
Solaris 10
Steps to Reproduce
Try to boot a diskless x86 MP system
Expected Result
system boots
Actual Result
OS hangs when starting non-boot cpus.
Error Message(s)
None, OS hangs.
In some case, "NFS server XXX not responding still trying"
Test Case
Workaround
Boot with "use_mp=0".
Submitter wants to work on bug
No
Additional configuration information
Test system 1 (xVM PV domU case):
- AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ / 2GB memory
- dom0 running opensolaris snv_83 bits (64-bit)
- PV domU opensolaris snv_83 bits (64-bit), with an nfs root filesystem
Test system 2 (xen/xVM PV domU case):
- Core(TM)2 Quad CPU Q6600 / 8GB memory
- dom0 running gentoo 2.6.20 linux (32-bit)
- PV domU opensolaris snv_83 bits (32-bit), with an nfs root filesystem
Test system 3 (metal, no xen):
- Intel(r) CPU T2500 @ 2.00GHz, dual core cpu
- 2GB memory
- Mainboard: N4L-VM-ASUS-DH
|
