OpenSolaris

Ok this one is weird. With fast CPUs we don't get CPU saturated by interrupts.
So it's generally better to not use soft rings. Under that conditions, there are
less gains to be expected from having a high interrupt blanking time and there 
are great gains for ping/pong workloads (e.g. single thread random read) to having a smaller value.

However when we set "rxdma-intr-time = 1;" in /kernel/drv/nxge.conf we observe
that a NIC comes up with the expected value as reported by ndd :

	ar02# ndd -get /dev/nxge4 rxdma-intr-time
	1

However the ping/pong rate is much lower than expected. If we then 
do
	ar02# ndd -set /dev/nxge4 rxdma-intr-time 8
	ar02# ndd -set /dev/nxge4 rxdma-intr-time 1

This speeds up the rate considerably. unloading the nxge module returns to the situation
in which ndd reports rxdma-intr-time = 1 but ping pong rate behaves as if the blanking was 8.

So between 2 system each with nxge nics. I count ping/pong per second.

With Peer's rxdma-intr-time = 1

After boot (with rxdma-intr-time = 1 in /kernel/drv/nxge.conf) :

	 ndd -get /dev/nxge4 rxdma-intr-time
	 1
	Rate : 4580 pp/sec.

	 ndd -set /dev/nxge4 rxdma-intr-time 1
	Rate : 4580 pp/sec

	 ndd -set /dev/nxge4 rxdma-intr-time 8
	Rate : 4580 pp/sec

	 ndd -set /dev/nxge4 rxdma-intr-time 1
	Rate :12000 pp/sec

What this looks like is that the nxge.conf value was read by the driver at modload but did not
setup the NIC with it. Then when we tune it to be 1, the driver thinks there is nothing to do
and does not change the NIC settings. Then when we set it to 8, the driver setups the NIC with 
the value 8 which is already what's in there, so the ping pong rate still does not change. Lastly 
we set it to 1 and the driver finally set the NIC to have the proper interrupt blanking time
giving a 2X+ boost.

N/A

N/A