I see data corruption during transfers over a loopback. I don't see such problem when sending data over a pipe only, nor when sending it over the network. S10 doesn't seem to be affected, it looks Nevada only. I can hit it on the latest Nevada ON code (will be snv_105) as well as on snv_98.
I first hit the problem when running STC-2 SSH test suite since it does transfer significant amount of data over a loopback. So, I wrote simple reader/write apps (attached - you must use them on the same architecture) that produce and read counters. I tested on x86 with nc(1) and the corruption seems to be only at the 1st byte from every chunk sent (that's why SSH always reports "Bad packet length" instead of "Bad packet MAC" that would happen if any byte could be corrupted).
run it like this (consumer first):
$ while :; do ./producer | nc localhost 2222; sleep 5; done
pid 284479: producing sorted counters...
pid 284486: producing sorted counters...
pid 284827: producing sorted counters...
pid 284919: producing sorted counters...
pid 284924: producing sorted counters...
$ while :; do nc -l 2222 | ./consumer; done
pid 284478: reading counters...
consumer: expected 21256883, got 21256897
pid 284485: reading counters...
consumer: expected 154691202, got 154691204
pid 284824: reading counters...
consumer: expected 20800113, got 20800114
pid 284917: reading counters...
consumer: expected 9614427, got 9614455
pid 284923: reading counters...
the corruption seems random, you can see it can be hit in 9M of transmitted buffers (not data) or not earlier than processing 150M of them. It doesn't seem to be platform specific - tested on Nevada SPARC (T2) and I can hit the problem there as well. Strange thing is that is seems to affect the 8th (last) byte of the sent chunk there - note that I don't do any ntoh/hton conversion:
pid 23143: reading counters...
consumer: expected 408811881, got 408811905
pid 23152: reading counters...
consumer: expected 780799648, got 780799667
pid 23163: reading counters...
consumer: expected 383774467, got 383774486
pid 23172: reading counters...
consumer: expected 6016086, got 6016090
...
This looks it may be caused by the changes introduced by 6622010. I have a hypothesis on this which I need
to test.
Work Around
Add the line below to /etc/system:
set ip:do_tcp_direct_sockfs = 0x0
Comments
I modified the consumer.c such that it only prints out the
mismatch but keeps on running. The output is interesting.
pid 103397: reading counters...
expected 1040499744, got 1040499747
expected 1040499745, got 1040499748
expected 1040499746, got 1040499744
expected 1040499747, got 1040499745
expected 1040499748, got 1040499746
expected 3118228176, got 3118228179
expected 3118228177, got 3118228180
expected 3118228178, got 3118228181
expected 3118228179, got 3118228182
expected 3118228180, got 3118228183
expected 3118228181, got 3118228176
expected 3118228182, got 3118228177
expected 3118228183, got 3118228178
The above indicates that it is a re-ordering problem, not
a corruption issue.
The immidiate fix for this problem is to simply disable sync streams from
being used. The next step would be to remove the sync streams code. See
CR 6826274.