Olympus trouble

In January 2013, the HP server which is olympus.fudo.org started having kernel crashes, observed with large data transfers.

Conditions which cause crashes

 * 1) FS transfer from asgard to olympus using dd over ssh via eth1 (private link)
 * 2) 2012-01-24 00:14: FS transfer from olympus to asgard. caused olympus kernel crash then reboot
 * 3) olympus> pv -pterab /dev/olympus/athena-root | ssh asgard dd of=/dev/asgard/athena-test bs=8096
 * 4) olympus syslog
 * 5) asgard syslog
 * 6) FS transfer from asgard to olympus. caused olympus kernel crash then reboot
 * 7) root@asgard:~ > pv -pterb /dev/asgard/thor-snapshot | ssh olympus.local dd of=/dev/olympus/thor-root
 * 8) root@asgard:~ > yes | ssh olympus.local dd of=/dev/olympus/thor-root bs=8096
 * 9) root@olympus:~ > pv -pterab /dev/olympus/athena-root | dd of=/dev/null bs=8096
 * 10) SSH'd in while move was occurring
 * 11) Unable to repeat -- even simultaneous with large scp, no crash
 * 12) Alongside dd of same file (Fedora image) from asgard to olympus, no issue
 * 13) root@asgard:~ > pv -pterb Fedora-18-x86_64-DVD.iso | ssh olympus.local dd of=/root/Fedora-18-x86_64-DVD.iso
 * 14) Tried this: pv -pterb /dev/asgard/thor-snapshot | ssh olympus dd of=/dev/olympus/thor-root bs=4096
 * 15) First time it failed with broken pipe, but olympus stayed up with a responsive shell
 * 16) Any attempt to restart failed immediately
 * 17) Tried direct ssh login from asgard to olympus, which triggered this crash (from syslog)

Conditions which do not cause crashes

 * 1) FS transfer from olympus to asgard using dd over ssh via eth1 - rate limited to 10 Mb/s
 * 2) 2012-01-24 00:39 ISO tranfer olympus to asgard on ftp via eth0 - success
 * 3) root@asgard:~ > wget ftp://reaper:#######@olympus/ubuntu-12.10-server-amd64.iso
 * 4) wget: 2013-01-24 06:39:55 (37.7 MB/s) - `ubuntu-12.10-server-amd64.iso.1' saved [722468864]
 * 5) ISO transfer from olympus to asgard using FTP via eth0 IPv6 - extreme switch
 * 6) root@asgard:~ > wget ftp://reaper:#######@olympus/ubuntu-12.10-server-amd64.iso
 * 7) 2013-01-24 06:39:55 (37.7 MB/s) - `ubuntu-12.10-server-amd64.iso.1' saved [722468864]
 * 8) ISO transfer from olympus to asgard using FTP via eth1 IPv4 - direct link
 * 9) root@asgard:~ > wget ftp://reaper:#######@olympus.local:/Fedora-18-x86_64-DVD.iso
 * 10) 2013-01-24 07:02:55 (31.0 MB/s) - `Fedora-18-x86_64-DVD.iso.1' saved [4573888512]
 * 11) ISO transfer from olypmus to asgard using scp via eth1 IPv4 - direct link
 * 12) root@asgard:~ > scp reaper@olympus.local:~/Fedora-18-x86_64-DVD.iso.
 * 13) Fedora-18-x86_64-DVD.iso                                         100% 4362MB  29.3MB/s   02:29
 * 14) Thu Jan 24 07:12:40 UTC 2013
 * 15) dd from olympus to olympus
 * 16) root@olympus:~ > pv -pterab /dev/olympus/athena-root | dd of=/dev/olympus/thor-root bs=8096
 * 17) Aborted early: 0:14:41 [11.5MB/s] [9.89MB/s] [==================================>                                                  ] 42% ETA 0:19:49
 * 18) 1115913+34866 records in
 * 19) 1115913+34866 records out
 * 20) 9141556896 bytes (9.1 GB) copied, 885.004 s, 10.3 MB/s
 * 21) root@asgard:~ > yes | ssh olympus.local dd of=/root/yesfile.txt
 * 22) root@olympus:~ > pv -tpreb /dev/olympus/athena-root | dd of=/dev/null bs=8096
 * 23) 21474836480 bytes (21 GB) copied, 269.652 s, 79.6 MB/s
 * 24) Repeated successfully