Our web-based storage system was able to completely saturate the gigabit link to a compute cluster. The serving nodes were two Sun x2100’s, each with an older Apple XRAID attached. All access was through the grizzly based webserver. Separate testing was able to show this system could handle between 1200 and 2200 small file requests/s per node.
Month: March 2010
Being Evil
Someone was using SiteSucker on a demo site and was behaving badly, this site also hosts our redmine install on the same DB and started to slow down my work. To block we had a few options, iptables the IP, block the IP w/in apache (htaccess) or do the evil thing, just block based on their user-agent. I chose the latter.
RewriteEngine on
...
RewriteCond %{HTTP_USER_AGENT} ^SiteSucker2.3.1
RewriteRule ^(.*) /badbot.html
This is evil since the persons crawler will get out badbot.html page. The person will then use firefox/ie/whatever to browse to the page and look to see why it’s not working, but since it’s sending a differend user-agent, they will be allowed to browse. For anyone who knows how to configure a crawler it isn’t an issue changing the supplied agent, but then again that person will likely be able to control their crawler and not kill my web server. Here’s a snapshot of the logs showing the crawler pulling badbot (size 174) followed by a browse attempt from safari.
82.232.60.250 - - [09/Mar/2010:19:52:41 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14379 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)" 82.232.60.250 - - [09/Mar/2010:19:52:42 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14377 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)" 82.232.60.250 - - [10/Mar/2010:11:03:13 -0500] "GET /cgi-bin/isadg/viewrg.pl?rgid=44 HTTP/1.1" 200 2286 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; fr-fr) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
Test details
Here’s the details of the performance testing listed below, hardware etc and the raw IOZone numbers.
Client Details:
- Dual-Core AMD Opteron(tm) Processor 1220 (Sun 2100)
- 2.8Ghz, 2GB Memory, 1Gbps network, RHEL4
Server Details:
- Intel Xeon 5120, 1.86Ghz (dell 1950)
- 1GB Memory, 1Gbps network, RHEL5
- Perc5i raid controller setup in 2 7-disk raid 5 volumes
- Disk: Dell MD1000 w/ 750Gb SATA 7200rpm drives
Untuned results (4G test, 4k records) in KB/s
random random bkwd
KB reclen write rewrite read reread read write read
4194304 4096 93137 68662 28869 29300 23118 57440 23186
Tuned results:
random random bkwd
KB reclen write rewrite read reread read write read
4194304 4096 95360 71281 108200 112603 67761 69528 68227
Notice the huge gains in read performance 28MB/s to 108MB/s for sequential and 23MB/s to 67MB/s for non-sequential.
Here’s the theoretical max based on running iozone on the server. It looks like nfs is adding overhead on rewrites and possibly random reads and writes with sequential reads/writes bottlenecking at the 1Gbps connection
random random bkwd
KB reclen write rewrite read reread read write read
2097152 4096 302379 234327 174385 196905 102032 144843 99866
NFS Performance
We’re testing a moderately fast nfs server. Testing on our data set of large files, ~100M we were seeing performance a little over 100MB/s. Now, when we shared the same filesystem out via nfs performance dropped to a mere 44MB/s and no matter what the r/wsize of the client, would not increase. Both machines are connected via a 1Gbps link that has been shown to move 941mb/s. Here’s a snapshot what the server workload looked like on the server. The first line shows the local testing, notice the 108MB/s and the second shows the new access. The interesting part is the device utilization during the tests, even though we were pushing half the requests/second and moving half the bandwidth, we were at 90% with a higher wait time 4ms vs 1.7ms.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 30.50 0.00 906.00 0.00 108530.00 0.00 239.58 1.61 1.78 1.08 98.15 sdb 870.00 0.00 500.50 0.00 44626.00 0.00 178.33 2.06 4.09 1.83 91.55
Now, let’s tweak the ioscheduling a little by setting slice_idle to 1 as recommended in the dell guide below. ‘echo 1 > /sys/block/sdb/queue/iosched/slice_idle’ This affects how long the io scheduler will wait in each queue before looking for more work. On local access where you will have lots of low latency requests you probably want to wait for more work, however over a network with lots of clients you will want to switch as soon as possible
sdb 1164.18 0.00 1049.75 0.00 71150.25 0.00 135.56 2.34 2.23 0.87 91.79
let’s go a little further and set it to 0
sdb 144.00 0.00 2428.50 0.00 92034.00 0.00 75.79 3.95 1.63 0.33 79.60
Now we’re up to 92MB/s which is above 90% of what our network can do. Good enough!