malformed.org does the blog thing

Sometimes machines will be performing poorly (especially with interactive sessions), and you want to figure out what’s wrong. The general bottlenecks are:

Disk
Network
CPU
Interrupts

FreeBSD has some pretty decent utilities that will let you get some basic information quickly, so you at least know where to start looking for problems.

For instance, for disk stuff you can use iostat:

[bash:~] iostat -w 1
tty da0 fd0 pass0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 1 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2 0 7 2 89
0 38 4.64 244 1.11 0.00 0 0.00 0.00 0 0.00 2 0 5 2 91
0 38 5.25 273 1.40 0.00 0 0.00 0.00 0 0.00 2 0 3 1 94
0 38 5.29 263 1.36 0.00 0 0.00 0.00 0 0.00 2 0 4 0 93
0 38 4.91 258 1.24 0.00 0 0.00 0.00 0 0.00 4 0 8 1 87
0 38 3.57 292 1.02 0.00 0 0.00 0.00 0 0.00 2 0 8 0 90
0 38 4.07 239 0.95 0.00 0 0.00 0.00 0 0.00 2 0 3 1 94
0 38 4.79 234 1.09 0.00 0 0.00 0.00 0 0.00 2 0 6 2 90
0 38 5.14 215 1.08 0.00 0 0.00 0.00 0 0.00 2 0 5 1 93
0 38 4.83 231 1.09 0.00 0 0.00 0.00 0 0.00 3 0 4 3 90
0 38 6.69 240 1.56 0.00 0 0.00 0.00 0 0.00 2 0 7 1 90
0 38 5.94 242 1.40 0.00 0 0.00 0.00 0 0.00 2 0 5 2 91
0 38 6.17 259 1.56 0.00 0 0.00 0.00 0 0.00 2 0 5 1 92
0 38 7.58 259 1.92 0.00 0 0.00 0.00 0 0.00 1 0 5 2 92
0 38 6.19 259 1.57 0.00 0 0.00 0.00 0 0.00 1 0 5 1 93
0 38 6.57 256 1.65 0.00 0 0.00 0.00 0 0.00 2 0 3 1 94
0 38 5.53 267 1.44 0.00 0 0.00 0.00 0 0.00 2 0 5 1 92
^C
[bash:~]

You can see, at a glance, that the server is doing 1-2MB/s (not much), that there’s plenty of CPU, etc.

For bandwidth you can use netstat in a similar way:


[bash:~] netstat -w 1
input (Total) output
packets errs bytes packets errs bytes colls
1236 0 211280 1377 0 1528798 0
1244 0 201469 1377 0 1591548 0
1178 0 191679 1291 0 1515261 0
1143 0 196220 1268 0 1499036 0
^C
[bash:~]

by multiplying the bytes by 8, you can get a pretty good idea of real time bandwidth (13-14Mbit in the above case), see the packets/second, and any errors or collisions.

If neither of those show what the problem is (99% of the time they do), you can use “systat -v” to see a real time representation of interrupts, cpu usage, and some VM metrics.

Both of the above are formatted much better in a console :)

I haven’t found decent tools to do the above tasks under linux. iostat kind of works, but its got a lot more information, and scrolls off the screen much more rapidly than the BSD one, so it’s harder to get a good idea about how much disk activity there really is. The only halfway decent thing I’ve found for checking network usage is iptraf, but it has a lot more overhead (and a lot more difficult to start up on a heavily loaded machine). It also often doesn’t count NFS traffic (when the kernel daemon is used maybe? haven’t really tracked it down), so it may be completely inaccurate anyway.

Do tools like this exist for linux? I just want to see what specifically is making the response time so poor when a machine is busy, and I haven’t found a decent way to do that.