The two fast respawn 2fort servers run on different hardware, have you noticed it on both of them?
In fact, #06 is running by itself on a vserver. I've just checked it's performance and it doesn't even come close to stressing out the hardware, even during peak times.
#04 shares resources with a few other TF2 servers on dedicated hardware, and that server too is not stressed.
Does anyone else notice problems with lag on these servers?