I should introduce you to Swizzle or Scudzā¦
Sorry, being sarcastic. 
It depends on a few factors. When I play on the CGS Server Iām around 8-20. However, the server is hosted in Dallas, I live in Baton Rouge, my downstream connection is around 20-30Mbps and my machine is SuperBAD! There are other things at play but those are usually the main ones. When I connect to other servers the latency varies. Could be anywhere from 30-100 but not usually more.
Your physical distance really dictates how fast you will be able to connect. If the server has a bad connection you will connect at itās crappy speed, nothing you can do about that. Try an official Valve server, they are usually pretty strong. Lastly, downloading websites is not a good test of your connection speed. Go to a local bandwidth speed test site and try it out. Iād give you one but I doubt any of the ones I use are close. You mentioned being wireless before. Iād really try wired. Just because it says you are connected at 54Mpbs or whatever doesnāt mean you arenāt getting huge amounts of dropped packets or some other variable. Try a wire temporarily if need be and see if that helps.
Take it easy. 
Edit: Hereās one, I know nothing about it but they say they are #1. 
http://www.bandwidthtest.co.uk/