It shouldn't but it does. Network bandwidth between many locations is often limited and/or bottlenecked. e.g. Portland Oregon is a bottleneck for bandwith between North America and Japan. This also affects Korea, etc. However, there is a LOT of bandwidth there, and it isn't a problem except under unusual circumstances [ read massive worm attacks ]. For connections between North America and Europe, you shouldn't worry though.
Australia is a BAD place to host - bandwith is extremely expensive there. Similarly most of Africa is a complete digital wasteland.
The physical location of the server determines the number of physical hops that that the request/response must make. Each hop adds to the delay, and the available network bandwidth at each hop is the major limiting factor there. (This is a bit of an oversimplification.)
That is generally a minor consideration - the major factors are the load on the server the site is on, the clients speed (duh), and the distance the host is located from a backbone.
Another major problem is the local ISPs and what they do. If they cache pages, content can be outdated. (They do this to save bandwidth sometimes.)
See:
http://www.cybergeography.org/atlas/alcatel_large.gif
http://www.servepath.com/why/network.htm
http://www.cybergeography.org/atlas/isp_maps.html
http://www.accd.edu/sac/lrc/john/wwwtest2.htm
Of course you can also spend several thousand dollars to get better information:
http://www.telegeography.com/pubs/internet/reports/ig_gbl/
You can consider North America, (most of) Europe, (most of) North East Asia to be safe hosting locations. i.e. There is a reason why JodoHost has their servers in the US and not in India. Hosting in Australia, Vietnam, or Chad would simply be dumb. There is a lot of bandwidth and very good hosting in SE Asia (i.e. Singapore, Hong Kong, etc.) but for sites where bandwidth is a worry, stick closer to home (internet wise that is).
There are gaming IDCs in Hong Kong and China that cater specifically to the online gaming industry in Asia because they can offer superior bandwidth and costs compared to other high-bandwidth providers.
For the absolute best available, look into good, profitable porn sites and see what they use. A large amount of all internet technology is either developed or first implemented in the online porn industry. Those perverts certainly know what they are doing
[ Or should I say "who" they are doing...
] Of course that kind of hosting is extremely expensive though. Easily $500 per month.
Do a traceroute to webwiz to find the path to the server, and then check where the hops are. That will give you your answer.