Web Wiz - Green Windows Web Hosting
Web Wiz Homepage Search Web Wiz

  New Posts New Posts RSS Feed - Search Robots
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Search Robots
 Post Reply Post Reply Page  <123
Author
 Rating: Topic Rating: 1 Votes, Average 5.00  Topic Search Topic Search  Topic Options Topic Options
billd3 View Drop Down
Senior Member
Senior Member


Joined: 19 February 2003
Location: United States
Status: Offline
Points: 530 		Post Options Post Options   Thanks (0) Thanks(0)   Quote billd3 Quote  Post ReplyReply Direct Link To This Post Posted: 06 April 2008 at 7:07pm
Yesterday, we had over 200 guests, ALL YAHOO.
 
What the heck is up with the yahoos at yahoo lately? Why do they need to hit a single site with that much power?  I mean 200 of all YAHOO. Google sure doesn't hit us that hard, no one else does. I thing YAHOO has some serious issues, and always have, actually. Someone needs to contact them and thell them what they are doing.
It's one thing to see that every few days, or even every week, but we see such nonsense almost daily now.
BillD
http://theamcpages.com
http://theamcforum.com
Back to Top
WebWiz-Bruce View Drop Down
Admin Group
Admin Group
Avatar
Web Wiz Developer

Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844 		Post Options Post Options   Thanks (0) Thanks(0)   Quote WebWiz-Bruce Quote  Post ReplyReply Direct Link To This Post Posted: 07 April 2008 at 8:18am
Some of them do go over the top, the best thing to do is create a robots.txt file in the root of your web site and put in crawl delays to slow down some of these bots. They will then still index the site but without going mad and sucking all your bandwidth.

Below is the one used for Web Wiz Guide:-

robots.txt
User-agent: *
Disallow: /images/
Disallow: /includes/

User-agent: googlebot
Crawl-delay: 60

User-agent: yahoo
Crawl-delay: 60

User-agent: Slurp
Crawl-delay: 60

User-agent: msnbot
Crawl-delay: 60

User-agent: Teoma
Crawl-delay: 60

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

User-agent: fast
Disallow: /
Web Wiz Forums Hosting
ASP.NET Web Hosting
Back to Top
freakyfred View Drop Down
Groupie
Groupie


Joined: 29 March 2007
Location: United Kingdom
Status: Offline
Points: 171 		Post Options Post Options   Thanks (0) Thanks(0)   Quote freakyfred Quote  Post ReplyReply Direct Link To This Post Posted: 07 April 2008 at 1:02pm
Yahoo had me with 260 the other day. Made the site look popular.
Back to Top
123Simples View Drop Down
Senior Member
Senior Member
Avatar

Joined: 08 July 2007
Location: United Kingdom
Status: Offline
Points: 1192 		Post Options Post Options   Thanks (0) Thanks(0)   Quote 123Simples Quote  Post ReplyReply Direct Link To This Post Posted: 07 April 2008 at 3:15pm
Thanks for that Bruce
I have just uploaded my txt file now
Visit 123 Simples Web Design
Back to Top
 Post Reply Post Reply Page  <123

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.

Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.