Web Wiz - Green Windows Web Hosting
Web Wiz Homepage Search Web Wiz

  New Posts New Posts RSS Feed - Search Robots
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Search Robots
 Post Reply Post Reply
Author
  Topic Search Topic Search  Topic Options Topic Options
Ivar View Drop Down
Groupie
Groupie


Joined: 24 November 2002
Location: Netherlands
Status: Offline
Points: 114 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Ivar Quote  Post ReplyReply Direct Link To This Post Topic: Search Robots
    Posted: 11 July 2007 at 6:24pm
I see there are a lot of Search Robots on my forum. Sometimes more than a 100, manly Yahoo.
What can I do about that ??
 
 
Back to Top
WebWiz-Bruce View Drop Down
Admin Group
Admin Group
Avatar
Web Wiz Developer

Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844 		Post Options Post Options   Thanks (0) Thanks(0)   Quote WebWiz-Bruce Quote  Post ReplyReply Direct Link To This Post Posted: 11 July 2007 at 6:39pm
Same on this forum, if you findout let me know.
Web Wiz Forums Hosting
ASP.NET Web Hosting
Back to Top
statRacing View Drop Down
Newbie
Newbie


Joined: 15 July 2007
Status: Offline
Points: 4 		Post Options Post Options   Thanks (0) Thanks(0)   Quote statRacing Quote  Post ReplyReply Direct Link To This Post Posted: 22 July 2007 at 2:56am
You can add this to your robots.txt to block Yahoo:

User-agent: Slurp
Disallow: /
Back to Top
jckruger View Drop Down
Groupie
Groupie
Avatar

Joined: 18 April 2006
Location: Australia
Status: Offline
Points: 135 		Post Options Post Options   Thanks (0) Thanks(0)   Quote jckruger Quote  Post ReplyReply Direct Link To This Post Posted: 20 August 2007 at 1:59am
As I don't want any robots on my site my robots.txt file looks like this:

User-Agent: *
Disallow: /
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.

Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.