Web Wiz Forums Modifications

Print Page | Close Window

Search Robots

Printed From: Web Wiz Forums
Category: Web Wiz Web App Support Forums
Forum Name: Web Wiz Forums Modifications
Forum Description: Mod's and Add-on's for Web Wiz Forums.
URL: https://forums.webwiz.net/forum_posts.asp?TID=23724
Printed Date: 30 March 2026 at 4:19am
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com

Topic: Search Robots

Posted By: Ivar
Subject: Search Robots
Date Posted: 11 July 2007 at 6:24pm

I see there are a lot of Search Robots on my forum. Sometimes more than a 100, manly Yahoo.

What can I do about that ??

Replies:

Posted By: WebWiz-Bruce
Date Posted: 11 July 2007 at 6:39pm

Same on this forum, if you findout let me know.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: statRacing
Date Posted: 22 July 2007 at 2:56am

You can add this to your robots.txt to block Yahoo:

User-agent: Slurp
Disallow: /

Posted By: jckruger
Date Posted: 20 August 2007 at 1:59am

As I don't want any robots on my site my robots.txt file looks like this:

User-Agent: *
Disallow: /