Web Wiz Support and Community Forums : Unwanted pages in search engines https://forums.webwiz.net/ Copyright (c) 2006-2013 Web Wiz Forums - All Rights Reserved. Tue, 07 Apr 2026 13:16:13 +0000 Fri, 08 Jun 2007 10:29:21 +0000 http://blogs.law.harvard.edu/tech/rss Web Wiz Forums 12.08 360 https://forums.webwiz.net/RSS_post_feed.asp?TID=23466 <![CDATA[Web Wiz Support and Community Forums]]> https://forums.webwiz.net/forum_images/web_wiz_forums.png https://forums.webwiz.net/ <![CDATA[Unwanted pages in search engines : This looks very much like...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123468.html#123468 Author: WebWiz-Bruce
Subject: 23466
Posted: 08 June 2007 at 10:29am

This looks very much like the robots.txt file used for this site.

Parts of it shouldn't be used, as crawl delays are built into this one to make search robots index the site more slowly to stop them sucking bandwidth as this site is continually crawled by many search engines as there are over 100,000 sites linking to this site. Most sites wouldn't want this as it could prevent your site being index properly.

Your robots.txt file which you place into the root of your site should look like:-


User-agent: *
Disallow: /forum_images/
Disallow: /includes/
Disallow: printer_friendly_posts.asp 
Disallow: calendar_week.asp



Edited by -boRg- - 08 June 2007 at 10:29am]]> Fri, 08 Jun 2007 10:29:21 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123468.html#123468 <![CDATA[Unwanted pages in search engines : Why is the cgi-bin file disallowed?I...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123465.html#123465 Author: cathal
Subject: 23466
Posted: 08 June 2007 at 10:14am

Why is the cgi-bin file disallowed? I cant even see it in my forum. Is it hidden?]]> Fri, 08 Jun 2007 10:14:34 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123465.html#123465 <![CDATA[Unwanted pages in search engines : michael wrote:Like printer_friendly_posts.asp...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123463.html#123463 Author: _PDG_
Subject: 23466
Posted: 08 June 2007 at 8:45am

Originally posted by michael michael wrote:

Like printer_friendly_posts.asp and calendar_week.asp etc.
 
 
Or for the newbies a simple way to just Cut/Paste info and save as a .Txt file named Robots ( robots.txt ) Wink
 
Quote
 
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /includes/
Disallow: /404error.asp
Disallow: printer_friendly_posts.asp 
Disallow: calendar_week.asp
 

User-agent: googlebot
Crawl-delay: 20

User-agent: yahoo
Crawl-delay: 60

User-agent: Slurp
Crawl-delay: 60

User-agent: msnbot
Crawl-delay: 60

User-agent: Teoma
Crawl-delay: 20

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

User-agent: fast
Disallow: /


Edited by _PDG_ - 08 June 2007 at 8:46am]]> Fri, 08 Jun 2007 08:45:38 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123463.html#123463 <![CDATA[Unwanted pages in search engines : Like printer_friendly_posts.asp...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123379.html#123379 Author: michael
Subject: 23466
Posted: 05 June 2007 at 7:03pm

Like printer_friendly_posts.asp and calendar_week.asp etc.]]> Tue, 05 Jun 2007 19:03:51 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123379.html#123379 <![CDATA[Unwanted pages in search engines : What files / folders will i need...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123372.html#123372 Author: cathal
Subject: 23466
Posted: 05 June 2007 at 4:13pm

What files / folders will i need to exclude from the search to stop these pages appearing]]> Tue, 05 Jun 2007 16:13:01 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123372.html#123372 <![CDATA[Unwanted pages in search engines : Create a robots.txt file that...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123369.html#123369 Author: michael
Subject: 23466
Posted: 05 June 2007 at 2:12pm

Create a robots.txt file that excludes those pages. Most "good" search engines should honor that file.
Go to any domain like microsoft.com webwiz.net or whatever and call robots.txt to see a sample.
]]> Tue, 05 Jun 2007 14:12:35 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123369.html#123369 <![CDATA[Unwanted pages in search engines : How do i prevent pages like this: --Calender...]]> https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123368.html#123368 Author: cathal
Subject: 23466
Posted: 05 June 2007 at 1:46pm

How do i prevent pages like this:
 
--Calender Pages
 
--Printer Friendly Pages
 
from appearing in Google, Windows Live Search and other search engines
]]> Tue, 05 Jun 2007 13:46:16 +0000 https://forums.webwiz.net/unwanted-pages-in-search-engines_topic23466_post123368.html#123368