Web Wiz - Green Windows Web Hosting
Web Wiz Homepage Search Web Wiz

  New Posts New Posts RSS Feed - Unwanted pages in search engines
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Unwanted pages in search engines
 Post Reply Post Reply
Author
  Topic Search Topic Search  Topic Options Topic Options
cathal View Drop Down
Groupie
Groupie


Joined: 24 April 2007
Location: United Kingdom
Status: Offline
Points: 60 		Post Options Post Options   Thanks (0) Thanks(0)   Quote cathal Quote  Post ReplyReply Direct Link To This Post Topic: Unwanted pages in search engines
    Posted: 05 June 2007 at 1:46pm
How do i prevent pages like this:
 
 
 
from appearing in Google, Windows Live Search and other search engines
Back to Top
michael View Drop Down
Senior Member
Senior Member
Avatar

Joined: 08 April 2002
Location: United States
Status: Offline
Points: 4670 		Post Options Post Options   Thanks (0) Thanks(0)   Quote michael Quote  Post ReplyReply Direct Link To This Post Posted: 05 June 2007 at 2:12pm
Create a robots.txt file that excludes those pages. Most "good" search engines should honor that file.
Go to any domain like microsoft.com webwiz.net or whatever and call robots.txt to see a sample.
Blog | MPG Tracker
Back to Top
cathal View Drop Down
Groupie
Groupie


Joined: 24 April 2007
Location: United Kingdom
Status: Offline
Points: 60 		Post Options Post Options   Thanks (0) Thanks(0)   Quote cathal Quote  Post ReplyReply Direct Link To This Post Posted: 05 June 2007 at 4:13pm
What files / folders will i need to exclude from the search to stop these pages appearing
Back to Top
michael View Drop Down
Senior Member
Senior Member
Avatar

Joined: 08 April 2002
Location: United States
Status: Offline
Points: 4670 		Post Options Post Options   Thanks (0) Thanks(0)   Quote michael Quote  Post ReplyReply Direct Link To This Post Posted: 05 June 2007 at 7:03pm
Like printer_friendly_posts.asp and calendar_week.asp etc.
Blog | MPG Tracker
Back to Top
_PDG_ View Drop Down
Newbie
Newbie
Avatar

Joined: 12 April 2007
Location: Sweden
Status: Offline
Points: 7 		Post Options Post Options   Thanks (0) Thanks(0)   Quote _PDG_ Quote  Post ReplyReply Direct Link To This Post Posted: 08 June 2007 at 8:45am
Originally posted by michael michael wrote:

Like printer_friendly_posts.asp and calendar_week.asp etc.
 
 
Or for the newbies a simple way to just Cut/Paste info and save as a .Txt file named Robots ( robots.txt ) Wink
 
Quote
 
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /includes/
Disallow: /404error.asp
Disallow: printer_friendly_posts.asp 
Disallow: calendar_week.asp
 

User-agent: googlebot
Crawl-delay: 20

User-agent: yahoo
Crawl-delay: 60

User-agent: Slurp
Crawl-delay: 60

User-agent: msnbot
Crawl-delay: 60

User-agent: Teoma
Crawl-delay: 20

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

User-agent: fast
Disallow: /


Edited by _PDG_ - 08 June 2007 at 8:46am
Back to Top
cathal View Drop Down
Groupie
Groupie


Joined: 24 April 2007
Location: United Kingdom
Status: Offline
Points: 60 		Post Options Post Options   Thanks (0) Thanks(0)   Quote cathal Quote  Post ReplyReply Direct Link To This Post Posted: 08 June 2007 at 10:14am
Why is the cgi-bin file disallowed? I cant even see it in my forum. Is it hidden?
Back to Top
WebWiz-Bruce View Drop Down
Admin Group
Admin Group
Avatar
Web Wiz Developer

Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844 		Post Options Post Options   Thanks (0) Thanks(0)   Quote WebWiz-Bruce Quote  Post ReplyReply Direct Link To This Post Posted: 08 June 2007 at 10:29am
This looks very much like the robots.txt file used for this site.

Parts of it shouldn't be used, as crawl delays are built into this one to make search robots index the site more slowly to stop them sucking bandwidth as this site is continually crawled by many search engines as there are over 100,000 sites linking to this site. Most sites wouldn't want this as it could prevent your site being index properly.

Your robots.txt file which you place into the root of your site should look like:-


User-agent: *
Disallow: /forum_images/
Disallow: /includes/
Disallow: printer_friendly_posts.asp 
Disallow: calendar_week.asp



Edited by -boRg- - 08 June 2007 at 10:29am
Web Wiz Forums Hosting
ASP.NET Web Hosting
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.

Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.