Web Wiz - Green Windows Web Hosting
Web Wiz Homepage Search Web Wiz

  New Posts New Posts RSS Feed - Robots
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Topic ClosedRobots
 Post Reply Post Reply Page  12>
Author
  Topic Search Topic Search  Topic Options Topic Options
Duby View Drop Down
Groupie
Groupie
Avatar

Joined: 23 August 2005
Location: Croatia
Status: Offline
Points: 80 		Direct Link To This Post Topic: Robots
    Posted: 29 June 2007 at 11:00am
I really need a help with Yahoo robots and I hope that someone will help me to solve my problem.Embarrassed

I have created robots.txt (not roboots.txt) file:

User-agent: *
Disallow: /Forum
Disallow: /some other directories (it isn't matter)

But, I have a problem. Somehow, Yahoo search robots and Google robots are still searching my site. What did I do wrong and how can I stop them?Cry
 






Edited by Duby - 29 June 2007 at 11:40am
Duby
Back to Top
WebWiz-Bruce View Drop Down
Admin Group
Admin Group
Avatar
Web Wiz Developer

Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844 		Direct Link To This Post Posted: 29 June 2007 at 11:19am
This file needs t be called:-

robots.tx

Not

roboots.txt
Web Wiz Forums Hosting
ASP.NET Web Hosting
Back to Top
Duby View Drop Down
Groupie
Groupie
Avatar

Joined: 23 August 2005
Location: Croatia
Status: Offline
Points: 80 		Direct Link To This Post Posted: 29 June 2007 at 11:32am
Originally posted by -boRg- -boRg- wrote:

This file needs t be called:-

robots.tx

Not

roboots.txt


This file needs to be called robots.txt, not robots.tx.Smile

It is called robots.txt. It has stoped other robotos but I can't stop Yahoo Search robots.

If you have time, take a look at LINK.

Yesterday there were 144 Yahoo search robots reading my forum. Yes, I know I have an interesting forum, but....SmileWink
Duby
Back to Top
WebWiz-Bruce View Drop Down
Admin Group
Admin Group
Avatar
Web Wiz Developer

Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844 		Direct Link To This Post Posted: 29 June 2007 at 1:18pm
The forum code itself detects search engines.

What you could do is edit the common.asp file and have all search robots redirected to another file such as an insufficient permission file. This would put a stop to those pesky search engines:-

If OSType = "Search Robot" Then Response.Redirect("anotherpage.htm")
Web Wiz Forums Hosting
ASP.NET Web Hosting
Back to Top
Duby View Drop Down
Groupie
Groupie
Avatar

Joined: 23 August 2005
Location: Croatia
Status: Offline
Points: 80 		Direct Link To This Post Posted: 29 June 2007 at 2:46pm
I'll try your advice -boRg-. I'll try to redirect them to the Private messenger file.

Thank you very much!
Duby
Back to Top
Duby View Drop Down
Groupie
Groupie
Avatar

Joined: 23 August 2005
Location: Croatia
Status: Offline
Points: 80 		Direct Link To This Post Posted: 29 June 2007 at 2:47pm
Or should it be outside the forum?Embarrassed
Duby
Back to Top
Duby View Drop Down
Groupie
Groupie
Avatar

Joined: 23 August 2005
Location: Croatia
Status: Offline
Points: 80 		Direct Link To This Post Posted: 29 June 2007 at 3:13pm
They are all gone, 0 robots searching my forum. Thank you very much -boRg-!
Duby
Back to Top
manxboy View Drop Down
Newbie
Newbie


Joined: 19 July 2004
Location: Isle Of Man
Status: Offline
Points: 14 		Direct Link To This Post Posted: 25 March 2008 at 12:34pm
I am also having this trouble, I have added the following line to my common.asp but it has made no difference.

If OSType = "Search Robot Yahoo" Then Response.Redirect("notreal.htm")

Can anyone help me?
Back to Top
 Post Reply Post Reply Page  12>

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.

Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.