Robots
Printed From: Web Wiz Forums
Category: General Discussion
Forum Name: Web Design Discussion
Forum Description: Discussion on web design and development subjects.
URL: https://forums.webwiz.net/forum_posts.asp?TID=23624
Printed Date: 28 March 2026 at 10:35pm
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com
Topic: Robots
Posted By: Duby
Subject: Robots
Date Posted: 29 June 2007 at 11:00am
I really need a help with Yahoo robots and I hope that someone will help me to solve my problem. I have created robots.txt (not roboots.txt) file: User-agent: * Disallow: /Forum Disallow: /some other directories (it isn't matter) But, I have a problem. Somehow, Yahoo search robots and Google robots are still searching my site. What did I do wrong and how can I stop them?  ------------- Duby |
Replies:
Posted By: WebWiz-Bruce
Date Posted: 29 June 2007 at 11:19am
|
This file needs t be called:- robots.tx Not roboots.txt ------------- https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting |
Posted By: Duby
Date Posted: 29 June 2007 at 11:32am
This file needs to be called robots.txt, not robots.tx.  It is called robots.txt. It has stoped other robotos but I can't stop Yahoo Search robots. If you have time, take a look at http://humnasutli.inet.hr/forum/active_users.asp - LINK . Yesterday there were 144 Yahoo search robots reading my forum. Yes, I know I have an interesting forum, but....  ------------- Duby |
-boRg- wrote:Posted By: WebWiz-Bruce
Date Posted: 29 June 2007 at 1:18pm
|
The forum code itself detects search engines. What you could do is edit the common.asp file and have all search robots redirected to another file such as an insufficient permission file. This would put a stop to those pesky search engines:- If OSType = "Search Robot" Then Response.Redirect("anotherpage.htm") ------------- https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting |
Posted By: Duby
Date Posted: 29 June 2007 at 2:46pm
|
I'll try your advice -boRg-. I'll try to redirect them to the Private messenger file. Thank you very much! ------------- Duby |
Posted By: Duby
Date Posted: 29 June 2007 at 2:47pm
|
Or should it be outside the forum?
------------- Duby |
Posted By: Duby
Date Posted: 29 June 2007 at 3:13pm
|
They are all gone, 0 robots searching my forum. Thank you very much -boRg-!
------------- Duby |
Posted By: manxboy
Date Posted: 25 March 2008 at 12:34pm
|
I am also having this trouble, I have added the following line to my common.asp but it has made no difference. If OSType = "Search Robot Yahoo" Then Response.Redirect("notreal.htm") Can anyone help me? |
Posted By: Duby
Date Posted: 25 March 2008 at 1:31pm
Are you sure that you are having trouble with Yahoo Search Robots? ------------- Duby |
Posted By: manxboy
Date Posted: 25 March 2008 at 2:30pm
|
Yes I think so If you follow this link you will see - http://www.manxrc.com/forums/active_users.asp?SID=812b2f111ea74cb3f3fd171738da8aba - Link Is this the same area you were? |
Posted By: Duby
Date Posted: 27 March 2008 at 6:39am
Yes, your problem are Yahoo Robots. Did you create robots.txt file? Try to replace notreal.htm with some real webpage outside your forum. ------------- Duby |
Posted By: true_king
Date Posted: 10 June 2008 at 5:58pm
| In A Page That Is Password Protected Or Only Access For Special Suscribed User, How Google Can Access On It? |
Posted By: jamie.townsend
Date Posted: 11 June 2008 at 7:40am
They can't, unless you write code to allow them. But this them opens the window to people using firefox who can mask themselves as a robot and read all of the articles. Forums allow this because they assum the author would like the posts to be indexed. If you don't want this to happen then do as Bruce stated above. |