Print Page | Close Window

Prevent googlebot from opening reply forms etc

Printed From: Web Wiz Forums
Category: Web Wiz Web App Support Forums
Forum Name: Web Wiz Forums
Forum Description: Support forum for Web Wiz Forums application.
URL: https://forums.webwiz.net/forum_posts.asp?TID=23605
Printed Date: 07 April 2026 at 7:48am
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com


Topic: Prevent googlebot from opening reply forms etc
Posted By: Mattias Nordin
Subject: Prevent googlebot from opening reply forms etc
Date Posted: 26 June 2007 at 9:29pm
A suggestion, add "no follow" to all edit and reply buttons to prevent google from indexing pages like /new_reply_form.asp?M=Q&PID=136&PN=1&TR=1
 
I can see that google spent all day indexing pages that the googlebot should not have access to. If you have a lot of threads then google will have hundreds of unneccesary links to follow
 
You may solve this by adding the "nofollow" attribute to the links on all quote, reply, login and add new buttons. 
 
Examlpe, this is how it looks today:

...<a href="new_reply_form.asp?TID=360&PN=1&TR=1"><img src="forum_images/post_reply.gif"  border="0" align="absmiddle" alt="Post Reply"/></a>
<a href="new_topic_form.asp?FID=2"><img src="forum_images/new_post.gif"  border="0" align="absmiddle" alt="Post New Topic"/></a>...
To add a "nofollow" attribute on all links on a specific page use the meta tag:
<META name="ROBOTS" content="NOINDEX, NOFOLLOW">
The meta tag above works but the drawback is that all links on that page is affected and if the forum is included on someones webpage this option may not be to popular. Another solution that might work (depending on the spider) is to add rel="nofollow" and class="robots-nocontent" to the links.
 
Example:
 

<a href="new_topic_form.asp?FID=2" REL="nofollow" class="robots-nocontent"><img src="forum_images/new_post.gif"  border="0" align="absmiddle" alt="Post New Topic"/></a>...
I have to mention that i use one of the older versions of the 
forum, perhaps stuff has changed. Keep up the good work.
PS here you will find a list of all the lates google 
activity on my website. Note that googlebot has spent
time indexing many of the reply.asp pages. http://mattiasnordin.net/web07/visitorLog3/front/spider/show_spider_link_list.asp?rSpiderName=Google - Googlebot list here DS
Best regards
Mattias 



Replies:
Posted By: WebWiz-Bruce
Date Posted: 27 June 2007 at 11:44am
The nofollow tag is already used in Web Wiz Forums but it is certainly worth adding in the places you mention

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: Mattias Nordin
Date Posted: 27 June 2007 at 12:03pm
Thank you borg :)


Posted By: Mattias Nordin
Date Posted: 28 June 2007 at 4:02pm
While you're at it you might want to prevent the spiders from following all links in the calendar. Googlebot seems to be stupid enough to follow all kind of links to blank pages in the calendar...
 
If you're interested you may view the activity on http://mattiasnordin.net/web07/visitorLog3/front/spider/show_spider_link_list.asp?rSpiderName=Google - this page . Newest at the bottom.
 
 


Posted By: WebWiz-Bruce
Date Posted: 28 June 2007 at 5:14pm
The problem with that is that people will want their calender events indexed by Google.

Have been looking into the issue of search engines following links to pages it can't index like the profile, replay, etc. pages and although Google and MSN often obey directives like rel="nofollow" allot of search engines like Yahoo completely ignore this but does obay class="robots-nocontent"

Even with a directive not to follow a link to a page often Google and MSN will still visit the page anyway although they would not index it.

The rel="nofollow" has been added to the next version to page links to pages that search engines are not permitted like reply, new post, etc. which should mean less visits to those pages from search engines but won't stop it completely, and should also help reduce bandwidth as have a number of large forums with lots of links to then where search engines are sucking up allot of bandwidth indexing pages it's not permitted on.


-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: WebWiz-Bruce
Date Posted: 28 June 2007 at 5:20pm
Sorry my mistake the class="robots-nocontent" will not work on links.

It is only used by Yahoo at the moment to omit sections of an XHTML page from being indexed, you place it on a div or span section, on a link it has no effect as the link will be followed anyway.

So it would seem that although rel="nofollow" will have some effect on Google and MSN there isn't anything that will prevent Yahoo from following a link.


-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: Mattias Nordin
Date Posted: 09 July 2007 at 12:52pm
Related link:
 
Google, Yahoo, MSN Unite On Support For Nofollow Attribute For Links
http://blog.searchenginewatch.com/blog/050118-204728 - http://blog.searchenginewatch.com/blog/050118-204728
 
:)


Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 1:23pm
Just a shame Yahoo didn't stick to this as this article is from 2005 and Yahoo presently seem to ignore rel="nofollow", but most of the other search engines seem to follow it.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: Mattias Nordin
Date Posted: 09 July 2007 at 1:30pm
oh, it was that old.
 
Anyway, it's a bit strange that the spiders are not smarter. Google is still browsing through all kind of combinations of the calendar. If they downloaded a copy of each page then they would notice that all of the downloaded pages are 98% identical. Just the date differ... If googlebot is about to index all days in the calendar, then it would take a lot of time. LOL
 
MSNbot is having a hard time accepting that a page does no longer exist. Msnbot have visited a page, that is returning 404, about 10 times allready.
 
It's interesting to follow the spider activity. I haven't done this before and this is why i'm interested :)


Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 1:37pm
I just let them get on with it, although I have placed in delays on this site of 60 seconds to slow them down, even then they still suck around 10gig of bandwidth per month.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: ctscott
Date Posted: 09 July 2007 at 1:55pm
not to change the subject, but the left/right scrolling in the first post was terrible from a UI perspective.  will this be eliminated in the next version?Disapprove

-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia


Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 2:12pm
What left right scrolling?

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: ctscott
Date Posted: 09 July 2007 at 6:51pm
the first post has three scroll bars in it. the bottom one is the one have have to use to read what's posted in the first two.
 
http:// -  


-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia


Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 7:41pm
That is exactly the same as in version 8.

The data that was copied and pasted into that first post was set to <pre> which is often used for cde to prevent line breaks messing up the code.

In this case the CSS creates scroll bars just for that post rather than having the entire screen scroll. The same happens with images and other content which is to large for the post area. Without it the entire screen would scroll which is even worse.


-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: ctscott
Date Posted: 10 July 2007 at 1:14pm
thanks.  knowing why i can live with that.  i had noticed every now and then it was in some posts.  removing screen print.

-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia


Posted By: WebWiz-Bruce
Date Posted: 10 July 2007 at 4:09pm
If you use browsers like Firefox and Opera their implementations of the CSS overflow property is much less temperamental than IE and looks allot better.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: WebWiz-Bruce
Date Posted: 11 July 2007 at 2:38pm
Mattis have now implemented the nofollow on this forum to see how it goes for pages that only members can access like writing replies, and looking at the active users page it's no-longer full of listing of search engines trying index pages that they have no access to, so should save on some unwanted bandwidth use.


-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: WebWiz-Bruce
Date Posted: 11 July 2007 at 9:23pm
OK spoke to soon likes like all the major search engines, Yahoo, Google, etc. just ignore the nofollow directive and still visit the page anyway.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting


Posted By: Mattias Nordin
Date Posted: 12 July 2007 at 8:11am

Borg, i think that your change will work pretty well. The problem is probably that pages that allready exist in the organic google list will be vistited very often just to make sure the page still exists. Your change would hopfully prevent that the page will be addedd to the organic list in the first place. I don't think that the nofollow attribure will give the same signal as response 404.

Off topic. Nice layout on the forum. Something feels new ;)


Posted By: ruycnd
Date Posted: 16 July 2007 at 9:15am
Wink


Posted By: Mattias Nordin
Date Posted: 16 July 2007 at 9:22am

If i remeber correctly the "disallow" items in a robots.txt file needs to have a full path. I dont think that wildcards are supported. This would mean that it's not possible for borg to create a robots.txt page that would work for everyone becuase it's up to the user to install the script at any given location.

I think that the action borg allready have taken are good enough :)



Print Page | Close Window

Forum Software by Web Wiz Forums® version 12.08 - https://www.webwizforums.com
Copyright ©2001-2026 Web Wiz Ltd. - https://www.webwiz.net