Web Wiz Forums - Prevent googlebot from opening reply forms etc

Print Page | Close Window

Prevent googlebot from opening reply forms etc

Printed From: Web Wiz Forums
Category: Web Wiz Web App Support Forums
Forum Name: Web Wiz Forums
Forum Description: Support forum for Web Wiz Forums application.
URL: https://forums.webwiz.net/forum_posts.asp?TID=23605
Printed Date: 07 April 2026 at 7:48am
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com

Topic: Prevent googlebot from opening reply forms etc

Posted By: Mattias Nordin
Subject: Prevent googlebot from opening reply forms etc
Date Posted: 26 June 2007 at 9:29pm

A suggestion, add "no follow" to all edit and reply buttons to prevent google from indexing pages like /new_reply_form.asp?M=Q&PID=136&PN=1&TR=1

I can see that google spent all day indexing pages that the googlebot should not have access to. If you have a lot of threads then google will have hundreds of unneccesary links to follow

You may solve this by adding the "nofollow" attribute to the links on all quote, reply, login and add new buttons.

Examlpe, this is how it looks today:


...<a href="new_reply_form.asp?TID=360&PN=1&TR=1"><img src="forum_images/post_reply.gif"  border="0" align="absmiddle" alt="Post Reply"/></a> 
<a href="new_topic_form.asp?FID=2"><img src="forum_images/new_post.gif"  border="0" align="absmiddle" alt="Post New Topic"/></a>...

To add a "nofollow" attribute on all links on a specific page use the meta tag:

The meta tag above works but the drawback is that all links on that page is affected and if the forum is included on someones webpage this option may not be to popular. Another solution that might work (depending on the spider) is to add rel="nofollow" and class="robots-nocontent" to the links.

Example:


<a href="new_topic_form.asp?FID=2" REL="nofollow" class="robots-nocontent"><img src="forum_images/new_post.gif"  border="0" align="absmiddle" alt="Post New Topic"/></a>...

I have to mention that i use one of the older versions of the 
forum, perhaps stuff has changed. Keep up the good work.

PS here you will find a list of all the lates google 
activity on my website. Note that googlebot has spent 
time indexing many of the reply.asp pages.  http://mattiasnordin.net/web07/visitorLog3/front/spider/show_spider_link_list.asp?rSpiderName=Google - Googlebot list here  DS

Best regards
Mattias

Replies:

Posted By: WebWiz-Bruce
Date Posted: 27 June 2007 at 11:44am

The nofollow tag is already used in Web Wiz Forums but it is certainly worth adding in the places you mention

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: Mattias Nordin
Date Posted: 27 June 2007 at 12:03pm

Thank you borg :)

Posted By: Mattias Nordin
Date Posted: 28 June 2007 at 4:02pm

While you're at it you might want to prevent the spiders from following all links in the calendar. Googlebot seems to be stupid enough to follow all kind of links to blank pages in the calendar...

If you're interested you may view the activity on http://mattiasnordin.net/web07/visitorLog3/front/spider/show_spider_link_list.asp?rSpiderName=Google - this page . Newest at the bottom.

Posted By: WebWiz-Bruce
Date Posted: 28 June 2007 at 5:14pm

The problem with that is that people will want their calender events indexed by Google.

Have been looking into the issue of search engines following links to pages it can't index like the profile, replay, etc. pages and although Google and MSN often obey directives like rel="nofollow" allot of search engines like Yahoo completely ignore this but does obay class="robots-nocontent"

Even with a directive not to follow a link to a page often Google and MSN will still visit the page anyway although they would not index it.

The rel="nofollow" has been added to the next version to page links to pages that search engines are not permitted like reply, new post, etc. which should mean less visits to those pages from search engines but won't stop it completely, and should also help reduce bandwidth as have a number of large forums with lots of links to then where search engines are sucking up allot of bandwidth indexing pages it's not permitted on.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: WebWiz-Bruce
Date Posted: 28 June 2007 at 5:20pm

Sorry my mistake the class="robots-nocontent" will not work on links.

It is only used by Yahoo at the moment to omit sections of an XHTML page from being indexed, you place it on a div or span section, on a link it has no effect as the link will be followed anyway.

So it would seem that although rel="nofollow" will have some effect on Google and MSN there isn't anything that will prevent Yahoo from following a link.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: Mattias Nordin
Date Posted: 09 July 2007 at 12:52pm

Related link:

Google, Yahoo, MSN Unite On Support For Nofollow Attribute For Links

http://blog.searchenginewatch.com/blog/050118-204728 - http://blog.searchenginewatch.com/blog/050118-204728

Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 1:23pm

Just a shame Yahoo didn't stick to this as this article is from 2005 and Yahoo presently seem to ignore rel="nofollow", but most of the other search engines seem to follow it.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: Mattias Nordin
Date Posted: 09 July 2007 at 1:30pm

oh, it was that old.

Anyway, it's a bit strange that the spiders are not smarter. Google is still browsing through all kind of combinations of the calendar. If they downloaded a copy of each page then they would notice that all of the downloaded pages are 98% identical. Just the date differ... If googlebot is about to index all days in the calendar, then it would take a lot of time.

MSNbot is having a hard time accepting that a page does no longer exist. Msnbot have visited a page, that is returning 404, about 10 times allready.

It's interesting to follow the spider activity. I haven't done this before and this is why i'm interested :)

Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 1:37pm

I just let them get on with it, although I have placed in delays on this site of 60 seconds to slow them down, even then they still suck around 10gig of bandwidth per month.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: ctscott
Date Posted: 09 July 2007 at 1:55pm

not to change the subject, but the left/right scrolling in the first post was terrible from a UI perspective. will this be eliminated in the next version?

-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia

Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 2:12pm

What left right scrolling?

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: ctscott
Date Posted: 09 July 2007 at 6:51pm

the first post has three scroll bars in it. the bottom one is the one have have to use to read what's posted in the first two.

http:// -

-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia

Posted By: WebWiz-Bruce
Date Posted: 09 July 2007 at 7:41pm

That is exactly the same as in version 8.

The data that was copied and pasted into that first post was set to <pre> which is often used for cde to prevent line breaks messing up the code.

In this case the CSS creates scroll bars just for that post rather than having the entire screen scroll. The same happens with images and other content which is to large for the post area. Without it the entire screen would scroll which is even worse.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: ctscott
Date Posted: 10 July 2007 at 1:14pm

thanks. knowing why i can live with that. i had noticed every now and then it was in some posts. removing screen print.

-------------
______________________
http://www.cfbtrivia.com" rel="nofollow - College Football Trivia

Posted By: WebWiz-Bruce
Date Posted: 10 July 2007 at 4:09pm

If you use browsers like Firefox and Opera their implementations of the CSS overflow property is much less temperamental than IE and looks allot better.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: WebWiz-Bruce
Date Posted: 11 July 2007 at 2:38pm

Mattis have now implemented the nofollow on this forum to see how it goes for pages that only members can access like writing replies, and looking at the active users page it's no-longer full of listing of search engines trying index pages that they have no access to, so should save on some unwanted bandwidth use.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: WebWiz-Bruce
Date Posted: 11 July 2007 at 9:23pm

OK spoke to soon likes like all the major search engines, Yahoo, Google, etc. just ignore the nofollow directive and still visit the page anyway.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: Mattias Nordin
Date Posted: 12 July 2007 at 8:11am

Borg, i think that your change will work pretty well. The problem is probably that pages that allready exist in the organic google list will be vistited very often just to make sure the page still exists. Your change would hopfully prevent that the page will be addedd to the organic list in the first place. I don't think that the nofollow attribure will give the same signal as response 404.

Off topic. Nice layout on the forum. Something feels new ;)

Posted By: ruycnd
Date Posted: 16 July 2007 at 9:15am

Posted By: Mattias Nordin
Date Posted: 16 July 2007 at 9:22am

If i remeber correctly the "disallow" items in a robots.txt file needs to have a full path. I dont think that wildcards are supported. This would mean that it's not possible for borg to create a robots.txt page that would work for everyone becuase it's up to the user to install the script at any given location.

I think that the action borg allready have taken are good enough :)