| Author |
|
manxboy 
Newbie
Joined: 19 July 2004
Location: Isle Of Man
Status: Offline
Points: 14
|
 Post Options
 Thanks(0)
 Quote  Reply
 Posted: 26 March 2008 at 10:58pm |
|
Thanks for the info. I may add a robots.txt file in but it is good to read your document on this regarding the security concerns.
Is it intended in a later version of the forums to add an option to stop them being crawled?
|
 |
WebWiz-Bruce
Admin Group
Web Wiz Developer
Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844
|
Post Options
Thanks(0)
Quote Reply
Posted: 27 March 2008 at 9:37am |
|
The problem with search engine robots is they don't always declare themselves as such.
The only real way to prevent your forum from being index is to disallow Guests from being able to access forums. This way search engines will also not index your forum.
|
|
|
|
manxboy
Newbie
Joined: 19 July 2004
Location: Isle Of Man
Status: Offline
Points: 14
|
Post Options
Thanks(0)
Quote Reply
Posted: 27 March 2008 at 9:44am |
|
Thanks Bruce, unfortunately I want to allow guests to read the forum so I may have to put up with the robots.
|
|
manxboy
Newbie
Joined: 19 July 2004
Location: Isle Of Man
Status: Offline
Points: 14
|
Post Options
Thanks(0)
Quote Reply
Posted: 01 April 2008 at 9:04am |
|
There are now 53 robots browsing my forum.
I have but a file, robots.txt in the root of my site now with the code listed previously in this thread contained.
How long does it take for this to come into effect or is it instant?
|
|
WebWiz-Bruce
Admin Group
Web Wiz Developer
Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844
|
Post Options
Thanks(0)
Quote Reply
Posted: 01 April 2008 at 9:59am |
|
It can take awhile as the search bots don't continually read the robot.txt file. But you should see it starting to take effect over the next few days.
|
|
|
|
ctscott
Senior Member
Joined: 27 May 2003
Location: United States
Status: Offline
Points: 246
|
Post Options
Thanks(0)
Quote Reply
Posted: 01 April 2008 at 11:40am |
 WebWiz-Bruce wrote:
Not at the present time, but it is a good idea for future releases. |
if this enhancement is implementated please make it optional. i prefer to know they are crawling.
|
|
|
|
WebWiz-Bruce
Admin Group
Web Wiz Developer
Joined: 03 September 2001
Location: Bournemouth
Status: Offline
Points: 9844
|
Post Options
Thanks(0)
Quote Reply
Posted: 01 April 2008 at 12:19pm |
|
There is a new SEO optimising section in the admin area of version 10, it could be added as an option under that.
|
|
|
|
WebCity
Groupie
Joined: 11 December 2003
Location: United States
Status: Offline
Points: 154
|
Post Options
Thanks(0)
Quote Reply
Posted: 06 April 2008 at 6:58pm |
Scotty32 wrote:
You only really want to use the Robots.txt if you do not want traffic from search engines - which may mean you get no new visitors.
There are different ways you can stop search engines:
1) stop guests from viewing all or most forums, this will then include the Search Robots.
2) use a robots.txt file in the root of your site (eg so its www.site.com/robots.txt)
you then add the following to the robots file:
User-Agents: *
Disallow: /forum/ |
As i said, with both of these methods you will stop search engines from crawling your website, which will then mean you do not display in search results, which will then mean you get no new members. If how ever your site relays on word of mouth, or other methods of advertising it may not affect you.
For more information, i wrote a page about the Robots.txt File on my website, but please note that when i wrote it i mistakenly added a "allow:" command which apparently doesnt exist
For google how ever, you can use the Webmaster Center to tell google how often you want it to crawl your website. It will only affect google, but it has the advantage that you will not disappear from googles search results.
|
I tried the suggestions above but they didn't work the google bots kept coming in and eating more of my bandwidth. So I tried the blocking IP address thing. I blocked one then 2 then 3 and seen a pattern. I noticed the IP address is the same but the last 2 numbers was different so I blocked the all the numbers that where the same in the IP address. That has seemed to stop the google bots. This may cause problems later but for now it seems to stop the bandwidth eating monsters. The big rule on our board is if you like to eat bandwidth you will be Denied Access
Edited by WebCity - 06 April 2008 at 7:01pm
|
|
Topic Options