You only really want to use the Robots.txt if you do not want traffic from search engines - which may mean you get no new visitors.
There are different ways you can stop search engines:
1) stop guests from viewing all or most forums, this will then include the Search Robots.
2) use a robots.txt file in the root of your site (eg so its www.site.com/robots.txt)
you then add the following to the robots file:
User-Agents: *
Disallow: /forum/ |
As i said, with both of these methods you will stop search engines from crawling your website, which will then mean you do not display in search results, which will then mean you get no new members. If how ever your site relays on word of mouth, or other methods of advertising it may not affect you.
For more information, i wrote a page about the
Robots.txt File on my website, but please note that when i wrote it i mistakenly added a "allow:" command which apparently doesnt exist
For google how ever, you can use the
Webmaster Center to tell google how often you want it to crawl your website. It will only affect google, but it has the advantage that you will not disappear from googles search results.
Topic Options
Post Options
Thanks(0)
