Robots.txt

Post Reply

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
johnk Members Profile Find Members Posts Newbie Joined: 04 May 2006 Status: Offline Points: 10	Post Options Post Reply Quote johnk Report Post Thanks(0) Quote Reply Topic: Robots.txt Posted: 13 May 2006 at 1:52am
	WWF v8 Got everything up and running last night. This morning I discovered that my site had been crawled and virtually every file associated with the forum had been indexed. Any guidelines re excluding files/directories so that forum posts will be indexed but the "non content" files will be excluded? Thanks johnk

WebWiz-Bruce Members Profile Find Members Posts Admin Group Web Wiz Developer Joined: 03 September 2001 Location: Bournemouth Status: Offline Points: 9844	Post Options Post Reply Quote WebWiz-Bruce Report Post Thanks(0) Quote Reply Posted: 13 May 2006 at 10:40am
	In the file includes/setup_options.asp set the following variable to True:- Const blnNoFollowTagInLinks = False
	Web Wiz Forums Hosting ASP.NET Web Hosting

johnk Members Profile Find Members Posts Newbie Joined: 04 May 2006 Status: Offline Points: 10	Post Options Post Reply Quote johnk Report Post Thanks(0) Quote Reply Posted: 15 May 2006 at 9:49am
	Will this still allow posts to the forum to be crawled? Thanks johnk

WebWiz-Bruce Members Profile Find Members Posts Admin Group Web Wiz Developer Joined: 03 September 2001 Location: Bournemouth Status: Offline Points: 9844	Post Options Post Reply Quote WebWiz-Bruce Report Post Thanks(0) Quote Reply Posted: 15 May 2006 at 10:53am
	Yes, but any new posts that include links will not be followed by a rebot, therefore they shouldn't be indexed.
	Web Wiz Forums Hosting ASP.NET Web Hosting

Post Reply

Forum Jump

Forum Permissions View Drop Down

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot delete your posts in this forum
You cannot edit your posts in this forum
You cannot create polls in this forum
You cannot vote in polls in this forum

About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.