Web Design Discussion - getting bots to crawl the whole site

Print Page | Close Window

getting bots to crawl the whole site

Printed From: Web Wiz Forums
Category: General Discussion
Forum Name: Web Design Discussion
Forum Description: Discussion on web design and development subjects.
URL: https://forums.webwiz.net/forum_posts.asp?TID=14460
Printed Date: 29 March 2026 at 10:25am
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com

Topic: getting bots to crawl the whole site

Posted By: Steve B
Subject: getting bots to crawl the whole site
Date Posted: 28 March 2005 at 3:42pm

I have a website which has the front end built around MAMBO and a seperate forums using phpBB. I have submitted both the url of the main site and a seperate url of the forums to the search engines but I have noticed that the bots are only crawling the forum index and not the forum or indiviual posts. I have seen many topics/posts listed in search engines that have been indexed out of other forums but can't seem to get them to crawl the whole of mine.

Has anyone any ideas as to how I can sort out this problem please?

-------------
www.thephotographyforums.com - a friendly photography site.

Replies:

Posted By: WebWiz-Bruce
Date Posted: 29 March 2005 at 4:09am

Maybe this is a problem you need to post on MAMBO or phpBB sites as they products competing against the software from this site!!

Now, if you had used the forum from written by and produced by this site, Web Wiz Forums, then it would have all pages indexed by search engines.

-------------
https://www.webwiz.net/web-wiz-forums/forum-hosting.htm" rel="nofollow - Web Wiz Forums Hosting
https://www.webwiz.net/web-hosting/windows-web-hosting.htm" rel="nofollow - ASP.NET Web Hosting

Posted By: huwnet
Date Posted: 31 March 2005 at 5:24pm

You need to see the info on the phpbb faq about rewriting session ids for bots.

Or you could just wait for 3.x which will have it built in

Posted By: Steve B
Date Posted: 02 April 2005 at 7:20pm

Thanks huwnet for the helpful reply. After some more research I have managed to find a mod/hack that was written to address this issue. This was implemented earlier in the week and seems to be working now.

boRg- I am sorry that my site is based on a competitor's software but would like to say that the whole of www.thephotographyforums.com has been based on software that was provided by the hosting company that we use, it wasn't chosen for any other reason. I would not begin to know the benefits or disadvantages of either system, neither do I want to start sl*ging off either. My site has had to much work done to it at this point for me to consider changing.

I found this site whilst looking for web promotion advice and find the free tools here very useful, from there I stumbled into your forums and posted for advice as I liked the general tone of the place, I didn't mean to cause any trouble.

Thanks again.

-------------
www.thephotographyforums.com - a friendly photography site.