Search Robots?
Printed From: Web Wiz Forums
Category: Web Wiz Web App Support Forums
Forum Name: Web Wiz Forums
Forum Description: Support forum for Web Wiz Forums application.
URL: https://forums.webwiz.net/forum_posts.asp?TID=10867
Printed Date: 08 April 2026 at 5:36pm
Software Version: Web Wiz Forums 12.08 - https://www.webwizforums.com
Topic: Search Robots?
Posted By: Dr Moocowz
Subject: Search Robots?
Date Posted: 14 June 2004 at 5:35pm
|
Every once in a while on my forum, I'll see in the Active Users something like the following... Guest 1 June 14 2004 at 1:19pm June 14 2004 at 1:50pm 31 minutes Search Robot MSN I check the server logs and I can't really see anything malicious going on... I guess my question is what exactly is a Search Robot, what does it do, and should I be concerned about it doing something malicious or accessing something it shouldn't have access to? Thanks ahead of time. |
Replies:
Posted By: thekiwi
Date Posted: 15 June 2004 at 6:07am
Search Robots index the internet .... eg GoogleBot for Google etc. If you dont want your site to be indexed (spidered) then add the appropriate info to a robots.txt in the root of your site. ------------- Cheers TheKiwi http://www.infobahn.co.nz - Internet Infobahn - website design and hosting |
Dr Moocowz wrote:Posted By: dj air
Date Posted: 15 June 2004 at 6:07am
|
they are exactly what they say they are. they search the forums . and getting data from the forum so people from the search engines can search the forum, for an answer for the search word. hope that makes sense. there is nothing to worry about with them. run down in the above - u post a message. - a search bot comes to your forum and cashes/adds the details of the post to their database. - someone on the search engine site (google for example). search for something . -t can use the data from the search that the search bot did. - to show results from the search on google. |
Posted By: dpyers
Date Posted: 15 June 2004 at 7:58am
|
Some search robots harvest email addresses from pages that are able to be displayed. As a rule of thumb, never put your email address on a page. One of the reasons forums like this have Private Mail is so members can email each other without exposing their email address to the world. ------------- 
Lead me not into temptation... I know the short cut, follow me. |
Posted By: Scotty32
Date Posted: 15 June 2004 at 10:50am
|
you could block the search engines (robots) from accessing your site with the robot.txt in the root folder http://www.google.com/webmasters/3.html#B4 - http://www.google.com/webmasters/3.html#B4 theres sum info off google about the robot.txt file just make a robot.txt to block search engines (robots) from accessing your forum ------------- S2H.co.uk - http://www.s2h.co.uk/wwf/" rel="nofollow - WebWiz Mods and Skins For support on my mods + skins, please use http://www.s2h.co.uk/forum/" rel="nofollow - my forum . |
Posted By: dpyers
Date Posted: 15 June 2004 at 2:48pm
|
The robots.txt file only works for well behaved bots who take the trouble to read and follow it. Bad Bots ignore it and also any robots meta tags. -------------
Lead me not into temptation... I know the short cut, follow me. |
Posted By: web-geek
Date Posted: 26 June 2004 at 4:57am
|
i kno this isnt my post and its not really about the forum, but instead of kicking out the bots is there anyway to encourage them to come to ur site? like feeding mice cheese to get them to go to the mouse trap. im jst finding that with all my sites ive had they are neva or i hav neva found out if they hav been crawled or not and they do not appear in search engines. |
Posted By: dj air
Date Posted: 26 June 2004 at 2:14pm
does your pages have titles, and descriptions you can add the following
to the pages aswell that can help, once they been they know when to
come back.<meta name="expires" content="never"> |
Posted By: dpyers
Date Posted: 26 June 2004 at 3:52pm
|
google, yahoo, aol, and the other engines all have forms you can use to submit your site to them. You should also submit to dmoz - the open directory project - as many search engines use them as source info - but they do manual checks and it can take months/years before they get around to you. If you have a link to your homepage on a related site that google regularly visits, their bot will follow the link to you and index you as well. Google will usually to a surface scan of your site within a few days/weeks of submission. They'll to a deep crawl within a few months. -------------
Lead me not into temptation... I know the short cut, follow me. |