| Author |
 Topic Search  Topic Options
|
Dr Moocowz 
Newbie
Joined: 26 January 2004
Location: United States
Status: Offline
Points: 16
|
 Post Options
 Thanks(0)
 Quote  Reply
 Topic: Search Robots?
Posted: 14 June 2004 at 5:35pm |
Every once in a while on my forum, I'll see in the Active Users something like the following...
Guest 1 June 14 2004 at 1:19pm June 14 2004 at 1:50pm 31 minutes Search Robot MSN
I check the server logs and I can't really see anything malicious going on...
I guess my question is what exactly is a Search Robot, what does it do, and should I be concerned about it doing something malicious or accessing something it shouldn't have access to?
Thanks ahead of time.
|
 |
thekiwi
Mod Builder Group
Joined: 23 November 2003
Location: New Zealand
Status: Offline
Points: 392
|
Post Options
Thanks(0)
Quote Reply
Posted: 15 June 2004 at 6:07am |
 Dr Moocowz wrote:
Every once in a while on my forum, I'll see in the Active Users something like the following...
Guest 1 June 14 2004 at 1:19pm June 14 2004 at 1:50pm 31 minutes Search Robot MSN
I check the server logs and I can't really see anything malicious going on...
I guess my question is what exactly is a Search Robot, what does it
do, and should I be concerned about it doing something malicious or
accessing something it shouldn't have access to?
Thanks ahead of time. |
Search Robots index the internet .... eg GoogleBot for Google etc.
If you dont want your site to be indexed (spidered) then add the appropriate info to a robots.txt in the root of your site.
|
|
|
|
dj air
Senior Member
Joined: 05 April 2002
Location: United Kingdom
Status: Offline
Points: 3627
|
Post Options
Thanks(0)
Quote Reply
Posted: 15 June 2004 at 6:07am |
they are exactly what they say they are.
they search the forums . and getting data from the forum so people from the search engines can search the forum, for an answer for the search word.
hope that makes sense.
there is nothing to worry about with them.
run down in the above
- u post a message.
- a search bot comes to your forum and cashes/adds the details of the post to their database.
- someone on the search engine site (google for example). search for something .
-t can use the data from the search that the search bot did.
- to show results from the search on google.
|
|
dpyers
Senior Member
Joined: 12 May 2003
Status: Offline
Points: 3937
|
Post Options
Thanks(0)
Quote Reply
Posted: 15 June 2004 at 7:58am |
Some search robots harvest email addresses from pages that are able to be displayed.
As a rule of thumb, never put your email address on a page. One of the reasons forums like this have Private Mail is so members can email each other without exposing their email address to the world.
|
Lead me not into temptation... I know the short cut, follow me.
|
|
Scotty32
Moderator Group
Joined: 30 November 2002
Location: Manchester, UK
Status: Offline
Points: 1682
|
Post Options
Thanks(0)
Quote Reply
Posted: 15 June 2004 at 10:50am |
you could block the search engines (robots) from accessing your site with the robot.txt in the root folder
http://www.google.com/webmasters/3.html#B4 theres sum info off google about the robot.txt file
just make a robot.txt to block search engines (robots) from accessing your forum
|
|
|
|
dpyers
Senior Member
Joined: 12 May 2003
Status: Offline
Points: 3937
|
Post Options
Thanks(0)
Quote Reply
Posted: 15 June 2004 at 2:48pm |
|
The robots.txt file only works for well behaved bots who take the trouble to read and follow it. Bad Bots ignore it and also any robots meta tags.
|
Lead me not into temptation... I know the short cut, follow me.
|
|
web-geek
Newbie
Joined: 23 June 2004
Status: Offline
Points: 5
|
Post Options
Thanks(0)
Quote Reply
Posted: 26 June 2004 at 4:57am |
i kno this isnt my post and its not really about the forum, but instead of kicking out the bots is there anyway to encourage them to come to ur site? like feeding mice cheese to get them to go to the mouse trap.
im jst finding that with all my sites ive had they are neva or i hav neva found out if they hav been crawled or not and they do not appear in search engines.
|
|
dj air
Senior Member
Joined: 05 April 2002
Location: United Kingdom
Status: Offline
Points: 3627
|
Post Options
Thanks(0)
Quote Reply
Posted: 26 June 2004 at 2:14pm |
does your pages have titles, and descriptions you can add the following
to the pages aswell that can help, once they been they know when to
come back.
<meta name="expires" content="never">
<meta name="robots" content="INDEX,FOLLOW">
<meta name="revisit-after" content="7 Days">
|
|
