Web Wiz Support and Community Forums : Search Robots? https://forums.webwiz.net/ Copyright (c) 2006-2013 Web Wiz Forums - All Rights Reserved. Wed, 08 Apr 2026 16:12:42 +0000 Sat, 26 Jun 2004 15:52:45 +0000 http://blogs.law.harvard.edu/tech/rss Web Wiz Forums 12.08 360 https://forums.webwiz.net/RSS_post_feed.asp?TID=10867 <![CDATA[Web Wiz Support and Community Forums]]> https://forums.webwiz.net/forum_images/web_wiz_forums.png https://forums.webwiz.net/ <![CDATA[Search Robots? : google, yahoo, aol, and the other...]]> https://forums.webwiz.net/search-robots_topic10867_post60472.html#60472 Author: dpyers
Subject: 10867
Posted: 26 June 2004 at 3:52pm

google, yahoo, aol, and the other engines all have forms you can use to submit your site to them.

You should also submit to dmoz - the open directory project - as many search engines use them as source info - but they do manual checks and it can take months/years before they get around to you.

If you have a link to your homepage on a related site that google regularly visits, their bot will follow the link to you and index you as well.

Google will usually to a surface scan of your site within a few days/weeks of submission. They'll to a deep crawl within a few months.

]]> Sat, 26 Jun 2004 15:52:45 +0000 https://forums.webwiz.net/search-robots_topic10867_post60472.html#60472 <![CDATA[Search Robots? : does your pages have titles, and...]]> https://forums.webwiz.net/search-robots_topic10867_post60462.html#60462 Author: dj air
Subject: 10867
Posted: 26 June 2004 at 2:14pm

does your pages have titles, and descriptions you can add the followingto the pages aswell that can help, once they been they know when tocome back.
<meta name="expires" content="never">
<meta name="robots" content="INDEX,FOLLOW">
<meta name="revisit-after" content="7 Days">

]]> Sat, 26 Jun 2004 14:14:53 +0000 https://forums.webwiz.net/search-robots_topic10867_post60462.html#60462 <![CDATA[Search Robots? : i kno this isnt my post and its...]]> https://forums.webwiz.net/search-robots_topic10867_post60442.html#60442 Author: web-geek
Subject: 10867
Posted: 26 June 2004 at 4:57am

i kno this isnt my post and its not really about the forum, but instead of kicking out the bots is there anyway to encourage them to come to ur site? like feeding mice cheese to get them to go to the mouse trap.

im jst finding that with all my sites ive had they are neva or i hav neva found out if they hav been crawled or not and they do not appear in search engines.

]]> Sat, 26 Jun 2004 04:57:13 +0000 https://forums.webwiz.net/search-robots_topic10867_post60442.html#60442 <![CDATA[Search Robots? : The robots.txt file only works...]]> https://forums.webwiz.net/search-robots_topic10867_post59692.html#59692 Author: dpyers
Subject: 10867
Posted: 15 June 2004 at 2:48pm

The robots.txt file only works for well behaved bots who take the trouble to read and follow it. Bad Bots ignore it and also any robots meta tags.]]> Tue, 15 Jun 2004 14:48:40 +0000 https://forums.webwiz.net/search-robots_topic10867_post59692.html#59692 <![CDATA[Search Robots? : you could block the search engines...]]> https://forums.webwiz.net/search-robots_topic10867_post59673.html#59673 Author: Scotty32
Subject: 10867
Posted: 15 June 2004 at 10:50am

you could block the search engines (robots) from accessing your site with the robot.txt in the root folder

http://www.google.com/webmasters/3.html#B4 theres sum info off google about the robot.txt file

just make a robot.txt to block search engines (robots) from accessing your forum
]]> Tue, 15 Jun 2004 10:50:25 +0000 https://forums.webwiz.net/search-robots_topic10867_post59673.html#59673 <![CDATA[Search Robots? : Some search robots harvest email...]]> https://forums.webwiz.net/search-robots_topic10867_post59664.html#59664 Author: dpyers
Subject: 10867
Posted: 15 June 2004 at 7:58am

Some search robots harvest email addresses from pages that are able to be displayed.

As a rule of thumb, never put your email address on a page. One of the reasons forums like this have Private Mail is so members can email each other without exposing their email address to the world.

]]> Tue, 15 Jun 2004 07:58:02 +0000 https://forums.webwiz.net/search-robots_topic10867_post59664.html#59664 <![CDATA[Search Robots? : they are exactly what they say...]]> https://forums.webwiz.net/search-robots_topic10867_post59654.html#59654 Author: dj air
Subject: 10867
Posted: 15 June 2004 at 6:07am

they are exactly what they say they are.

they search the forums . and getting data from the forum so people from the search engines can search the forum, for an answer for the search word.

hope that makes sense.

there is nothing to worry about with them.

run down in the above

- u post a message.

- a search bot comes to your forum and cashes/adds the details of the post to their database.

- someone on the search engine site (google for example). search for something .

-t can use the data from the search that the search bot did.

- to show results from the search on google.

]]> Tue, 15 Jun 2004 06:07:41 +0000 https://forums.webwiz.net/search-robots_topic10867_post59654.html#59654 <![CDATA[Search Robots? : Dr Moocowz wrote:Every once...]]> https://forums.webwiz.net/search-robots_topic10867_post59653.html#59653 Author: thekiwi
Subject: 10867
Posted: 15 June 2004 at 6:07am

Originally posted by Dr Moocowz Dr Moocowz wrote:

Every once in a while on my forum, I'll see in the Active Users something like the following...

Guest 1   June 14 2004 at 1:19pm   June 14 2004 at 1:50pm    31 minutes   Search Robot    MSN

I check the server logs and I can't really see anything malicious going on...

I guess my question is what exactly is a Search Robot, what does itdo, and should I be concerned about it doing something malicious oraccessing something it shouldn't have access to?

Thanks ahead of time.


Search Robots index the internet .... eg GoogleBot for Google etc.
If you dont want your site to be indexed (spidered) then add the appropriate info to a robots.txt in the root of your site.
]]> Tue, 15 Jun 2004 06:07:03 +0000 https://forums.webwiz.net/search-robots_topic10867_post59653.html#59653 <![CDATA[Search Robots? : Every once in a while on my forum,...]]> https://forums.webwiz.net/search-robots_topic10867_post59609.html#59609 Author: Dr Moocowz
Subject: 10867
Posted: 14 June 2004 at 5:35pm

Every once in a while on my forum, I'll see in the Active Users something like the following...

Guest 1   June 14 2004 at 1:19pm   June 14 2004 at 1:50pm    31 minutes   Search Robot    MSN

I check the server logs and I can't really see anything malicious going on...

I guess my question is what exactly is a Search Robot, what does it do, and should I be concerned about it doing something malicious or accessing something it shouldn't have access to?

Thanks ahead of time.

]]> Mon, 14 Jun 2004 17:35:59 +0000 https://forums.webwiz.net/search-robots_topic10867_post59609.html#59609