Hello,

I know it's difficult but...

Do any of you know if there is some Crawler, spider or bot that follow links and index the meta tags of the pages it follows? Starting in pages filled manually or in a database?

I would love to create a search engine strictly for the subject of my site (brazilian environment) but not adding site by site in a DB like the WWG app because it will take much more time than I have... with a crawler I think I could do it...

so, anyone know (a free one)?

Thanks!

First create an asp page which gets the urls for crawing from a tempory database. use the httpxml component to get the url and regexp object to  nick the metatags then add the desciptions and the urls to the search database.

Then create a 'schedueld task' on your computer and set it to navigate to that page every 10 minutes or so.

Not the best option but it should work..

I thought of that but this would only allow me to get the metatags and etc... the thing I need most is to follow the links, so I can start with a score of very good brazilian environment sites and folow the smaller ones from that.

If I don't find one I may be doing this, but not sure if the work will be worthy...

Thanks anyway!