Web Wiz - Green Windows Web Hosting
Web Wiz Homepage Search Web Wiz

  New Posts New Posts RSS Feed - Crawler, spider or bots
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Crawler, spider or bots
 Post Reply Post Reply
Author
  Topic Search Topic Search  Topic Options Topic Options
faubo View Drop Down
Senior Member
Senior Member
Avatar

Joined: 30 May 2002
Location: Brazil
Status: Offline
Points: 560 		Post Options Post Options   Thanks (0) Thanks(0)   Quote faubo Quote  Post ReplyReply Direct Link To This Post Topic: Crawler, spider or bots
    Posted: 25 May 2003 at 11:52pm

Hello,

I know it's difficult but...

Do any of you know if there is some Crawler, spider or bot that follow links and index the meta tags of the pages it follows? Starting in pages filled manually or in a database?

I would love to create a search engine strictly for the subject of my site (brazilian environment) but not adding site by site in a DB like the WWG app because it will take much more time than I have... with a crawler I think I could do it...

so, anyone know (a free one)?

Thanks!
I don't know how to make you click here
Back to Top
Mart View Drop Down
Senior Member
Senior Member
Avatar

Joined: 30 November 2002
Status: Offline
Points: 2304 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Mart Quote  Post ReplyReply Direct Link To This Post Posted: 26 May 2003 at 12:06pm

First create an asp page which gets the urls for crawing from a tempory database. use the httpxml component to get the url and regexp object to  nick the metatags then add the desciptions and the urls to the search database.

Then create a 'schedueld task' on your computer and set it to navigate to that page every 10 minutes or so.

Not the best option but it should work..
Back to Top
faubo View Drop Down
Senior Member
Senior Member
Avatar

Joined: 30 May 2002
Location: Brazil
Status: Offline
Points: 560 		Post Options Post Options   Thanks (0) Thanks(0)   Quote faubo Quote  Post ReplyReply Direct Link To This Post Posted: 26 May 2003 at 10:54pm

I thought of that but this would only allow me to get the metatags and etc... the thing I need most is to follow the links, so I can start with a score of very good brazilian environment sites and folow the smaller ones from that.

If I don't find one I may be doing this, but not sure if the work will be worthy...

Thanks anyway!
I don't know how to make you click here
Back to Top
 Post Reply Post Reply

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.

Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.