Actually, writing a web crawler isn't a big thing. I've done them in perl, c++ and java. The perl one was a dog though as they really need to be multi-threaded and that wasn't an option at the time. IIRC, I wrote the java one as part of a java 101 type exercise in threading.
You could probably do a .net one if you can handle the web server timeout issue. My java crawler ran through an app server, not a web server and the pearl and c++ ones hit outbound tcp ports directly, didn't go through a web or an app server.
The issue isn't so much one of crawling pages, but what do you do with the data on those pages. What words and phrases do you extract? How do you id content instead of something like java script. Do you ignore meta tags? Comments? Finally, how do you rationalize the data into a db that allows you the ability to scan it fast with some degree of relevance to the search term.
Topic Options
Post Options
Thanks(0)
I said that writting one that parses information is difficult 
Gullanian wrote: