how to extract data from websites

In asp, you can use serverXMLHTTP to pull back a screen in a string that you can then search/screenscrape. You can also use it to post form data although some sites will check http_referrer to ensure that form data is only posted from their domain.

Many of the comparison sites use an api to pick up a feed from the sites they compare .

There's an example of serverxmlhttp here http://new2asp.com/Section_Files/PullStrings/PullStrings.asp

You can use either get or put. a put will ususally require adding a query string to the url.

You can probably contact the sites you want to get the info from to find out what api's they use. There's industry specifix xml feeds and often site specific ones.

Always get permission from the sites to either screen scrape or use their api. You'll be exposed to legal liability if you don't. They can also complain to your webhost and your isp about theft of content, badwidth theft, etc. You could wake up one morning and find yourself on a badgut list with no website and no internet connection.


Edited by dpyers - 28 January 2006 at 12:21am