I need a FAST crawler to crawl a site and gather meteorological information and insert it into a database on regular intervals (about every 10 min or so).
There are, I believe about 2000 pages to crawl and about 10 pieces of data to collect from each page. (I have the list ready), In addition, one field in the databse will be needed for time, and one time zone will need to be collected from another NTS server.
This database will be made available to some client software, that will be developed after this is done.
If good work is done on this step, additional work may be possible.
This project must be completed within one week.
I would prefer a PHP and MySql solution.