Create a C# Application to monitor and scrape data from pre-defined public websites to a central location

We wish to create an application, written in C# which has the ability to monitor one or more public websites for new content being added and for existing content being updated, then when this occurs to scrape a copy of such material and to place a copy into a predefined internal location. We will maintain a list of Identified sources (for example, [login to view URL]) which need to be monitored, either at regular intervals (hourly, daily, etc) or constantly for new items or items which have changed and which meet criteria stipulated, such as relating to policy, regulation, consultative paper, etc. Where a new or changed item is identified, the content of that item should be scraped and written to an internal database. In monitoring a site, the app should have the capability to determine that the item has been previously retrieved. Where a previously retrieved item is changed, a copy of the changed version is to be harvested, preferably with a link to the previous news item. The following data fields will be required: • Source website URL • Source (using unique ID provided) • Date retrieved (DDMMYYYY) • Date Published (DDMMYYYY) • Title • Sub-title (if applicable) • Version number (if applicable) • All text, broken down as follows: • Chapter titles, section titles which are not numbered to contain just the text • Sections which are numbered to show section numbering as provided along with text • Any graphic or table to be harvested as shown • Footnotes and cross-references to be maintained Technical Requirements - We need to be able to run the collector as a windows service ultimately, however we are happy to take a windows forms exe as a PoC, we can then compile this into a windows service as necessary. As described above, there will be a list of sources that we need to collect data for (these will be in different formats. So each source will need to be custom written. There will be many sections for one regulation. Please see the example above. In terms of storage format, we would like JSON output with a file per regulation, named accordingly to sit in a configurable place on our disk on our server. We would like the file name to include the version number and use this to prevent updating the regulation be3ing duplicated or updated when not necessary. Initial Project for PoC We would like to build a collector for the following regulation handbooks. There will be some common code, then we suggest some plugin code per source to handle the different formats of the documents. The initial sources and documents we would like to collect for the PoC are as follows: UK Financial Conduct Authority – FCA Handbook UK Prudential Regulatory Authority – PRA Rulebook US Office of the Comptroller of the Currency (OCC) European Central Bank / EU Journal Bank for International Settlements, Basel Committee on Banking Supervision These sources can all be readily found on the internet using simply google searches to retrieve the correct URLs.