Daily Trending Products Detection Program

The goal of this project is to develop a program that can identify trending products on a daily basis from specific e-commerce websites (the list will be provided later). The program should retrieve relevant product data, analyze it to identify trending products, and then output the results in a useful and structured format. **Key Responsibilities**: 1. **Data Acquisition**: You are responsible for developing a system that retrieves product data from the provided list of e-commerce websites. This may involve web scraping or API integration, depending on what each platform allows and provides. Python, with libraries such as BeautifulSoup and requests for web scraping, or requests alone for API integration, is our preferred language for this task. 2. **Data Analysis**: The program should analyze the retrieved data to identify trending products. A product's trendiness can be determined by factors such as a high number of recent sales, a spike in social media mentions, or a rapid climb in the sales rank. Tools like pandas, NumPy, or SciPy in Python can be used for this task. 3. **Automation**: This entire process needs to run automatically on a daily basis. Depending on the deployment environment, this can be accomplished by setting up a cron job, using a task scheduler, or other methods. 4. **Output**: The program should output the results in a clear and structured format. This should be a CSV file, a JSON file, and a database. 5. **Error Handling and Logging**: It's crucial to handle potential errors gracefully so the program can recover or, at the very least, log the error for troubleshooting. Logging mechanisms need to be implemented throughout the program. **Required Skills**: 1. Python programming 2. Familiarity with web scraping libraries such as BeautifulSoup, Scrapy, etc. 3. Experience with API integration 4. Data analysis and manipulation with pandas, NumPy, or similar libraries 5. Task automation and scheduling 6. Logging and error handling 7. Familiarity with JSON, CSV, or databases for data storage and output **Deliverables**: 1. Source code of the program. 2. Documentation detailing how to run, maintain, and troubleshoot the program. The documentation should also describe the data flow, and explain the logic behind how trending products are determined. 3. A sample output of the program. **Additional Notes**: 1. All web scraping should respect the target website's terms of service and [login to view URL] files, and comply with all relevant laws and regulations. 2. While developing, always consider the load you're putting on the target websites and design your program in such a way that it minimizes the potential disruption. 3. Ensure the program is designed with scalability and maintainability in mind. 4. Communication is key. Regular progress updates are expected.