Python optimization

• Stare: Closed
• Premiu: $40
• Intrări primite: 15
• Câștigător: KyrillosRizkalla

Winner

Sumarul concursului

Hi everybody,
the task is to optimize a Python script (3.11) that is working fine, that parses xml and writes them to Mongodb.

Activities:

1. Optimize the script according to a set of rules that will be shared privately. For example, insert timestamps on the "last read", "last write", "read error", "write error" fields on the DB. Another example is to optimize the type of call to retrieve these xmls, urls etc

2. Set up a structured reading mechanism in the script, in order to handle both latency and "resource currently unavailable" and other similar scenarios etc. The service should not stop, but continue to work. If a resource is not scanned, setup the errors write sa described in point 1., the service continues to work.
The same mechanism goes for writing.

3. Modify the "data writing logic" on the various collections of the db, structuring a Work Queue (in retry, if necessary). Reading and writing should be asynchronous processes. Currently, the script reads and writes simultaneously, making it slow.

4. Set up writing optimization mechanisms with a set of controls. When a xml has been written correctly, it updates the corresponding timestamp. The number of items not scanned by the xml should be reported on the db in a dedicated field.

5. Last but not least, the service must be performant and scalable. The resources to be scanned are in the order of thousands per hour, so the chosen technology must be scalable.

I'll share the script and few other little details, please contact me.
I'll assign the job to the one's that will have proven experience in python / scrape high experience and after receiving a written proposal (1 page) of how you will manage the job, what techniques you'll apply and technologies/library used.

Thanks

Aptitudinile recomandate

• MongoDB
• Python
• Arhitectură software