Improve existing SPARK streaming job in SCALA to call out to external http api
£20-250 GBP
S-au achitat serviciile după ce au fost prestate
I have some existing code that works. It is written in SCALA. It is a spark streaming job that uses dataframes (not RDD) and reads in a JSON file and then makes a call to an external HTTP api endpoint and writes to KAFKA.
The end-to-end flow works OK but the call to external HTTP api is SLOW.
I use scalaj-http library to make that http call. This task is to take the existing code and replace only this library with something else to make it fast. At the moment, I am getting 50 tps(transations per second). I would like to achieve 2000tps or more. If you can achieve more TPS than that, you can be eligible for bonus. You can experiment with Akka-http, Play-WS, STTP, HTTP4S, RosHTTP, Dispatch etc.
Please NOTE: This is NOT a lot of work so please refrain from bidding crazy amounts for this project. Ideally this project will be a first step of many work done in future. Sensible bids please. Not looking for 3-figure jobs.
Message me for more details:
ID Proiect: #29152531
Detalii despre proiect
5 freelanceri plasează o ofertă medie de 166£ pentru proiect
Hi, Thank you for invite. I can help you with it. Please let’s connect and discuss more on your requirements. Thanks, Naresh.
Hi, I have experience with spark streaming and worked with rest API calls in scala Please reach me. Thanks, Mohanakrishnan
I would check your code with some profiler.. and then would experiment with play-ws so to check if it can be fast.. and would try other things also