Improve existing SPARK streaming job in SCALA to call out to external http api

Închis Postat la acum 3 ani S-au achitat serviciile după ce au fost prestate
Închis S-au achitat serviciile după ce au fost prestate

I have some existing code that works. It is written in SCALA. It is a spark streaming job that uses dataframes (not RDD) and reads in a JSON file and then makes a call to an external HTTP api endpoint and writes to KAFKA.

The end-to-end flow works OK but the call to external HTTP api is SLOW.

I use scalaj-http library to make that http call. This task is to take the existing code and replace only this library with something else to make it fast. At the moment, I am getting 50 tps(transations per second). I would like to achieve 2000tps or more. If you can achieve more TPS than that, you can be eligible for bonus. You can experiment with Akka-http, Play-WS, STTP, HTTP4S, RosHTTP, Dispatch etc.

Please NOTE: This is NOT a lot of work so please refrain from bidding crazy amounts for this project. Ideally this project will be a first step of many work done in future. Sensible bids please. Not looking for 3-figure jobs.

Message me for more details:

Scala Spark

ID Proiect: #29152531

Detalii despre proiect

5 propuneri Proiect la distanță Activ acum 3 ani

5 freelanceri plasează o ofertă medie de 166£ pentru proiect

nmogilip

Hi, Thank you for invite. I can help you with it. Please let’s connect and discuss more on your requirements. Thanks, Naresh.

£222 GBP în 5 zile
(9 recenzii)
4.8
mohanakrishnanba

Hi, I have experience with spark streaming and worked with rest API calls in scala Please reach me. Thanks, Mohanakrishnan

£100 GBP în 3 zile
(0 recenzii)
0.0
shivaupes

Hi .. I have 5 plus exp as a data engineer with good hands on implementing big data pipeline using Apache spark on different cloud platforms like azure and aws across both sql and no sql databases. I am a customer obse Mai multe

£156 GBP în 3 zile
(0 recenzii)
1.6
mohdmsl

I would check your code with some profiler.. and then would experiment with play-ws so to check if it can be fast.. and would try other things also

£100 GBP în 3 zile
(0 recenzii)
0.0