-
|
Hi folks 👋🏽 , We're using the Typelevel stack at work on a greenfield project, so I took some time to run some Gatling load tests we have to compare the HTTP response times performance using the latest RC. We also use Http4s with the Ember server implementation. Here's the version matrix I've managed to test, all with Scala 3.2.2:
We run our service on AWS and the main database is DynamoDB. Though, all these tests were run on my machine against Localstack. The load test performs multiple CRUD operations regarding a user profile (simply a User ID and a name for now). Low input values
Cats Effect 3.4.8 / Http4s 0.23.18Response time P99: 126 ms / Max: 177 ms
Cats Effect 3.5.0-RC3 / Http4s 0.23.19-RC3Response time P99: 137 ms / Max: 607 ms
Cats Effect 3.5.0-RC3 / Http4s 0.23.18I've run this one only to ensure changes in Http4s didn't affect the results, and it seems consistent across multiple runs. Response time P99: 136 ms / Max: 1682 ms
High input values
Cats Effect 3.4.8 / Http4s 0.23.18Response time P99: 18.848 seconds / Max: 19.850 seconds
Cats Effect 3.5.0-RC3 / Http4s 0.23.19-RC3Response time P99: 19.247 seconds / Max: 20.077 seconds
SummaryMultiple runs using different input values seem consistent, showing a ~2% decrease in response times. However, the main bottleneck in our case is DynamoDB I/O, so we have bigger issues to solve 😄 (luckily this is not PROD yet), but I thought it was worth reporting. Now I realize this may not be of great help, given that the code belongs to the company I work for, so I can't share implementation details. Though, I'd be happy to run further tests if you think I could try other settings / versions, etc. |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 8 replies
-
|
This is immensely helpful thank you! This is in line with @wjoel's results and suggests that my I think at a minimum we need to rework the timers data structure and fix the busy-loop scenario. |
Beta Was this translation helpful? Give feedback.
-
|
Following up on this: any chance you would be willing to take a look at |
Beta Was this translation helpful? Give feedback.
-
|
That's awesome to hear, will definitely give it a shot on Monday and report back, appreciate all the work you guys do 🙇🏽 |
Beta Was this translation helpful? Give feedback.
-
Cats Effect 3.5.0-RC5Following up with testing the latest RC5. Here's the version matrix I've managed to test, all with Scala 3.2.2:
Once again, all these tests were run on my machine against Localstack (AWS platform). The load test performs multiple CRUD operations regarding a user profile (simply a User ID and a name for now). Low input values
Cats Effect 3.4.9 / Http4s 0.23.18Response time P99: 145 ms / Max: 264 ms
Cats Effect 3.5.0-RC5 / Http4s 0.23.19-RC3Response time P99: 161 ms / Max: 1727 ms
Cats Effect 3.5.0-RC5 / Http4s 0.23.18Response time P99: 123 ms / Max: 161 ms
High input values
Cats Effect 3.4.9 / Http4s 0.23.18Response time P99: 18.151 seconds / Max: 21.639 seconds
Cats Effect 3.5.0-RC5 / Http4s 0.23.19-RC3Response time P99: 20.260 seconds / Max: 20.915 seconds
Cats Effect 3.5.0-RC5 / Http4s 0.23.18Two runs with the same arguments.
SummaryThere still seem to be a slight decrease in response times, even though one of the tests with low input values gave better results. Didn't have much more time to keep analyzing this (busy times at work!), but I could try to follow up over the weekend. |
Beta Was this translation helpful? Give feedback.
-
|
@gvolpe it's a lot to ask (this has been super helpful so far, thank you!), but would you be able to try http4s 0.23.19 RC1 and RC2 as well? I'm hoping we can narrow down when exactly the performance regression happened and hopefully do something about it. |
Beta Was this translation helpful? Give feedback.
-
|
@armanbilge not a problem, I'll try to make time for it, because these tests take ages to run! 😄 |
Beta Was this translation helpful? Give feedback.
-
|
In the meantime though, I think this (taken together with @wjoel's results) is the golden signal to move forward with 3.5 from a performance standpoint. |
Beta Was this translation helpful? Give feedback.
-
|
@armanbilge the regression seems to have happened on RC3. Here's the updated results: P99 response times for low input values
P99 response times for high input values
I didn't even test Http4s 0.23.19-RC1 as it seemed redundant :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! From those numbers it seems that even RC2 is a bit slower, but indeed most of the regression seems to be RC3. Hum, what could it be 🤔 |
Beta Was this translation helpful? Give feedback.
This is immensely helpful thank you! This is in line with @wjoel's results and suggests that my
heybenchmarks are genuinely not capturing a representative scenario.I think at a minimum we need to rework the timers data structure and fix the busy-loop scenario.