diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png deleted file mode 100644 index 0ef98d2e07..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png deleted file mode 100644 index cc7e5d030a..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png deleted file mode 100644 index ff553f6aba..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png deleted file mode 100644 index 05f0e34efd..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png deleted file mode 100644 index 2f841e1645..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png deleted file mode 100644 index 44f6b6c1a1..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png deleted file mode 100644 index e9ff23b366..0000000000 Binary files a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png and /dev/null differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png new file mode 100644 index 0000000000..2f3632306c Binary files /dev/null and b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png differ diff --git a/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png new file mode 100644 index 0000000000..478fcfedcf Binary files /dev/null and b/en/docs/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png differ diff --git a/en/docs/install-and-setup/setup/deployment-best-practices/performance-tests-results-async.md b/en/docs/install-and-setup/setup/deployment-best-practices/performance-tests-results-async.md index 8be42cb9c2..9b96c634bd 100644 --- a/en/docs/install-and-setup/setup/deployment-best-practices/performance-tests-results-async.md +++ b/en/docs/install-and-setup/setup/deployment-best-practices/performance-tests-results-async.md @@ -25,7 +25,7 @@ The performance tests were conducted for streaming APIs of WSO2 API Manager. Tes Apache JMeter Client c5.large 2 - 2 + 4 Apache JMeter Server 01 @@ -55,43 +55,25 @@ The performance tests were conducted for streaming APIs of WSO2 API Manager. Tes WSO2 API Manager c5.large 2 - 2 + 4 - Token type: JWT - The operating system is Ubuntu 18.04 -- MySQL version in RDS instance is 5.7 -- Java version is JDK 11 +- MySQL version in RDS instance is 8.0 +- Java version is Temurin JDK 17 ### Performance test scripts All scripts used to run the performance tests and analyze results are in the following repositories. -- [https://github.com/wso2/performance-common](https://github.com/wso2/performance-common) - - [https://github.com/wso2/performance-apim](https://github.com/wso2/performance-apim) ## Results -## Server Sent Events - -Test for SSE API was done by using an SSE back end which emitted events in different frequencies. The error rate with -concurrent users and the CPU and memory percentage consumed was recorded at that time. - -The following graph depicts the Error %, CPU % and Memory % variation in the server during the time in which the -request was processed. - -[![Server Sent Events Test Results]({{base_path}}/assets/img/setup-and-install/performance-test-results/sse_perf_test_results.png)]({{base_path}}/assets/img/setup-and-install/performance-test-results/sse_perf_test_results.png) - -**Key observations:** - -- Five thousand (5000) concurrent connections can be handled with zero errors, with an event rate of 10 Event/sec (Average size of an Event used was 82B) - -- CPU spike is expected during the processing time, however it will settle once the requests are served. - -## WebSub +### WebSub Test for WebSub API was done for varying number of subscribers and measuring the associated error, CPU and Memory % consumed. @@ -112,15 +94,11 @@ this will settle when the events are delivered. 10,000 concurrent users mean a lot, and it is not very common. It is recommended to scale horizontally or vertically to support more concurrent users with acceptable response times. When scaling horizontally, two or more Gateway nodes need to be used with a load balancer. Another load test must be carried out to measure the performance after scaling. -## WebSocket - -Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The ramp up period, error rate, concurrent connections, and CPU and memory usages were recorded during that time. - -The following graph depicts the ramp up period with error rate. +### WebSocket -websocket_ramp_up_period_vs_error_rate +Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The CPU and memory usages for concurrent connections were recorded during that time. The tests were run for 30 seconds. -The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message. Size of the payload was reduced when non-zero error rates were observed without any failed connections. +The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message. | Events/second | Message Size | Attempted Concurrent Connections | Failed Connections | Error Rate | |----------------|--------------|----------------------------------|--------------------|------------| @@ -128,43 +106,32 @@ The following table shows the number of concurrent connections with error rate. | | | 200 | 0 | 0% | | | | 300 | 0 | 0% | | | | 400 | 0 | 0% | -| | | 600 | 48 | 0% | -| | | 800 | 78 | 0% | -| | | 1000 | 74 | 0% | +| | | 600 | 0 | 0% | +| | | 800 | 0 | 0% | +| | | 1000 | 0 | 0% | | 5 | 500 bytes | 100 | 0 | 0% | | | | 200 | 0 | 0% | | | | 300 | 0 | 0% | | | | 400 | 0 | 0% | -| | | 600 | 42 | 32.90% | -| | | 800 | 64 | 65.25% | -| | | 1000 | 62 | 70.14% | +| | | 600 | 0 | 3.24% | +| | | 800 | 0 | 44.54% | +| | | 1000 | 0 | 45.59% | | 10 | 500 bytes | 100 | 0 | 0% | | | | 200 | 0 | 0% | -| | | 300 | 0 | 46.10% | -| | 250 bytes | 300 | 0 | 14.29% | -| | 125 bytes | 300 | 0 | 12.70% | -| | 60 bytes | 300 | 0 | 29.12% | -| | | 400 | 4 | 62.17% | -| | | 600 | 56 | 85.14% | +| | | 300 | 0 | 0% | +| | | 400 | 0 | 16.72% | +| | | 600 | 0 | 57.60% | + +!!! note + The backend used here is a Tornado backend. The error rates observed in the above graph are due to limitations in the Tornado backend. API Manager + was able to succesfully handle upto 1000 concurrent connections even in the 10 events per second scenario. **Key observations:** -- 400 concurrent connections can be made with zero error rate and without any failed connections, at five events per second. -- 200 concurrent connections can be made with zero error rate and without any failed connections, at ten events per second. +- 1000 concurrent connections can be made without any failed connections, at ten events per second. The following graphs depict the variation of CPU and Memory % with concurrent connections. -100_connections - -200_connections - -300_connections - -400_connections - -600_connections - -800_connections - -1000_connections +CPU Usage +Memory Usage