Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weekly benchmark report - 2023 #127

Open
fcollonval opened this issue Jan 11, 2023 · 20 comments
Open

Weekly benchmark report - 2023 #127

fcollonval opened this issue Jan 11, 2023 · 20 comments
Labels
documentation Improvements or additions to documentation

Comments

@fcollonval
Copy link
Member

This issue gathers the results of the weekly scheduled benchmark tests.

Those test are run against JupyterLab master comparing the HEAD to the commit one week older than the HEAD; so every benchmark test is the evolution compared to the previous test.

For more information please refer to the documentation.

Older reports are in: #85
@fcollonval fcollonval added the documentation Improvements or additions to documentation label Jan 11, 2023
@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file errorOutputs - 10x100 errors output collected
open
chromium
head 466 <- [492 - 503 - 515] -> 660
expected 461 <- [486 - 501 - 515] -> 674
Mean relative change 0.1% ± 1.6%
switch-from-copy
chromium
head 118 <- [142 - 156 - 217] -> 398
expected 110 <- [146 - 164 - 215] -> 411
Mean relative change -1.7% ± 4.6%
switch-to-copy
chromium
head 503 <- [506 - 509 - 514] -> 541
expected 503 <- [506 - 509 - 514] -> 535
Mean relative change 0.0% ± 0.2%
switch-from-txt
chromium
head 76 <- [97 - 105 - 114] -> 202
expected 65 <- [100 - 110 - 122] -> 187
Mean relative change -3.9% ± 2.4%
switch-to-txt
chromium
head 91 <- [111 - 121 - 141] -> 279
expected 95 <- [111 - 119 - 137] -> 270
Mean relative change 1.3% ± 3.7%
search
chromium
head 729 <- [794 - 805 - 829] -> 903
expected 756 <- [793 - 807 - 830] -> 918
Mean relative change -0.2% ± 1.0%
close
chromium
head 144 <- [172 - 186 - 201] -> 267
expected 159 <- [177 - 187 - 198] -> 283
Mean relative change -0.9% ± 3.9%

Changes are computed with expected as reference.


Expected = 8d42fc0af0e5579aea881680d6e169a24fa56c69 | Actual = 282ec9129964ccefec362c61eaef6340f9274eb2
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-15 02:52:48.088779107 +0000
+++ /dev/fd/62	2023-01-15 02:52:48.088779107 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_md_100_notebook
open
chromium
head 729 <- [827 - 862 - 893] -> 1322
expected 722 <- [829 - 851 - 871] -> 944
Mean relative change 2.8% ± 2.2%
switch-from-copy
chromium
head 353 <- [383 - 427 - 604] -> 1090
expected 346 <- [386 - 432 - 590] -> 1013
Mean relative change 0.1% ± 5.2%
switch-to-copy
chromium
head 511 <- [527 - 537 - 545] -> 728
expected 504 <- [524 - 533 - 545] -> 644
Mean relative change 0.6% ± 0.6%
switch-from-txt
chromium
head 142 <- [169 - 183 - 308] -> 558
expected 140 <- [168 - 181 - 312] -> 502
Mean relative change -0.0% ± 5.7%
switch-to-txt
chromium
head 260 <- [294 - 314 - 422] -> 684
expected 259 <- [296 - 317 - 436] -> 634
Mean relative change -0.7% ± 3.9%
search
chromium
head 1166 <- [1300 - 1360 - 1429] -> 1600
expected 1128 <- [1287 - 1360 - 1420] -> 1571
Mean relative change 0.6% ± 1.9%
close
chromium
head 239 <- [265 - 277 - 296] -> 457
expected 242 <- [261 - 273 - 287] -> 384
Mean relative change 3.3% ± 3.4%

Changes are computed with expected as reference.


Expected = 8d42fc0af0e5579aea881680d6e169a24fa56c69 | Actual = 282ec9129964ccefec362c61eaef6340f9274eb2
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-15 03:15:18.327388232 +0000
+++ /dev/fd/62	2023-01-15 03:15:18.327388232 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_code_100_notebook
open
chromium
head 567 <- [617 - 637 - 660] -> 784
expected 577 <- [616 - 635 - 661] -> 748
Mean relative change 0.2% ± 1.5%
switch-from-copy
chromium
head 415 <- [453 - 497 - 956] -> 1688
expected 412 <- [457 - 485 - 971] -> 1724
Mean relative change -0.5% ± 8.4%
switch-to-copy
chromium
head 516 <- [593 - 613 - 631] -> 717
expected 525 <- [598 - 612 - 632] -> 730
Mean relative change -0.4% ± 0.8%
switch-from-txt
chromium
head 130 <- [175 - 192 - 222] -> 651
expected 142 <- [179 - 193 - 228] -> 458
Mean relative change -0.9% ± 5.2%
switch-to-txt
chromium
head 210 <- [243 - 256 - 272] -> 434
expected 214 <- [243 - 254 - 271] -> 450
Mean relative change 0.9% ± 1.9%
search
chromium
head 823 <- [987 - 1016 - 1046] -> 1332
expected 836 <- [971 - 995 - 1035] -> 1289
Mean relative change 0.8% ± 2.1%
close
chromium
head 227 <- [263 - 280 - 299] -> 406
expected 233 <- [268 - 283 - 298] -> 391
Mean relative change -0.7% ± 2.7%

Changes are computed with expected as reference.


Expected = 8d42fc0af0e5579aea881680d6e169a24fa56c69 | Actual = 282ec9129964ccefec362c61eaef6340f9274eb2
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-15 03:28:01.297523434 +0000
+++ /dev/fd/62	2023-01-15 03:28:01.297523434 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_metadata_1000x100_notebook
open
chromium
head 3957 <- [4023 - 4045 - 4072] -> 4145
expected 4030 <- [4082 - 4101 - 4126] -> 4517
Mean relative change -1.5% ± 0.3%
switch-from-copy
chromium
head 150 <- [204 - 229 - 338] -> 726
expected 145 <- [208 - 244 - 343] -> 725
Mean relative change -3.7% ± 6.5%
switch-to-copy
chromium
head 503 <- [504 - 505 - 508] -> 803
expected 503 <- [505 - 505 - 508] -> 796
Mean relative change 0.3% ± 1.4%
switch-from-txt
chromium
head 67 <- [103 - 111 - 119] -> 144
expected 46 <- [101 - 109 - 117] -> 144
Mean relative change 1.7% ± 1.9%
switch-to-txt
chromium
head 30 <- [50 - 65 - 103] -> 302
expected 37 <- [52 - 75 - 106] -> 136
Mean relative change -4.9% ± 6.1%
close
chromium
head 251 <- [276 - 288 - 304] -> 354
expected 258 <- [279 - 290 - 302] -> 375
Mean relative change -0.5% ± 1.9%

Changes are computed with expected as reference.


Expected = 8d42fc0af0e5579aea881680d6e169a24fa56c69 | Actual = 282ec9129964ccefec362c61eaef6340f9274eb2
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-15 03:30:27.193000827 +0000
+++ /dev/fd/62	2023-01-15 03:30:27.193000827 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file longOutput - A single output with 100x100 divs
open
chromium
head 1230 <- [1349 - 1373 - 1446] -> 1752
expected 1252 <- [1336 - 1374 - 1407] -> 1771
Mean relative change 1.1% ± 1.7%
switch-from-copy
chromium
head 753 <- [823 - 879 - 938] -> 1186
expected 768 <- [818 - 866 - 940] -> 1246
Mean relative change 0.4% ± 1.5%
switch-to-copy
chromium
head 864 <- [971 - 1013 - 1048] -> 1252
expected 879 <- [975 - 1010 - 1046] -> 1231
Mean relative change -0.1% ± 1.0%
switch-from-txt
chromium
head 166 <- [187 - 196 - 207] -> 248
expected 171 <- [188 - 197 - 209] -> 298
Mean relative change -1.1% ± 1.4%
switch-to-txt
chromium
head 681 <- [742 - 780 - 863] -> 1070
expected 695 <- [742 - 787 - 865] -> 1035
Mean relative change -0.1% ± 1.6%
search
chromium
head 23472 <- [25974 - 26349 - 27063] -> 29663
expected 23605 <- [26240 - 26504 - 27133] -> 29441
Mean relative change -0.2% ± 1.5%
close
chromium
head 290 <- [317 - 330 - 345] -> 411
expected 285 <- [314 - 328 - 341] -> 376
Mean relative change 1.2% ± 1.8%

Changes are computed with expected as reference.


Expected = 8d42fc0af0e5579aea881680d6e169a24fa56c69 | Actual = 282ec9129964ccefec362c61eaef6340f9274eb2
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-15 05:39:36.469497147 +0000
+++ /dev/fd/62	2023-01-15 05:39:36.469497147 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file errorOutputs - 10x100 errors output collected
open
chromium
head 461 <- [494 - 506 - 518] -> 651
expected 460 <- [487 - 500 - 515] -> 690
Mean relative change 0.8% ± 1.9%
switch-from-copy
chromium
head 107 <- [148 - 167 - 221] -> 454
expected 123 <- [145 - 160 - 211] -> 473
Mean relative change 6.1% ± 5.3%
switch-to-copy
chromium
head 502 <- [506 - 510 - 513] -> 544
expected 503 <- [506 - 511 - 515] -> 572
Mean relative change -0.3% ± 0.2%
switch-from-txt
chromium
head 68 <- [99 - 111 - 119] -> 174
expected 51 <- [96 - 105 - 115] -> 199
Mean relative change 2.1% ± 2.5%
switch-to-txt
chromium
head 91 <- [108 - 116 - 125] -> 184
expected 94 <- [111 - 120 - 135] -> 258
Mean relative change -9.7% ± 2.9%
search
chromium
head 746 <- [796 - 809 - 829] -> 941
expected 670 <- [798 - 813 - 834] -> 900
Mean relative change -0.1% ± 1.1%
close
chromium
head 151 <- [181 - 194 - 219] -> 310
expected 148 <- [180 - 186 - 207] -> 260
Mean relative change 3.4% ± 4.0%

Changes are computed with expected as reference.


Expected = d4688fe784c2b7d8cb9af8a153af88c91fd8dcc8 | Actual = 6658ffc9d893c993e339932e9e25433f92341673
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-22 02:52:11.425919841 +0000
+++ /dev/fd/62	2023-01-22 02:52:11.425919841 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_code_100_notebook
open
chromium
head 406 <- [442 - 461 - 478] -> 571
expected 416 <- [439 - 452 - 465] -> 588
Mean relative change 1.6% ± 1.8%
switch-from-copy
chromium
head 246 <- [298 - 336 - 619] -> 1424
expected 259 <- [298 - 330 - 585] -> 1366
Mean relative change 4.7% ± 9.1%
switch-to-copy
chromium
head 503 <- [508 - 513 - 519] -> 641
expected 503 <- [509 - 512 - 516] -> 591
Mean relative change 0.5% ± 0.4%
switch-from-txt
chromium
head 92 <- [120 - 132 - 151] -> 399
expected 76 <- [117 - 126 - 144] -> 272
Mean relative change 4.2% ± 4.0%
switch-to-txt
chromium
head 134 <- [155 - 168 - 182] -> 268
expected 138 <- [152 - 161 - 177] -> 286
Mean relative change 4.2% ± 2.4%
search
chromium
head 767 <- [891 - 909 - 945] -> 1075
expected 849 <- [885 - 904 - 943] -> 1201
Mean relative change 0.1% ± 1.6%
close
chromium
head 152 <- [173 - 186 - 203] -> 312
expected 145 <- [173 - 188 - 200] -> 343
Mean relative change 1.4% ± 4.6%

Changes are computed with expected as reference.


Expected = d4688fe784c2b7d8cb9af8a153af88c91fd8dcc8 | Actual = 6658ffc9d893c993e339932e9e25433f92341673
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-22 03:00:26.998047711 +0000
+++ /dev/fd/62	2023-01-22 03:00:26.998047711 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_md_100_notebook
open
chromium
head 674 <- [751 - 778 - 794] -> 967
expected 652 <- [750 - 768 - 789] -> 892
Mean relative change 0.8% ± 1.4%
switch-from-copy
chromium
head 286 <- [323 - 368 - 533] -> 1006
expected 293 <- [323 - 358 - 531] -> 870
Mean relative change 2.8% ± 5.9%
switch-to-copy
chromium
head 503 <- [510 - 515 - 521] -> 696
expected 503 <- [509 - 514 - 519] -> 565
Mean relative change 0.6% ± 0.4%
switch-from-txt
chromium
head 114 <- [141 - 152 - 248] -> 441
expected 117 <- [140 - 150 - 249] -> 326
Mean relative change 2.1% ± 5.5%
switch-to-txt
chromium
head 216 <- [245 - 267 - 374] -> 567
expected 216 <- [249 - 268 - 350] -> 503
Mean relative change 3.3% ± 3.9%
search
chromium
head 1078 <- [1163 - 1211 - 1260] -> 1448
expected 1103 <- [1192 - 1231 - 1293] -> 1398
Mean relative change -1.9% ± 1.5%
close
chromium
head 187 <- [215 - 228 - 246] -> 319
expected 184 <- [211 - 223 - 239] -> 292
Mean relative change 3.8% ± 3.1%

Changes are computed with expected as reference.


Expected = d4688fe784c2b7d8cb9af8a153af88c91fd8dcc8 | Actual = 6658ffc9d893c993e339932e9e25433f92341673
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-22 03:05:53.126936353 +0000
+++ /dev/fd/62	2023-01-22 03:05:53.126936353 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_metadata_1000x100_notebook
open
chromium
head 4023 <- [4081 - 4105 - 4146] -> 4340
expected 4005 <- [4085 - 4115 - 4160] -> 4273
Mean relative change 0.0% ± 0.4%
switch-from-copy
chromium
head 143 <- [209 - 229 - 362] -> 1133
expected 145 <- [213 - 251 - 361] -> 795
Mean relative change 0.8% ± 8.3%
switch-to-copy
chromium
head 503 <- [505 - 506 - 513] -> 852
expected 503 <- [505 - 506 - 509] -> 784
Mean relative change 0.4% ± 1.7%
switch-from-txt
chromium
head 65 <- [104 - 114 - 123] -> 143
expected 62 <- [105 - 113 - 123] -> 167
Mean relative change 0.3% ± 1.9%
switch-to-txt
chromium
head 32 <- [53 - 78 - 108] -> 150
expected 40 <- [61 - 101 - 112] -> 151
Mean relative change -12.3% ± 4.7%
close
chromium
head 262 <- [284 - 295 - 313] -> 408
expected 257 <- [289 - 301 - 315] -> 538
Mean relative change -0.8% ± 2.7%

Changes are computed with expected as reference.


Expected = d4688fe784c2b7d8cb9af8a153af88c91fd8dcc8 | Actual = 6658ffc9d893c993e339932e9e25433f92341673
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-22 03:30:50.065877999 +0000
+++ /dev/fd/62	2023-01-22 03:30:50.065877999 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file longOutput - A single output with 100x100 divs
open
chromium
head 1329 <- [1466 - 1528 - 1599] -> 2035
expected 1309 <- [1442 - 1503 - 1567] -> 1948
Mean relative change 1.4% ± 2.1%
switch-from-copy
chromium
head 811 <- [913 - 968 - 1062] -> 1846
expected 797 <- [901 - 962 - 1050] -> 1379
Mean relative change 1.7% ± 1.9%
switch-to-copy
chromium
head 922 <- [1065 - 1132 - 1199] -> 1421
expected 874 <- [990 - 1081 - 1158] -> 1387
Mean relative change 4.6% ± 1.6%
switch-from-txt
chromium
head 171 <- [198 - 210 - 221] -> 277
expected 165 <- [193 - 204 - 219] -> 345
Mean relative change 1.8% ± 1.6%
switch-to-txt
chromium
head 755 <- [863 - 919 - 1003] -> 1260
expected 738 <- [838 - 893 - 969] -> 1249
Mean relative change 3.3% ± 1.8%
search
chromium
head 21428 <- [24236 - 25432 - 26272] -> 28669
expected 21151 <- [23799 - 24869 - 25942] -> 28858
Mean relative change 1.7% ± 1.8%
close
chromium
head 311 <- [337 - 353 - 370] -> 424
expected 298 <- [327 - 341 - 357] -> 418
Mean relative change 3.1% ± 2.0%

Changes are computed with expected as reference.


Expected = d4688fe784c2b7d8cb9af8a153af88c91fd8dcc8 | Actual = 6658ffc9d893c993e339932e9e25433f92341673
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-01-22 05:47:30.580082844 +0000
+++ /dev/fd/62	2023-01-22 05:47:30.580082844 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "109.0.5414.46"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_code_100_notebook
open
chromium
head 454 <- [480 - 492 - 504] -> 642
expected 399 <- [454 - 460 - 472] -> 514
Mean relative change ⚠️ 7.4% ± 1.5%
switch-from-copy
chromium
head 311 <- [378 - 403 - 943] -> 1306
expected 206 <- [287 - 321 - 614] -> 1026
Mean relative change ⚠️ 34.4% ± 11.4%
switch-to-copy
chromium
head 503 <- [506 - 511 - 517] -> 553
expected 502 <- [510 - 515 - 521] -> 563
Mean relative change ⚠️ -0.6% ± 0.3%
switch-from-txt
chromium
head 87 <- [127 - 140 - 151] -> 191
expected 76 <- [124 - 134 - 149] -> 271
Mean relative change ⚠️ -2.1% ± 3.2%
switch-to-txt
chromium
head 210 <- [241 - 252 - 263] -> 387
expected 131 <- [151 - 161 - 170] -> 239
Mean relative change ⚠️ 56.8% ± 2.8%
search
chromium
head 974 <- [983 - 987 - 991] -> 1051
expected 975 <- [979 - 983 - 988] -> 1007
Mean relative change ⚠️ 0.6% ± 0.3%
close
chromium
head 181 <- [221 - 245 - 305] -> 434
expected 186 <- [209 - 219 - 237] -> 347
Mean relative change ⚠️ 18.3% ± 6.8%

Changes are computed with expected as reference.


Expected = ac2807fdea9e2d431dce10419c82336cd23ebd17 | Actual = 63f32800e989c0113d1b8caec34d4f530e094e74
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-12 03:02:10.752941328 +0000
+++ /dev/fd/62	2023-02-12 03:02:10.752941328 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_md_100_notebook
open
chromium
head 690 <- [766 - 779 - 799] -> 1047
expected 624 <- [675 - 697 - 724] -> 898
Mean relative change 10.8% ± 2.0%
switch-from-copy
chromium
head 282 <- [341 - 361 - 718] -> 1010
expected 206 <- [270 - 294 - 565] -> 874
Mean relative change 20.6% ± 8.4%
switch-to-copy
chromium
head 503 <- [515 - 522 - 532] -> 618
expected 502 <- [512 - 516 - 522] -> 549
Mean relative change 2.1% ± 0.5%
switch-from-txt
chromium
head 103 <- [137 - 151 - 163] -> 228
expected 116 <- [137 - 148 - 263] -> 363
Mean relative change -19.6% ± 3.6%
switch-to-txt
chromium
head 245 <- [271 - 283 - 295] -> 432
expected 129 <- [155 - 212 - 240] -> 433
Mean relative change 38.5% ± 4.3%
search
chromium
head 1117 <- [1136 - 1144 - 1158] -> 1282
expected 1133 <- [1162 - 1176 - 1245] -> 1380
Mean relative change -3.8% ± 1.0%
close
chromium
head 205 <- [229 - 241 - 253] -> 358
expected 194 <- [221 - 235 - 249] -> 303
Mean relative change 4.2% ± 2.7%

Changes are computed with expected as reference.


Expected = ac2807fdea9e2d431dce10419c82336cd23ebd17 | Actual = 63f32800e989c0113d1b8caec34d4f530e094e74
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-12 03:07:35.378009258 +0000
+++ /dev/fd/62	2023-02-12 03:07:35.378009258 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file errorOutputs - 10x100 errors output collected
open
chromium
head 386 <- [652 - 684 - 733] -> 996
expected 574 <- [637 - 660 - 700] -> 937
Mean relative change 2.9% ± 3.7%
switch-from-copy
chromium
head 149 <- [203 - 229 - 326] -> 619
expected 138 <- [183 - 203 - 297] -> 531
Mean relative change 9.6% ± 5.6%
switch-to-copy
chromium
head 503 <- [510 - 515 - 523] -> 560
expected 503 <- [509 - 515 - 523] -> 556
Mean relative change 0.1% ± 0.3%
switch-from-txt
chromium
head 104 <- [140 - 153 - 166] -> 225
expected 84 <- [122 - 131 - 142] -> 200
Mean relative change 16.2% ± 2.6%
switch-to-txt
chromium
head 131 <- [153 - 166 - 181] -> 235
expected 96 <- [131 - 141 - 153] -> 216
Mean relative change 16.9% ± 2.5%
search
chromium
head 945 <- [957 - 965 - 975] -> 1053
expected 948 <- [959 - 965 - 974] -> 1024
Mean relative change 0.0% ± 0.5%
close
chromium
head 235 <- [275 - 293 - 328] -> 463
expected 224 <- [262 - 283 - 317] -> 424
Mean relative change 5.1% ± 4.7%

Changes are computed with expected as reference.


Expected = ac2807fdea9e2d431dce10419c82336cd23ebd17 | Actual = 63f32800e989c0113d1b8caec34d4f530e094e74
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-12 03:09:23.543372550 +0000
+++ /dev/fd/62	2023-02-12 03:09:23.543372550 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_metadata_1000x100_notebook
open
chromium
head 396 <- [466 - 481 - 498] -> 636
expected 352 <- [430 - 453 - 502] -> 600
Mean relative change ⚠️ 4.2% ± 4.6%
switch-from-copy
chromium
head 202 <- [259 - 321 - 455] -> 752
expected 190 <- [234 - 252 - 464] -> 747
Mean relative change ⚠️ 12.9% ± 11.0%
switch-to-copy
chromium
head 505 <- [511 - 517 - 523] -> 770
expected 503 <- [510 - 519 - 535] -> 749
Mean relative change ⚠️ -1.4% ± 2.5%
switch-from-txt
chromium
head 104 <- [132 - 143 - 158] -> 203
expected 81 <- [106 - 119 - 128] -> 171
Mean relative change ⚠️ 24.0% ± 4.3%
switch-to-txt
chromium
head 101 <- [121 - 129 - 139] -> 175
expected 82 <- [104 - 112 - 122] -> 291
Mean relative change ⚠️ 14.6% ± 4.1%
close
chromium
head 321 <- [348 - 360 - 373] -> 409
expected 310 <- [344 - 364 - 387] -> 865
Mean relative change ⚠️ -6.4% ± 8.6%

Changes are computed with expected as reference.


Expected = ac2807fdea9e2d431dce10419c82336cd23ebd17 | Actual = 63f32800e989c0113d1b8caec34d4f530e094e74
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-12 03:27:16.122769979 +0000
+++ /dev/fd/62	2023-02-12 03:27:16.122769979 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file longOutput - A single output with 100x100 divs
open
chromium
head 1132 <- [1173 - 1189 - 1215] -> 1440
expected 1065 <- [1159 - 1201 - 1227] -> 1340
Mean relative change ⚠️ 0.5% ± 1.3%
switch-from-copy
chromium
head 639 <- [678 - 720 - 779] -> 1021
expected 599 <- [659 - 698 - 777] -> 977
Mean relative change ⚠️ 2.3% ± 1.7%
switch-to-copy
chromium
head 663 <- [698 - 719 - 756] -> 914
expected 653 <- [688 - 701 - 731] -> 844
Mean relative change ⚠️ 2.6% ± 1.0%
switch-from-txt
chromium
head 142 <- [168 - 177 - 190] -> 222
expected 131 <- [152 - 159 - 168] -> 211
Mean relative change ⚠️ 11.7% ± 1.6%
switch-to-txt
chromium
head 567 <- [614 - 643 - 699] -> 881
expected 569 <- [604 - 638 - 692] -> 878
Mean relative change ⚠️ 1.2% ± 1.6%
search
chromium
head 6883 <- [7038 - 7171 - 9094] -> 9872
expected 6880 <- [7047 - 7147 - 7371] -> 9539
Mean relative change ⚠️ 2.2% ± 3.7%
close
chromium
head 271 <- [301 - 324 - 347] -> 408
expected 262 <- [289 - 308 - 326] -> 432
Mean relative change ⚠️ 4.3% ± 2.8%

Changes are computed with expected as reference.


Expected = ac2807fdea9e2d431dce10419c82336cd23ebd17 | Actual = 63f32800e989c0113d1b8caec34d4f530e094e74
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-12 04:10:11.638008493 +0000
+++ /dev/fd/62	2023-02-12 04:10:11.638008493 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file errorOutputs - 10x100 errors output collected
open
chromium
head 464 <- [501 - 515 - 531] -> 725
expected 463 <- [497 - 511 - 525] -> 683
Mean relative change 2.0% ± 2.1%
switch-from-copy
chromium
head 114 <- [148 - 165 - 218] -> 368
expected 108 <- [144 - 159 - 209] -> 391
Mean relative change 4.1% ± 4.2%
switch-to-copy
chromium
head 502 <- [506 - 510 - 516] -> 541
expected 502 <- [507 - 510 - 515] -> 541
Mean relative change 0.0% ± 0.2%
switch-from-txt
chromium
head 81 <- [104 - 116 - 125] -> 150
expected 74 <- [102 - 112 - 122] -> 154
Mean relative change 2.9% ± 2.1%
switch-to-txt
chromium
head 95 <- [116 - 124 - 133] -> 179
expected 92 <- [115 - 122 - 132] -> 166
Mean relative change 2.9% ± 1.9%
search
chromium
head 912 <- [917 - 920 - 929] -> 963
expected 913 <- [917 - 920 - 923] -> 960
Mean relative change 0.4% ± 0.3%
close
chromium
head 163 <- [189 - 200 - 213] -> 332
expected 154 <- [182 - 191 - 206] -> 329
Mean relative change 6.0% ± 4.2%

Changes are computed with expected as reference.


Expected = edfbea4f9c556875882dfbe71bf77d31e81e5883 | Actual = 91751d133da99465ce0ef4f882e82a3f9de3b28d
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-19 02:50:41.181808864 +0000
+++ /dev/fd/62	2023-02-19 02:50:41.181808864 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_code_100_notebook
open
chromium
head 445 <- [472 - 488 - 504] -> 646
expected 437 <- [468 - 477 - 494] -> 599
Mean relative change ⚠️ 2.5% ± 1.9%
switch-from-copy
chromium
head 308 <- [375 - 402 - 944] -> 1368
expected 306 <- [379 - 401 - 976] -> 1407
Mean relative change ⚠️ -1.1% ± 9.4%
switch-to-copy
chromium
head 502 <- [505 - 508 - 514] -> 590
expected 503 <- [508 - 512 - 518] -> 567
Mean relative change ⚠️ -0.7% ± 0.4%
switch-from-txt
chromium
head 97 <- [128 - 139 - 150] -> 209
expected 92 <- [131 - 139 - 150] -> 216
Mean relative change ⚠️ -1.7% ± 2.4%
switch-to-txt
chromium
head 209 <- [247 - 257 - 272] -> 351
expected 207 <- [240 - 250 - 262] -> 370
Mean relative change ⚠️ 3.2% ± 1.8%
search
chromium
head 969 <- [986 - 992 - 1002] -> 1043
expected 972 <- [980 - 984 - 989] -> 1041
Mean relative change ⚠️ 0.9% ± 0.5%
close
chromium
head 192 <- [233 - 248 - 298] -> 566
expected 167 <- [214 - 232 - 258] -> 552
Mean relative change ⚠️ 10.1% ± 9.0%

Changes are computed with expected as reference.


Expected = edfbea4f9c556875882dfbe71bf77d31e81e5883 | Actual = 91751d133da99465ce0ef4f882e82a3f9de3b28d
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-19 03:00:22.678058837 +0000
+++ /dev/fd/62	2023-02-19 03:00:22.678058837 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_md_100_notebook
open
chromium
head 705 <- [757 - 775 - 792] -> 1017
expected 702 <- [753 - 770 - 791] -> 849
Mean relative change 0.8% ± 1.2%
switch-from-copy
chromium
head 292 <- [344 - 362 - 714] -> 933
expected 286 <- [342 - 359 - 710] -> 1224
Mean relative change 0.5% ± 6.4%
switch-to-copy
chromium
head 502 <- [515 - 521 - 533] -> 594
expected 503 <- [512 - 520 - 529] -> 601
Mean relative change 0.5% ± 0.6%
switch-from-txt
chromium
head 99 <- [137 - 147 - 162] -> 251
expected 99 <- [137 - 148 - 159] -> 221
Mean relative change 0.1% ± 1.9%
switch-to-txt
chromium
head 244 <- [273 - 284 - 296] -> 420
expected 239 <- [270 - 281 - 291] -> 404
Mean relative change 1.4% ± 1.1%
search
chromium
head 1124 <- [1133 - 1141 - 1152] -> 1331
expected 1116 <- [1136 - 1144 - 1156] -> 1346
Mean relative change -0.5% ± 0.9%
close
chromium
head 206 <- [224 - 235 - 248] -> 282
expected 202 <- [226 - 235 - 246] -> 289
Mean relative change 0.4% ± 2.0%

Changes are computed with expected as reference.


Expected = edfbea4f9c556875882dfbe71bf77d31e81e5883 | Actual = 91751d133da99465ce0ef4f882e82a3f9de3b28d
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-19 03:06:39.208519919 +0000
+++ /dev/fd/62	2023-02-19 03:06:39.208519919 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file large_metadata_1000x100_notebook
open
chromium
head 4347 <- [4446 - 4513 - 4583] -> 6287
expected 4296 <- [4437 - 4479 - 4551] -> 4687
Mean relative change 1.1% ± 0.9%
switch-from-copy
chromium
head 178 <- [204 - 222 - 264] -> 466
expected 175 <- [207 - 222 - 262] -> 828
Mean relative change -0.1% ± 3.7%
switch-to-copy
chromium
head 503 <- [506 - 510 - 514] -> 535
expected 503 <- [505 - 508 - 514] -> 541
Mean relative change 0.2% ± 0.2%
switch-from-txt
chromium
head 76 <- [99 - 111 - 122] -> 163
expected 76 <- [105 - 113 - 122] -> 181
Mean relative change -1.9% ± 2.1%
switch-to-txt
chromium
head 85 <- [103 - 110 - 121] -> 159
expected 82 <- [104 - 111 - 119] -> 220
Mean relative change -0.2% ± 1.8%
close
chromium
head 271 <- [299 - 317 - 333] -> 376
expected 266 <- [299 - 315 - 331] -> 363
Mean relative change 0.4% ± 1.9%

Changes are computed with expected as reference.


Expected = edfbea4f9c556875882dfbe71bf77d31e81e5883 | Actual = 91751d133da99465ce0ef4f882e82a3f9de3b28d
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-19 03:34:48.608633906 +0000
+++ /dev/fd/62	2023-02-19 03:34:48.608633906 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@github-actions
Copy link

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table
Test file longOutput - A single output with 100x100 divs
open
chromium
head 1022 <- [1052 - 1061 - 1073] -> 1107
expected 1013 <- [1051 - 1061 - 1069] -> 1105
Mean relative change ⚠️ 0.1% ± 0.4%
switch-from-copy
chromium
head 569 <- [601 - 625 - 677] -> 821
expected 567 <- [602 - 624 - 678] -> 946
Mean relative change ⚠️ 0.0% ± 1.3%
switch-to-copy
chromium
head 594 <- [614 - 623 - 642] -> 755
expected 596 <- [618 - 628 - 641] -> 729
Mean relative change ⚠️ -0.2% ± 0.7%
switch-from-txt
chromium
head 114 <- [138 - 150 - 160] -> 200
expected 109 <- [137 - 150 - 160] -> 195
Mean relative change ⚠️ -0.2% ± 1.7%
switch-to-txt
chromium
head 518 <- [554 - 573 - 622] -> 738
expected 524 <- [555 - 570 - 620] -> 735
Mean relative change ⚠️ 0.2% ± 1.2%
search
chromium
head 5773 <- [5884 - 5939 - 6028] -> 9569
expected 5805 <- [5886 - 5945 - 6053] -> 9645
Mean relative change ⚠️ 0.3% ± 3.0%
close
chromium
head 204 <- [233 - 243 - 254] -> 339
expected 206 <- [233 - 241 - 249] -> 339
Mean relative change ⚠️ 0.9% ± 2.7%

Changes are computed with expected as reference.


Expected = edfbea4f9c556875882dfbe71bf77d31e81e5883 | Actual = 91751d133da99465ce0ef4f882e82a3f9de3b28d
Go to action log
Changelog covered

❗ Test metadata have changed 
--- /dev/fd/63	2023-02-19 03:51:31.368559078 +0000
+++ /dev/fd/62	2023-02-19 03:51:31.368559078 +0000
@@ -1,7 +1,7 @@
 {
   "benchmark": {
     "BENCHMARK_OUTPUTFILE": "lab-benchmark.json",
-    "BENCHMARK_REFERENCE": "head"
+    "BENCHMARK_REFERENCE": "actual"
   },
   "browsers": {
     "chromium": "110.0.5481.38"

@fcollonval fcollonval pinned this issue Jun 5, 2023
@fcollonval fcollonval unpinned this issue Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fcollonval