Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide benchmark code #22

Open
cfilipov opened this issue Jan 24, 2018 · 5 comments
Open

Provide benchmark code #22

cfilipov opened this issue Jan 24, 2018 · 5 comments

Comments

@cfilipov
Copy link

On your blog post introducing CwlSignal (emphasis added):

CwlSignal is fast; it can deliver multiple millions of values per second synchronously (more than 4 million per second per core on my Mac) and hundreds of thousands of values per second asynchronously (more than 200 000 values per second per core on my Mac). That makes it at least as fast as prominent competitor implementations and in most cases, between 2 and 10 times faster.

Can you provide more detail on how you arrived at this and what specific libraries you were comparing against? It would be great if others could be able to reproduce those results.

I ran into this repo recently and the result were unexpected considering the performance claim. CwlSignal ended up coming last in all tests, in some cases significantly.

I'm not trying to accuse you of misleading. Certainly there are going to be flaws in the way those tests are done*, but currently that's the only comparison of frp libs for ios that I could find.

* In particular, the version of each lib used in that repo had to be updated. One of the tests had to be modified because it was unfairly iterating far more on CwlSignal than others, etc...
@mattgallagher
Copy link
Owner

The benchmark I used for that article was the testSyncMapPerformance and testAsyncMapPerformance in the CwlSignal tests. I implemented equivalents in RxSwift and ReactiveSwift but I might need to dig them out of a backup.

I'll need to check these tests though. It sounds like something is wrong with either the tests or CwlSignal.

@cfilipov
Copy link
Author

Fixing some of the obvious issues with the test (for-in loop count on test 4, not using beta CwlSignal) brings things closer to expected perf, though still nowhere near the 2-10 times faster mark (of course these tests may likely not be covering those cases).

I really like the decisions made in this lib so as long as this isn't significantly slower than other reactive libs it's good enough for me.

Environment

macOS 10.13.3 (17D47)
2.6 GHz Intel Core i7, 16 GB 2133 MHz LPDDR3
Xcode Version 9.2 (9C40b)

Cartfile

github "ReactiveCocoa/ReactiveSwift" "2.0.0-rc.3"
github "ReactiveX/RxSwift" "4.1.1"
github "ReactiveKit/ReactiveKit" "v3.5.3"
github "mattgallagher/CwlSignal" "1.1.2"

Average times

Item ReactiveSwift RxSwift ReactiveKit CwlSignal
Test 1 0.044 s 0.034 s 0.033 s 0.037 s
Test 2 5.099 s 3.243 s 6.025 s 5.005 s
Test 3 5.198 s 5.133 s N/A 8.365 s
Test 4 30.706 s 39.615 s N/A 36.505 s
Full test output 

Test Suite 'Selected tests' started at 2018-01-23 16:28:28.982
Test Suite 'TestingFRPTests.xctest' started at 2018-01-23 16:28:28.983
Test Suite 'TestingFRPTests' started at 2018-01-23 16:28:28.984
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_1]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:68: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_1]' measured [Time, seconds] average: 0.037, relative standard deviation: 23.020%, values: [0.062069, 0.038210, 0.035723, 0.031896, 0.035400, 0.033061, 0.032896, 0.032278, 0.038264, 0.032627], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_1]' passed (0.641 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_2]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:137: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_2]' measured [Time, seconds] average: 5.005, relative standard deviation: 3.351%, values: [5.228273, 5.309745, 5.193727, 4.945750, 4.934049, 4.967629, 4.961953, 4.842207, 4.888626, 4.778883], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_2]' passed (50.304 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_3]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:187: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_3]' measured [Time, seconds] average: 8.365, relative standard deviation: 1.475%, values: [8.441410, 8.393016, 8.326471, 8.205041, 8.283094, 8.277699, 8.293259, 8.297448, 8.491969, 8.644860], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_3]' passed (83.912 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_4]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:250: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_4]' measured [Time, seconds] average: 36.505, relative standard deviation: 4.304%, values: [38.529602, 36.802578, 35.262177, 35.089258, 34.833321, 35.245257, 35.139443, 36.520165, 38.927094, 38.704399], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "Local Baseline", baselineAverage: 46.102, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_CwlSignal_4]' passed (365.308 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_1]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:53: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_1]' measured [Time, seconds] average: 0.033, relative standard deviation: 11.321%, values: [0.042841, 0.033231, 0.034467, 0.031033, 0.029359, 0.029152, 0.030983, 0.034416, 0.031538, 0.034288], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_1]' passed (0.585 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_2]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:115: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_2]' measured [Time, seconds] average: 6.025, relative standard deviation: 2.251%, values: [5.941797, 6.003043, 5.928967, 6.113958, 6.222029, 6.163573, 6.056419, 5.848333, 5.804356, 6.168137], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveKit_2]' passed (60.506 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_1]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:22: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_1]' measured [Time, seconds] average: 0.044, relative standard deviation: 9.912%, values: [0.055265, 0.046404, 0.044386, 0.041855, 0.041395, 0.040886, 0.040040, 0.040140, 0.042823, 0.042420], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_1]' passed (0.687 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_2]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:85: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_2]' measured [Time, seconds] average: 5.099, relative standard deviation: 1.135%, values: [5.115197, 5.119439, 5.119409, 5.146605, 5.033708, 5.102165, 5.103413, 5.144223, 5.151697, 4.954117], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_2]' passed (51.245 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_3]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:153: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_3]' measured [Time, seconds] average: 5.198, relative standard deviation: 2.964%, values: [5.432089, 5.286005, 4.983647, 5.275238, 5.325341, 5.226895, 5.072133, 5.316301, 4.930597, 5.131116], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_3]' passed (52.236 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_4]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:207: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_4]' measured [Time, seconds] average: 30.706, relative standard deviation: 1.695%, values: [30.448478, 31.650178, 30.843119, 30.823264, 30.641792, 30.107353, 29.702210, 31.039758, 31.183875, 30.616727], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_ReactiveSwift_4]' passed (307.310 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_1]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:38: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_1]' measured [Time, seconds] average: 0.034, relative standard deviation: 4.563%, values: [0.036757, 0.036498, 0.034696, 0.034235, 0.033963, 0.031600, 0.032806, 0.034076, 0.033764, 0.032384], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_1]' passed (0.596 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_2]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:100: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_2]' measured [Time, seconds] average: 3.243, relative standard deviation: 2.469%, values: [3.293827, 3.267836, 3.283611, 3.106348, 3.309114, 3.297825, 3.315767, 3.124179, 3.296485, 3.138304], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_2]' passed (32.689 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_3]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:168: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_3]' measured [Time, seconds] average: 5.133, relative standard deviation: 1.502%, values: [5.162558, 5.191683, 5.137047, 4.997215, 4.986285, 5.106888, 5.198257, 5.217204, 5.186524, 5.150147], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_3]' passed (51.586 seconds).
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_4]' started.
/Users/cfilipov/Workspace/TestingFRP/TestingFRPTests/TestingFRPTests.swift:227: Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_4]' measured [Time, seconds] average: 39.615, relative standard deviation: 10.528%, values: [37.366176, 36.730360, 36.027043, 35.534646, 36.143749, 35.849804, 43.124786, 46.086494, 44.734131, 44.556580], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[TestingFRPTests.TestingFRPTests test_measure_RxSwift_4]' passed (9599.283 seconds).
Test Suite 'TestingFRPTests' passed at 2018-01-23 19:26:05.880.
	 Executed 14 tests, with 0 failures (0 unexpected) in 10656.888 (10656.896) seconds
Test Suite 'TestingFRPTests.xctest' passed at 2018-01-23 19:26:05.880.
	 Executed 14 tests, with 0 failures (0 unexpected) in 10656.888 (10656.897) seconds
Test Suite 'Selected tests' passed at 2018-01-23 19:26:05.881.
	 Executed 14 tests, with 0 failures (0 unexpected) in 10656.888 (10656.899) seconds
Program ended with exit code: 0

@cfilipov
Copy link
Author

By the way, regarding this note on the blog post:

Note that the CwlSignal framework directly includes most of the files from the CwlUtils framework inside its own module so if you’re using CwlSignal, there’s no reason to also include CwlUtils (just use the same features via CwlSignal, instead). This manual inclusion of dependencies is certainly not ideal but is a workaround for an order of magnitude loss in performance due to Swift 3’s inability to specialize generics or inline functions between modules (and partly due to my desire to avoid reliance on package management until Swift Package Manager integrates with Xcode). I’m hopfeul that changes to Swift, Xcode and Swift Package Manager will allow a better solution in future.

I notice CwlSignal no longer includes CwlUtils directly anymore, does this mean that Swift now inlines across modules?

@mattgallagher
Copy link
Owner

I haven't forgotten about this issue... I do still want to provide some up-front benchmarking of CwlSignal. However, there's a lot about Swift and package management that affects performance – you often need to compromise for performance or convenience. I comment a little on this issue in this article:

https://www.cocoawithlove.com/blog/updating-for-2018.html#cwlsignal

@cfilipov
Copy link
Author

I can't believe I missed the update to your blog! Unrelated to this issue: it would be very helpful if you include "updated on" dates on your blog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattgallagher @cfilipov