FlatBuffers performance comparison

This repository comes with benchmarking code for different FlatBuffer implementations across programing languages.

Setup

• Benchmarks are CPU bound (no disk operations).
• Executed on a laptop CPU: Intel Core i7-8850H
• Dart uses JIT unless AOT is indicated.
• Dart benchmark_harness executes a 10-call timing loop repeatedly until 2 seconds have elapsed. The reported result is the average of the runtimes.
• Measured performance is in "operations per second": (10 * 1e6 / timeUs)

Results (more is better)

Date	Variant	Read	Write	Read (w.bytes)	Write (w.bytes)
2021-12-12	Dart official FB v2.0.5	11 636 660	7 907 070	9 185 325	6 176 130
2021-07-06	Dart official FB master	8 470 985	4 063 437	4 029 197	3 036 950
2021-07-06	Dart ObjectBox v1.1 FB	11 296 765	8 904 645	5 989 820	6 517 266
2021-02-26	Dart official FB v0.12	8 197 200	13 060	3 831 742	12 178
2021-02-26	Dart official FB master	8 878 415	3 284 045	4 032 722	2 592 991
2021-02-26	Dart ObjectBox v0.12 FB	9 074 660	5 044 802	4 732 340	4 212 460
2021-02-26	Dart AOT ObjectBox v0.12 FB	8 149 270	3 821 181	5 020 220	3 189 186
2021-02-26	Go	9 920 634	7 806 401	8 438 818	7 032 348

Note about results with a byte list: when reading a list from FlatBuffers in Dart, we call toList() to complete detach from the buffer (do not reference data inside the buffer). To be fair/have the numbers for comparison, we do the same in Go, though it doesn't have such a big impact there.

Benchmarked on 2021-12-12

• Dart SDK v2.13.4
• Dart flat_buffers v2.0.5

$ dart run ./benchmark/flatbuffers_official.dart
Measuring performance without byte list
Builder(RunTime): 1.264690966438896 us.
Reader(RunTime): 0.8593531133503943 us.
Measuring performance with byte list
Builder(RunTime): 1.6191369028825495 us.
Reader(RunTime): 1.0886931055787357 us.

Benchmarked on 2021-07-06

• Dart SDK v2.13.4
• Dart objectbox/flatbuffers fork - from objectbox-dart v1.1.0
• Dart flat_buffers master (as of 2012-07-06)
• Null safety enabled (no noticable performance improvement for this change alone)

Notes:

• read speed up by about 10 % was done by improving the benchmark to use const constructors of FB readers.
• both upstream master and ObjectBox FlatBuffers are now null-safe.

$ dart run ./benchmark/flatbuffers_official.dart
Measuring performance without byte list
Builder(RunTime): 2.4609702616502274 us.
Reader(RunTime): 1.1805002605954324 us.
Measuring performance with byte list
Builder(RunTime): 3.2927768109833697 us.
Reader(RunTime): 2.481883500446739 us.

$ dart run ./benchmark/flatbuffers_objectbox.dart
Measuring performance without byte list
Builder(RunTime): 1.123009326587794 us.
Reader(RunTime): 0.8852091727144895 us.
Measuring performance with byte list
Builder(RunTime): 1.5343855632803305 us.
Reader(RunTime): 1.6694992503948365 us.

Benchmarked on 2021-02-26 and before

• Dart SDK v2.10.5 (FlatBuffers)
• Dart SDK v2.12.0 (ObjectBox FlatBuffers)
• Dart flat_buffers v1.12.0
• Dart flat_buffers master (as of 2012-02-26) with some performance-related PRs we've got merged upstream
• Dart objectbox/flatbuffers fork - from objectbox-dart v0.12.0
• Go version go1.15.8
• Go flatbuffers v1.12.0

Dart JIT

# v0.12
$ pub run benchmark/flatbuffers_official.dart
Measuring performance without byte list
Builder(RunTime): 765.6720244929201 us.
Reader(RunTime): 1.2199287561606402 us.
Measuring performance with byte list
Builder(RunTime): 821.0898645876077 us.
Reader(RunTime): 2.6097788216872186 us.

# master
$ pub run benchmark/flatbuffers_official.dart
Measuring performance without byte list
Builder(RunTime): 3.0450252660971455 us.
Reader(RunTime): 1.1263271653780544 us.
Measuring performance with byte list
Builder(RunTime): 3.856550051195625 us.
Reader(RunTime): 2.4797140891084815 us.

$ pub run benchmark/flatbuffers_objectbox.dart
Measuring performance without byte list
Builder(RunTime): 1.9822381638140623 us.
Reader(RunTime): 1.1019696605713052 us.
Measuring performance with byte list
Builder(RunTime): 2.3739097819326473 us.
Reader(RunTime): 2.113119513813462 us.

Dart AOT

We measured only the fastest variant (ObjectBox fork):

$ dart2native benchmark/flatbuffers_objectbox.dart --output bench && ./bench
Generated: /flatbuffers_benchmark/flatbuffers_dart_benchmark/bench
Measuring performance without byte list
Builder(RunTime): 2.616991849387036 us.
Reader(RunTime): 1.227103777393558 us.
Measuring performance with byte list
Builder(RunTime): 3.1355955587468918 us.
Reader(RunTime): 1.9919445761341137 us.

Go plain results

To get (closer) to what Dart does: execute 10 iterations of the same operation for each measured loop.

$ go test -bench .
goos: linux
goarch: amd64
pkg: flatbuffers_go_benchmark
BenchmarkBuilderWithoutBytesList-12       928960              1281 ns/op
BenchmarkReaderWithoutBytesList-12       1226929              1008 ns/op
BenchmarkBuilderWithBytesList-12          819072              1422 ns/op
BenchmarkReaderWithBytesList-12          1000000              1185 ns/op