Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Era vm integration v2 #240

Draft
wants to merge 125 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
7458c5a
Brush up `fill_bootloader` benchmark
slowli Aug 5, 2024
f9afe6d
Benchmark transfers in `fill_bootloader`
slowli Aug 5, 2024
fc0352c
Benchmark tx failures in `fill_bootloader`
slowli Aug 5, 2024
e1db8a7
Document `fill_bootloader` benchmark
slowli Aug 5, 2024
e3e6a95
Make benchmarks parametric by VM
slowli Aug 6, 2024
18c230a
Exclude VM initialization / drop from measured time
slowli Aug 6, 2024
bdd5076
Bench load test transactions
slowli Aug 6, 2024
e82afe0
Add realistic / heavy load test transactions
slowli Aug 6, 2024
98e5cbc
Brush up FIXMEs
slowli Aug 6, 2024
afef861
Move `pop_snapshot_no_rollback()`
slowli Aug 6, 2024
5931b7c
Document load test benches
slowli Aug 6, 2024
b5fc63e
Run bench tests in CI
slowli Aug 6, 2024
c6302f6
Reduce txs in batch for tests
slowli Aug 6, 2024
bb41a48
Add era vm as dependency
IAvecilla Jul 23, 2024
ae7ef29
Bump this error version to match with era_vm version
IAvecilla Jul 23, 2024
ddb2b0d
Add new era_vm module to multivm crate
IAvecilla Jul 23, 2024
84e3629
Add vm creation logic and world storage for vm
IAvecilla Jul 23, 2024
48b2d9d
Utils and aux structs to match with trait implementation
IAvecilla Jul 23, 2024
f52f77c
Add vm snapshots
IAvecilla Jul 24, 2024
ee1e2e1
Apply format
IAvecilla Jul 24, 2024
7f0d6b6
Fix errors after merge for transaction data struct
IAvecilla Jul 24, 2024
7fff0b1
Delete feature of era_vm for old zksync-era version
IAvecilla Jul 24, 2024
1f922a2
Add missing modules for to integrate Lambda VM for benchmarking
IAvecilla Jul 25, 2024
1109ca8
Add bootloader state and event as modules
IAvecilla Jul 25, 2024
7bfa94d
Implement inspect function to execute transactions in lambda VM
IAvecilla Jul 25, 2024
ffebbeb
Change VM in benchmarks to use lambda vm
IAvecilla Jul 25, 2024
8d200da
Apply zk formatter
IAvecilla Jul 25, 2024
af7d1df
Add bootloader memory initialization
IAvecilla Jul 26, 2024
b6293ff
Fix decommit and implement write to storage
IAvecilla Jul 31, 2024
80a173c
Fix heap write for bootloader
IAvecilla Jul 31, 2024
fa0d75b
Add implementation for remaining gas function
IAvecilla Jul 31, 2024
ebf4110
Add implementation of l2 to l1 logs for worlds storage
IAvecilla Jul 31, 2024
00f44d6
Add test using the benchmarks setup and test bytecode
IAvecilla Jul 31, 2024
978245e
Add hooks logic to vm
IAvecilla Aug 6, 2024
8deca6f
Fix format
IAvecilla Aug 6, 2024
15cf1ad
Compile with new eravm changes
gianbelinche Aug 8, 2024
839fd27
Add current integration
gianbelinche Aug 8, 2024
d5c866d
Fix conflicts with era vm
gianbelinche Aug 12, 2024
a7615ab
Merge remote-tracking branch 'origin/aov-pla-1006-benchmark-vm-perfor…
Oppen Aug 12, 2024
6bde7e7
Fix build after merge
Oppen Aug 12, 2024
4603b17
Lambda VM criterion benchmarks
Oppen Aug 13, 2024
bea26df
Don't unwrap on TxHasEnded
Oppen Aug 13, 2024
4401569
bench: enable fill_bootloader benchmark for Lambda VM
Oppen Aug 14, 2024
a768368
bench: enable IAI benchmarks for Lambda VM
Oppen Aug 14, 2024
1cdea93
perf: reduce cloning on VM creation
Oppen Aug 14, 2024
c1ba1a4
fix: override iai to our fork newer valgrind compat
Oppen Aug 14, 2024
019c576
multivm: use local era_vm dependency
fkrause98 Aug 14, 2024
43c62d1
fix: properly set path for dep
fkrause98 Aug 14, 2024
f0c4951
fix: era_vm local path
fkrause98 Aug 15, 2024
a9079fa
fix: setup local dep for ci
fkrause98 Aug 15, 2024
314f0ff
Addapt era_vm to new world struct
MarcosNicolau Aug 15, 2024
9d89bee
Adapt to new VmState naming
MarcosNicolau Aug 15, 2024
08cf08f
Merge pull request #232 from lambdaclass/world-adaptation
MarcosNicolau Aug 15, 2024
8d09216
Adapt era_vm to new Storage
MarcosNicolau Aug 15, 2024
3b5cfd3
Merge pull request #233 from lambdaclass/pubdata-refunds-adapt
MarcosNicolau Aug 16, 2024
32a6e62
Add basic tester module for era_vm
MarcosNicolau Aug 16, 2024
c58540b
Add tests utils
MarcosNicolau Aug 16, 2024
0424f0e
Add basic bootloader tests
MarcosNicolau Aug 16, 2024
6768958
Add bootloader initial gas to vm
MarcosNicolau Aug 16, 2024
44337c5
Fix Execution instantiation
Aug 19, 2024
e306a2e
Fix tester module
MarcosNicolau Aug 20, 2024
31f0252
Add missing hooks Hook parsing
MarcosNicolau Aug 20, 2024
40e0d60
Add into system_log function
MarcosNicolau Aug 20, 2024
6a44603
Remove old tests
MarcosNicolau Aug 20, 2024
37f06af
Add get_current_execution_state
MarcosNicolau Aug 20, 2024
0e7c3e1
Add execution logs and statistics
MarcosNicolau Aug 20, 2024
c4cb6d6
Implement PostResult hook
MarcosNicolau Aug 20, 2024
fabe14b
Implement PubDataRequested hook
MarcosNicolau Aug 20, 2024
2b7c27c
Implement refunds
MarcosNicolau Aug 20, 2024
b53c810
Add push transaction inner to pass custom predefined refund
MarcosNicolau Aug 20, 2024
7ab61ac
Implement snapshots and rollbacks
MarcosNicolau Aug 20, 2024
2e2f63f
Make Vm needed fields and methods public
MarcosNicolau Aug 20, 2024
83c5ab4
Test util function to sort execution state
MarcosNicolau Aug 20, 2024
d4f4fdb
Add all remaining tests to era_vm
MarcosNicolau Aug 20, 2024
a525e56
Run zk fmt
MarcosNicolau Aug 20, 2024
ac13bf4
Fix PubDataRequested get_storage_diff
MarcosNicolau Aug 20, 2024
fcc3e70
Address warnings
MarcosNicolau Aug 20, 2024
96c6c9b
Merge branch 'era_vm_integration_v2' into tests/era_vm
MarcosNicolau Aug 20, 2024
32e98e7
Update get_storage_changes_from_snapshot return type
MarcosNicolau Aug 20, 2024
93ba138
Add whole state to VmDump
MarcosNicolau Aug 20, 2024
9cd4f22
Merge pull request #234 from lambdaclass/tests/era_vm
MarcosNicolau Aug 20, 2024
db03643
Fix hook 7 for benchmarks (#238)
JulianVentura Aug 20, 2024
9154b1d
Revert "Fix hook 7 for benchmarks (#238)"
MarcosNicolau Aug 22, 2024
67d2d32
Merge pull request #242 from lambdaclass/revert-last-commit
MarcosNicolau Aug 22, 2024
c2c6cd0
Bootloader trait
MarcosNicolau Aug 22, 2024
f097401
Manager and dispatcher tracers
MarcosNicolau Aug 22, 2024
f5a4b3f
Create basic tracers skeleton
MarcosNicolau Aug 22, 2024
ff82b18
Implement tracers on vm run method
MarcosNicolau Aug 22, 2024
24ef9c8
Refactor bootloader tracer
MarcosNicolau Aug 23, 2024
1bddb9b
Implement bootloader tracers in vm
MarcosNicolau Aug 23, 2024
6803dc6
zk fmt
MarcosNicolau Aug 23, 2024
0334465
Add bootloader hook fn to tracer
MarcosNicolau Aug 23, 2024
d72b6fc
Implement refund tracer
MarcosNicolau Aug 23, 2024
149bb70
Implement pubdata tracer
MarcosNicolau Aug 23, 2024
e0a1890
Implement result tracer
MarcosNicolau Aug 23, 2024
5010795
Move result parsing to tracer manager
MarcosNicolau Aug 23, 2024
fe31ef9
Implement call tracer for era_vm
MarcosNicolau Aug 23, 2024
69796c7
Accept tracer param in inspect
MarcosNicolau Aug 23, 2024
43604eb
Add after_vm function to VmTrait
MarcosNicolau Aug 23, 2024
8abf5c7
Implement circuits tracer
MarcosNicolau Aug 24, 2024
9b91063
Add precompiles tests
MarcosNicolau Aug 24, 2024
6d94011
General tracers refactor
MarcosNicolau Aug 24, 2024
36752f3
Add block_tip test
MarcosNicolau Aug 24, 2024
210d8ca
Remove result tracer and moved logic into vm run
MarcosNicolau Aug 26, 2024
3683239
General refactor
MarcosNicolau Aug 26, 2024
611dc84
Update vm new execute for tracers
MarcosNicolau Aug 26, 2024
5312aad
Fix opcode imports in call_tracer
MarcosNicolau Aug 27, 2024
172ccae
Add optional debug tracer
MarcosNicolau Aug 28, 2024
96bf03d
initial commit
juan518munoz Aug 28, 2024
ec31b9e
Remove ignore flag in call_tracer test
MarcosNicolau Aug 28, 2024
8dca5a6
fix: leverage zksync home to fetch benchmark paths (#248)
fkrause98 Aug 28, 2024
de0eb52
Add support of zk server with lambda vm
MarcosNicolau Aug 30, 2024
a31ae3a
Add finish_batch and sort storage_logs
MarcosNicolau Aug 30, 2024
602af24
Update vm_runner config to run our vm
MarcosNicolau Aug 30, 2024
5fb9466
Fix blob_tracer return when running program
MarcosNicolau Aug 30, 2024
96cb8dc
Merge pull request #251 from lambdaclass/fix-tracer
MarcosNicolau Aug 30, 2024
ef59fec
Merge branch 'implement-tracers' into era_vm_zk_server_v2
gianbelinche Sep 2, 2024
9773fe8
Finish merge
gianbelinche Sep 2, 2024
593c7ea
Make server compile
gianbelinche Sep 2, 2024
0e0c257
Increase timeout
gianbelinche Sep 2, 2024
87dd05c
Adress PR comments
gianbelinche Sep 3, 2024
ec01627
Merge pull request #253 from lambdaclass/era_vm_zk_server_and_tracers
gianbelinche Sep 4, 2024
f6be6d9
Merge branch 'era_vm_integration_v2' into era_vm_zk_server_v2
gianbelinche Sep 4, 2024
81f61c8
Merge pull request #254 from lambdaclass/era_vm_zk_server_v2
gianbelinche Sep 4, 2024
13d45a4
Merge pull request #247 from lambdaclass/era_vm_integration_rollbacka…
gianbelinche Sep 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci-core-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,11 @@ jobs:
run: ci_run yarn l1-contracts test

- name: Rust unit tests
run: ci_run zk test rust
run: |
ci_run zk test rust
# Benchmarks are not tested by `cargo nextest` unless specified explicitly, and even then `criterion` harness is incompatible
# with how `cargo nextest` runs tests. Thus, we run criterion-based benchmark tests manually.
ci_run zk f cargo test --release -p vm-benchmark --bench criterion --bench fill_bootloader

loadtest:
runs-on: [matterlabs-ci-runner]
Expand Down
54 changes: 50 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ strum = "0.26"
tempfile = "3.0.2"
test-casing = "0.1.2"
test-log = "0.2.15"
thiserror = "1"
thiserror = "1.0.62"
thread_local = "1.1"
tikv-jemallocator = "0.5"
tiny-keccak = "2"
Expand Down Expand Up @@ -298,3 +298,6 @@ zksync_contract_verification_server = { version = "0.1.0", path = "core/node/con
zksync_node_api_server = { version = "0.1.0", path = "core/node/api_server" }
zksync_tee_verifier_input_producer = { version = "0.1.0", path = "core/node/tee_verifier_input_producer" }
zksync_base_token_adjuster = { version = "0.1.0", path = "core/node/base_token_adjuster" }

[patch.crates-io]
iai = { git = "https://github.com/lambdaclass/iai", branch = "fix/valgrind-forward-compatibility" }
2 changes: 2 additions & 0 deletions core/lib/multivm/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ zk_evm_1_4_0.workspace = true
zk_evm_1_3_3.workspace = true
zk_evm_1_3_1.workspace = true
vm2.workspace = true
era_vm = {path = "../../../../"}
# era_vm = {git = "https://github.com/lambdaclass/era_vm.git", branch = "main"}

circuit_sequencer_api_1_3_3.workspace = true
circuit_sequencer_api_1_4_0.workspace = true
Expand Down
21 changes: 19 additions & 2 deletions core/lib/multivm/src/glue/tracers/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
//! - Add this trait as a trait bound for `T` in `MultiVMTracer` implementation.
//! - Implement the trait for `T` with a bound to `VmTracer` for a specific version.

use zksync_state::WriteStorage;
use zksync_state::{ImmutableStorageView, WriteStorage};

use crate::{tracers::old_tracers::OldTracers, HistoryMode};

Expand All @@ -44,6 +44,7 @@ pub trait MultiVMTracer<S: WriteStorage, H: HistoryMode>:
+ IntoVm1_4_1IntegrationTracer<S, H>
+ IntoVm1_4_2IntegrationTracer<S, H>
+ IntoOldVmTracer
+ IntoEraVmTracer<S, H>
{
fn into_tracer_pointer(self) -> MultiVmTracerPointer<S, H>
where
Expand All @@ -57,6 +58,10 @@ pub trait IntoLatestTracer<S: WriteStorage, H: HistoryMode> {
fn latest(&self) -> crate::vm_latest::TracerPointer<S, H::Vm1_5_0>;
}

pub trait IntoEraVmTracer<S: WriteStorage, H: HistoryMode> {
fn into_era_vm(&self) -> Box<dyn crate::era_vm::tracers::traits::VmTracer<S>>;
}

pub trait IntoVmVirtualBlocksTracer<S: WriteStorage, H: HistoryMode> {
fn vm_virtual_blocks(
&self,
Expand Down Expand Up @@ -106,6 +111,17 @@ where
}
}

impl<S, T, H> IntoEraVmTracer<S, H> for T
where
S: WriteStorage,
H: HistoryMode,
T: crate::era_vm::tracers::traits::VmTracer<S> + Clone + 'static,
{
fn into_era_vm(&self) -> Box<dyn crate::era_vm::tracers::traits::VmTracer<S>> {
Box::new(self.clone())
}
}

impl<S, T, H> IntoVmVirtualBlocksTracer<S, H> for T
where
S: WriteStorage,
Expand Down Expand Up @@ -180,6 +196,7 @@ where
+ IntoVmBoojumIntegrationTracer<S, H>
+ IntoVm1_4_1IntegrationTracer<S, H>
+ IntoVm1_4_2IntegrationTracer<S, H>
+ IntoOldVmTracer,
+ IntoOldVmTracer
+ IntoEraVmTracer<S, H>,
{
}
4 changes: 2 additions & 2 deletions core/lib/multivm/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ pub use crate::{
tracers::{MultiVMTracer, MultiVmTracerPointer},
},
versions::{
vm_1_3_2, vm_1_4_1, vm_1_4_2, vm_boojum_integration, vm_fast, vm_latest, vm_m5, vm_m6,
vm_refunds_enhancement, vm_virtual_blocks,
era_vm, vm_1_3_2, vm_1_4_1, vm_1_4_2, vm_boojum_integration, vm_fast, vm_latest, vm_m5,
vm_m6, vm_refunds_enhancement, vm_virtual_blocks,
},
vm_instance::VmInstance,
};
Expand Down
Loading