Blocks Running Workflow

[FrancoGiachetta]

This PR adds a workflow which executes a small range of blocks both the vm and native, dumps the final state that the execution throws and compares with each other. If there's any diff, it will fail.

Closes #852

Checklist

• Linked to Github Issue
• Unit tests added
• Integration tests added.
• This change requires new documentation.
  • Documentation has been added/updated.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.82%. Comparing base (5648031) to head (586efc1).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #869   +/-   ##
=======================================
  Coverage   82.82%   82.82%           
=======================================
  Files         120      120           
  Lines       34941    34941           
=======================================
  Hits        28941    28941           
  Misses       6000     6000           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Benchmarking results

Benchmark for program factorial_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	11.065 ± 0.291	10.806	11.600	24.68 ± 0.65
cairo-native (embedded AOT)	3.330 ± 0.032	3.274	3.377	7.43 ± 0.07
cairo-native (embedded JIT using LLVM's ORC Engine)	3.374 ± 0.019	3.343	3.403	7.52 ± 0.04
cairo-native (standalone AOT)	0.673 ± 0.001	0.672	0.675	1.50 ± 0.00
cairo-native (standalone AOT with -march=native)	0.448 ± 0.000	0.448	0.449	1.00

Benchmark for program fib_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	11.135 ± 0.290	10.831	11.505	1378.50 ± 41.33
cairo-native (embedded AOT)	2.866 ± 0.030	2.832	2.915	354.83 ± 6.43
cairo-native (embedded JIT using LLVM's ORC Engine)	2.884 ± 0.027	2.840	2.927	357.05 ± 6.30
cairo-native (standalone AOT)	0.008 ± 0.000	0.008	0.009	1.04 ± 0.02
cairo-native (standalone AOT with -march=native)	0.008 ± 0.000	0.008	0.010	1.00

Benchmark for program logistic_map

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	4.451 ± 0.035	4.397	4.506	62.67 ± 0.59
cairo-native (embedded AOT)	3.000 ± 0.042	2.954	3.093	42.25 ± 0.63
cairo-native (embedded JIT using LLVM's ORC Engine)	3.170 ± 0.029	3.120	3.214	44.64 ± 0.47
cairo-native (standalone AOT)	0.115 ± 0.000	0.114	0.115	1.61 ± 0.01
cairo-native (standalone AOT with -march=native)	0.071 ± 0.000	0.071	0.073	1.00

[github-actions]

Benchmark results Main vs HEAD.

Command	Mean [s]	Min [s]	Max [s]	Relative
head factorial_2M.cairo (JIT)	3.123 ± 0.029	3.087	3.194	1.02 ± 0.01
base factorial_2M.cairo (JIT)	3.117 ± 0.021	3.089	3.166	1.01 ± 0.01
head factorial_2M.cairo (AOT)	3.073 ± 0.014	3.056	3.100	1.00
base factorial_2M.cairo (AOT)	3.092 ± 0.026	3.063	3.150	1.01 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
head fib_2M.cairo (JIT)	2.667 ± 0.019	2.635	2.698	1.02 ± 0.01
base fib_2M.cairo (JIT)	2.666 ± 0.016	2.631	2.683	1.02 ± 0.01
head fib_2M.cairo (AOT)	2.629 ± 0.016	2.603	2.647	1.00 ± 0.01
base fib_2M.cairo (AOT)	2.627 ± 0.016	2.605	2.660	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
head logistic_map.cairo (JIT)	2.937 ± 0.011	2.920	2.952	1.05 ± 0.01
base logistic_map.cairo (JIT)	2.942 ± 0.019	2.915	2.971	1.06 ± 0.01
head logistic_map.cairo (AOT)	2.788 ± 0.018	2.757	2.826	1.00
base logistic_map.cairo (AOT)	2.801 ± 0.014	2.779	2.822	1.00 ± 0.01

[pefontana]

Amaizing work @FrancoGiachetta !!!

[FrancoGiachetta]

Changes done

[JulianGCalderon]

I think this works, but it's difficult to say as it can only be tested in the CI. Just to make sure, could you test it with some failing blocks? That way we can ensure that the workflow detects blocks with diffs.

[FrancoGiachetta]

Yes, makes sense. I will replace some of them with the ones that seem to be failing recently

[JulianGCalderon]

It seems to be working OK. Should we tag the workflow as Required?

[pefontana]

It seems to be working OK. Should we tag the workflow as Required?

Sure! I will add them