From 343fb9d2db66be6e3bb4fb06fd4ec7a09a865aab Mon Sep 17 00:00:00 2001 From: EmilLuta Date: Fri, 15 Nov 2024 02:35:37 +0100 Subject: [PATCH] some MD changes --- .../lib/circuit_prover_service/README.md | 22 ++++--- .../crates/lib/prover_job_processor/README.md | 58 +++++++++---------- 2 files changed, 36 insertions(+), 44 deletions(-) diff --git a/prover/crates/lib/circuit_prover_service/README.md b/prover/crates/lib/circuit_prover_service/README.md index 3748d090cda..3cc8a80e966 100644 --- a/prover/crates/lib/circuit_prover_service/README.md +++ b/prover/crates/lib/circuit_prover_service/README.md @@ -9,13 +9,13 @@ The primitives exported by this lib are job runners, namely: - heavy_wvg_runner - circuit_prover_runner -The rest of the codebase simply covers the internals of creating a runner, which is an implementation -of `ProverJobProcessor`. +The rest of the codebase simply covers the internals of creating a runner, which is an implementation of +`ProverJobProcessor`. ## Witness Vector Generator Runner -Runners related to synthesizing Witness Vector (the CPU heavy part of circuit proving). They are tied -to `prover_jobs_fri` table and operate over `ProverJobsFri` object storage bucket. +Runners related to synthesizing Witness Vector (the CPU heavy part of circuit proving). They are tied to +`prover_jobs_fri` table and operate over `ProverJobsFri` object storage bucket. Witness Vector Generators have big gaps in resource usages. Node proofs are the heavy jobs (~9GB RAM), whilst all others are rather light (~2GB RAM). @@ -26,17 +26,15 @@ There are 2 ways to deal with this: 2. run multiple light WVG jobs, with a small amount of heavy WVG jobs. This implementation favors number 2. As such, `MetadataLoader` abstraction was introduced to force loading lighter and -heavier jobs. -Heavier picker will try to prioritize nodes. If none are available, it falls back to light jobs in order to maximize -usage. +heavier jobs. Heavier picker will try to prioritize nodes. If none are available, it falls back to light jobs in order +to maximize usage. ### Job Picker Interacts with the database to get a job (as described above), loads the data from object store and then hydrates the -circuit. -In current implementation, Ram Permutation circuits are sent separately in order to save RAM in basic witness -generation & reduce the amount of storage used by object store. -A further optimization will be introduced later on, which will remove the necessity of witness hydration on circuits. +circuit. In current implementation, Ram Permutation circuits are sent separately in order to save RAM in basic witness +generation & reduce the amount of storage used by object store. A further optimization will be introduced later on, +which will remove the necessity of witness hydration on circuits. ### Executor @@ -95,4 +93,4 @@ sequenceDiagram cp_e-->>cp_s: Provide metadata & proof cp_s-->>os: Save proof cp_s-->>db: Update job metadata -``` \ No newline at end of file +``` diff --git a/prover/crates/lib/prover_job_processor/README.md b/prover/crates/lib/prover_job_processor/README.md index 535bc951294..a58d9b095bb 100644 --- a/prover/crates/lib/prover_job_processor/README.md +++ b/prover/crates/lib/prover_job_processor/README.md @@ -4,9 +4,8 @@ Prover Job Processor aims to be a small "framework" that allows building prover ## Context -Previously, prover components were hand tailored and had similar issues spread across the codebase. -The "framework"'s purpose is to standardize implementations and lift the undifferentiated work from prover component -developers. +Previously, prover components were hand tailored and had similar issues spread across the codebase. The "framework"'s +purpose is to standardize implementations and lift the undifferentiated work from prover component developers. ## How it works? @@ -20,18 +19,16 @@ The "framework" exports 4 main primitives: ### Executor This is the most important trait. It is meant to execute the crypto primitives (or any other payloads) and defines what -the inputs are, what is the metadata that has to travel with it and what the output will be. -Executors will receive information from Job Picker and will provide it further to Job Saver. +the inputs are, what is the metadata that has to travel with it and what the output will be. Executors will receive +information from Job Picker and will provide it further to Job Saver. For example, this could witness vector generator (synthesis of witness vector) or circuit prover (GPU circuit proving & -verification). -Each would define what they need as input to operate and what they'll output. +verification). Each would define what they need as input to operate and what they'll output. ### Job Picker The starting point of the process. This trait is tied to Executor and will pick a metadata & input that corresponds to -the Executor. -Job Picker picks information and provides it to Executor. +the Executor. Job Picker picks information and provides it to Executor. As examples, for witness vector generator it would be a query to the database & a query to object storage. For circuit prover, it would be waiting on the communication channel between witness vector generator and circuit prover. @@ -39,8 +36,7 @@ prover, it would be waiting on the communication channel between witness vector ### Job Saver The final point of the process. This trait is tied to Executor and will receive metadata & output that corresponds to -the Executor. -Job Saver receives information from Executor and saves it. +the Executor. Job Saver receives information from Executor and saves it. Continuing with the same examples, for witness vector generator it would send the information to the communication channel between witness vector generator & circuit prover. For circuit prover example, it would simply store the @@ -58,12 +54,12 @@ sequenceDiagram participant p as Job Picker participant e as Executor participant s as Job Saver - + p-->>p: Get metadata & input p-->>e: Provide metadata & input e-->>e: Execute e-->>s: Provide metadata & output - s-->>s: Save output + s-->>s: Save output ``` ## How to use it? @@ -78,9 +74,8 @@ With these out of the way, you can specify the Executor and even integrate the c you could fully cover it with unit tests to make sure the functionality works as intended. Moving forward, you'll need to understand where you get this information and where you store it. These are your Job -Picker & Job saver. -NOTE: Just like the executor, you need to implement the logic of executing/picking/saving a single job, the "framework" -will take care of looping it over and transmitting the details from one end to another. +Picker & Job saver. NOTE: Just like the executor, you need to implement the logic of executing/picking/saving a single +job, the "framework" will take care of looping it over and transmitting the details from one end to another. Once done, provide them as arguments to JobRunner, call `your_job_runner.run()` and you're good to go. @@ -93,8 +88,8 @@ There are a few things that we've glossed over, let's get into details: ### Back-offs & cancelling As you might've guessed, from a production point of view, you need to make sure that the process can die gracefully (k8s -sigterm), without being a nuisance to your dependencies (think DB or object store). -As such, job picker can have an optional component responsible for back-off & cancelling. +sigterm), without being a nuisance to your dependencies (think DB or object store). As such, job picker can have an +optional component responsible for back-off & cancelling. ### How do components communicate @@ -125,33 +120,32 @@ The "framework" wants to achieve the following: ### 1. Reduce code complexity & technical debt (modularize the codebase) Previously, most prover components were custom written. This meant that the same logic was reimplemented across multiple -components. -Whilst the "framework" doesn't fully solve the problem, it drastically reduces the amount of code needed to start a new -components. +components. Whilst the "framework" doesn't fully solve the problem, it drastically reduces the amount of code needed to +start a new components. The rest of the code duplication can be tackled in the future as part of the node framework. ### 2. Empower testability of the prover codebase -Due to the entangled nature of the code, prover codebase was difficult to test. -Current modular setup enables testing in isolation each component. (not exactly true, given cryptography dependencies -are too heavy - but will be true in the new prover implementation) +Due to the entangled nature of the code, prover codebase was difficult to test. Current modular setup enables testing in +isolation each component. (not exactly true, given cryptography dependencies are too heavy - but will be true in the new +prover implementation) ### 3. Optimize prover components for speed and multi-datacenter/multi-cloud setups Previously, provers were running "sync". Load job, once loaded, execute it, once executed, save it's result. Whilst this -is fine, all steps can be done in parallel. -This becomes super important when database and running machine are far away and the round trip to database can cause up -to 50% of the entire time. In a multi-cloud (read as future) setup, this becomes even more painful. +is fine, all steps can be done in parallel. This becomes super important when database and running machine are far away +and the round trip to database can cause up to 50% of the entire time. In a multi-cloud (read as future) setup, this +becomes even more painful. ### 4. Increase speed of delivery of prover components -Boojum release was rather slow and even releasing the current `circuit_prover` took longer than anticipated. -Given upcoming prover updates, this release sets us for success going forward. Furthermore, experimenting with different +Boojum release was rather slow and even releasing the current `circuit_prover` took longer than anticipated. Given +upcoming prover updates, this release sets us for success going forward. Furthermore, experimenting with different setups becomes a matter of days, rather than months. ### 5. Enable external shops to implement their own flavors of prover components -Most external folks have to fork zksync-era and keep an up-to-date fork if anything needs to be modified. -The framework allows using the executors, whilst defining custom pickers/savers. This will be a massive time-save for -any external shop that wants to innovate on top of zksync-era's provers. +Most external folks have to fork zksync-era and keep an up-to-date fork if anything needs to be modified. The framework +allows using the executors, whilst defining custom pickers/savers. This will be a massive time-save for any external +shop that wants to innovate on top of zksync-era's provers.