-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Renderfarm submitters
WIP Sorry, there is no detailed tutorial or guide for this at the moment. Basically you need to set up one of the following Render Farm environments, and create a suitable submitter. If the referenced Render Farm Submitter is detected, Meshroom will show an additional button. Setting up the environment can be complicated, so it is nothing for the casual user. (You would need different servers (physical or virtual) that can distribute and manage the Job to connected machines (Example). There will be a tutorial or guide on this in the future, but for now you are on your own. (So please don´t open a new issue on this)
https://renderman.pixar.com/tractor https://renderman.pixar.com/forum/download.php https://renderman.pixar.com/store
meshroom/submitters/simpleFarmSubmitter.py meshroom/submitters/simpleFarmConfig.json
https://alicevision.org/img/meshroom/renderfarm.png
https://github.com/alicevision/meshroom/pull/81 https://materialsproject.github.io/fireworks
https://github.com/alicevision/meshroom/issues/1039 https://www.youtube.com/watch?v=OYVZvlXdBsg (reference)
[Experimental]
- Get the Meshroom OpenCue Submitter https://github.com/alicevision/meshroom/pull/992
- Place it in the Meshroom Submitters folder
- The Meshroom Submitter is using OpenCue PyOutline.
- Cuebot server with OpenCue database, RQD Render Agents on the target machines, PyCue and PyOutline on the primary machine are required to be set up. https://www.opencue.io/docs/concepts/opencue-overview/
- The process is described here: https://www.opencue.io/docs/getting-started/
- RQD can now run a submitted Meshroom Job on the render-farm machines
This can theoretically be setup in the Google Cloud environment as described here: https://cloud.google.com/solutions/creating-a-render-farm-on-gcp-using-opencue
https://www.youtube.com/watch?v=uOi3azKJ3Xs (reference) https://www.youtube.com/watch?v=Vk-huejruG0 (reference for local setup)
Note: This has not been tested
https://github.com/alicevision/meshroom/issues/357#issuecomment-535851216
- slurm on render farm
Slurm needs bash script to load jobs with specific comments such #SBATCH (https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html) .
slurmFarmSubmitter converts the Meshroom nodes to those scripts through python + jinja template https://github.com/sergiy-nazarenko/hafarm/blob/master/slurm_job.schema
slurmFarmSubmitter.py needs to be placed in meshroom/submitters/slurmFarmSubmitter.py
python subprocess module to submit a Slurm sbatch job:
https://github.com/alicevision/meshroom/blob/dev/slurm/meshroom/submitters/slurmSubmitter.py
Details on Slurm:
https://vsoch.github.io/lessons/slurm/ (good overview and tutorials)
https://aws.amazon.com/de/hpc/parallelcluster/
https://docs.aws.amazon.com/parallelcluster/latest/ug/schedulers.slurm.html
https://cloud.google.com/blog/products/gcp/easy-hpc-clusters-on-gcp-with-slurm
https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine
https://github.com/alicevision/meshroom/wiki/Meshroom-on-AWS
leverage AWS SQS (Simple Queue Service), S3 (Simple Storage Service) and boto https://github.com/boto/boto
to do: submitter
reference: https://www.youtube.com/watch?v=_Oqo383uviw