Running Multi-modal spatial transformer models

[klken26]

Hello, I noticed that on this GitHub page, substantial information has been given to run the unimodal STNs. I tried running the multi-modal STN (I used http://efrosgans.eecs.berkeley.edu/gangealing/pretrained/horse.pt for the multi-modal horse model and I attempted to use cluster 0 of the horse clustering model) by following your 'using-pre-trained-clustering-models' (https://github.com/wpeebles/gangealing#using-pre-trained-clustering-models) section by adding the extra arguments. my command is as such:

!torchrun --nproc_per_node=1 applications/mixed_reality.py --ckpt horse --objects --label_path assets/objects/horse_cluster0/horse_cluster0_saddle.png --average_path assets/averages/horse_cluster0.png --num_heads 4 --sigma 0.3 --opacity 1 --cluster 0 --real_size 1024 --resolution 8192 --real_data_path data/final_horse

but I am getting the following error on Google Colab:

Also, I tried to run the multi-modal STN via the google colab link provided in this github page. These are the arguments i used:

args.real_size = int(video_size)
args.real_data_path = video_path
args.fps = fps
args.batch = batch_size
args.blend_alg = blend_alg
args.transform = ['similarity', 'flow']
args.flow_size = 128
args.stn_channel_multiplier = 0.5
args.num_heads = 4
args.distributed = False # Colab only uses 1 GPU
args.clustering = True
args.cluster = [0] #to prevent errors. try using 'args.cluster = 0' to replicate another error i got.
args.objects = True
args.no_flip_inference = not use_flipping
args.save_frames = memory_efficient_but_slower
args.overlay_congealed = False
args.ckpt = save_model
args.override = False
args.save_correspondences = False
args.out = 'visuals'
args.average_path = f'assets/averages/horse_cluster0.png'

and this is the error i got from using the multi-modal STN via the google colab link:

I'm not sure what I'm doing wrong - Could I have some assistance or guidance regarding this matter? Thank you in advance.