All inpaint

.gitignore

@@ -5,7 +5,6 @@ __pycache__
 *.pth
 UCF-101/
 results/
-vae
 build/
 opensora.egg-info/
 wandb/
@@ -22,10 +21,69 @@ wandb/
 cache_dir/
 wandb/
 test*
-sample_videos/
+sample_video*/
 512*
 720*
 1024*
 debug*
 private*
-caption*
+caption*
+.deepspeed_env
+256*
+sample_image*/
+taming*
+*test*
+sft*
+flash*
+65x256*
+alpha_vae
+*node*
+cache/
+Open-Sora-Plan_models/
+sample_image*cfg*
+*tmp*
+*pymp*
+check.py
+bucket.py
+whileinf.py
+validation_dir/
+runs/
+samples/
+inpaint*/
+bs32x8x1*
+*tmp*
+*pymp*
+check.py
+bucket.py
+whileinf.py
+bs4x8x16_*
+*.zip
+*validation/
+bs1x8x32*
+bs16x8x1*
+bs8x8x2*
+bs8x8x1*
+bs8x8x8*
+bs1x8x16*
+checklora.py
+dim4todim8.py
+*vae8_any*320x320*
+samples/
+runs/
+*validation/
+training_log*txt
+filter_motion*
+json2*.py
+motionfun*
+res_dist*
+filter_json_aes_m*
+stage2*.json
+kernel_meta
+ge_check_op.json
+WFVAE_DISTILL_FORMAL
+read_video*
+bs32x8x2*
+filter_json_aes_m*
+json2json*
+makenpu_json*
+

docs/Train_And_Eval_CausalVideoVAE.md

@@ -32,14 +32,15 @@ Model training requires two key files: one is the `config.json` file, which conf
 
 ### Model Configuration File
 
-Taking the release version model configuration file `release.json` as an example:
+Taking the v1.1.0 version model configuration file as an example:
 
 ```json
 {
   "_class_name": "CausalVAEModel",
   "_diffusers_version": "0.27.2",
+  "_name_or_path": "../results/pretrained_488_tail",
   "attn_resolutions": [],
-  "decoder_attention": "AttnBlock3D",
+  "decoder_attention": "AttnBlock3DFix",
   "decoder_conv_in": "CausalConv3d",
   "decoder_conv_out": "CausalConv3d",
   "decoder_mid_resnet": "ResnetBlock3D",
@@ -58,32 +59,32 @@ Taking the release version model configuration file `release.json` as an example
   "decoder_temporal_upsample": [
     "",
     "",
-    "TimeUpsample2x",
-    "TimeUpsample2x"
+    "TimeUpsampleRes2x",
+    "TimeUpsampleRes2x"
   ],
   "double_z": true,
   "dropout": 0.0,
   "embed_dim": 4,
-  "encoder_attention": "AttnBlock3D",
-  "encoder_conv_in": "CausalConv3d",
+  "encoder_attention": "AttnBlock3DFix",
+  "encoder_conv_in": "Conv2d",
   "encoder_conv_out": "CausalConv3d",
   "encoder_mid_resnet": "ResnetBlock3D",
   "encoder_resnet_blocks": [
-    "ResnetBlock3D",
-    "ResnetBlock3D",
+    "ResnetBlock2D",
+    "ResnetBlock2D",
     "ResnetBlock3D",
     "ResnetBlock3D"
   ],
   "encoder_spatial_downsample": [
-    "SpatialDownsample2x",
-    "SpatialDownsample2x",
-    "SpatialDownsample2x",
+    "Downsample",
+    "Downsample",
+    "Downsample",
     ""
   ],
   "encoder_temporal_downsample": [
-    "TimeDownsample2x",
-    "TimeDownsample2x",
     "",
+    "TimeDownsampleRes2x",
+    "TimeDownsampleRes2x",
     ""
   ],
   "hidden_size": 128,
@@ -93,15 +94,17 @@ Taking the release version model configuration file `release.json` as an example
     4,
     4
   ],
+  "in_channels": 3,
   "loss_params": {
     "disc_start": 2001,
     "disc_weight": 0.5,
     "kl_weight": 1e-06,
     "logvar_init": 0.0
   },
-  "loss_type": "opensora.models.ae.videobase.losses.LPIPSWithDiscriminator",
+  "loss_type": "opensora.models.ae.videobase.losses.LPIPSWithDiscriminator3D",
   "lr": 1e-05,
   "num_res_blocks": 2,
+  "out_channels": 3,
   "q_conv": "CausalConv3d",
   "resolution": 256,
   "z_channels": 4

examples/prompt.txt

@@ -0,0 +1,8 @@
+yoji shinkawa painting of a stylish sniper demon
+a beautiful paint of cultists dancing surrounds a huge alpaca in desert at night, by zdzislaw beksinski, trending on artstation.
+male king arthur and his squirrel wife
+frontal portrait of ragged, worried twin women, by john singer sargent and j. c. leyendecker.
+a portrait of a female robot made from code, very intricate details, octane render, 8 k, trending on artstation 
+only memories remain, trending on artstation 
+dream swimming pool with nobody 
+a dog doing weights. epic oil painting. 

examples/prompt_list_0.txt

@@ -1,16 +1,20 @@
-A quiet beach at dawn, the waves gently lapping at the shore and the sky painted in pastel hues.
-A quiet beach at dawn, the waves softly lapping at the shore, pink and orange hues painting the sky, offering a moment of solitude and reflection.
-The majestic beauty of a waterfall cascading down a cliff into a serene lake.
-Sunset over the sea.
-a cat wearing sunglasses and working as a lifeguard at pool.
-Slow pan upward of blazing oak fire in an indoor fireplace.
-Yellow and black tropical fish dart through the sea.
-a serene winter scene in a forest. The forest is blanketed in a thick layer of snow, which has settled on the branches of the trees, creating a canopy of white. The trees, a mix of evergreens and deciduous, stand tall and silent, their forms partially obscured by the snow. The ground is a uniform white, with no visible tracks or signs of human activity. The sun is low in the sky, casting a warm glow that contrasts with the cool tones of the snow. The light filters through the trees, creating a soft, diffused illumination that highlights the texture of the snow and the contours of the trees. The overall style of the scene is naturalistic, with a focus on the tranquility and beauty of the winter landscape.
-a dynamic interaction between the ocean and a large rock. The rock, with its rough texture and jagged edges, is partially submerged in the water, suggesting it is a natural feature of the coastline. The water around the rock is in motion, with white foam and waves crashing against the rock, indicating the force of the ocean's movement. The background is a vast expanse of the ocean, with small ripples and waves, suggesting a moderate sea state. The overall style of the scene is a realistic depiction of a natural landscape, with a focus on the interplay between the rock and the water.
-A serene waterfall cascading down moss-covered rocks, its soothing sound creating a harmonious symphony with nature.
+A young man at his 20s is sitting on a piece of cloud in the sky, reading a book.
+An extreme close-up of an gray-haired man with a beard in his 60s, he is deep in thought pondering the history of the universe as he sits at a cafe in Paris, his eyes focus on people offscreen as they walk as he sits mostly motionless, he is dressed in a wool coat suit coat with a button-down shirt, he wears a brown beret and glasses and has a very professorial appearance, and the end he offers a subtle closed-mouth smile as if he found the answer to the mystery of life, the lighting is very cinematic with the golden light and the Parisian streets and city in the background, depth of field, cinematic 35mm film.
+A close-up of a woman’s face, illuminated by the soft light of dawn, her expression serene and content as she wakes up in a cozy bedroom.
+An intense close-up of a detective’s face, lit by a single desk lamp, his eyes scanning a wall covered in photos and notes, deep in thought.
+Audience members in a theater are captured in a series of medium shots, with a young man and woman in formal attire centrally positioned and illuminated by a spotlight effect.
 A soaring drone footage captures the majestic beauty of a coastal cliff, its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea. Seabirds can be seen taking flight around the cliff's precipices. As the drone slowly moves from different angles, the changing sunlight casts shifting shadows that highlight the rugged textures of the cliff and the surrounding calm sea. The water gently laps at the rock base and the greenery that clings to the top of the cliff, and the scene gives a sense of peaceful isolation at the fringes of the ocean. The video captures the essence of pristine natural beauty untouched by human structures.
-The video captures the majestic beauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The surrounding landscape is lush and green, with trees and foliage adding to the natural beauty of the scene. The camera angle provides a bird's eye view of the waterfall, allowing viewers to appreciate the full height and grandeur of the waterfall. The video is a stunning representation of nature's power and beauty.
-A vibrant scene of a snowy mountain landscape. The sky is filled with a multitude of colorful hot air balloons, each floating at different heights, creating a dynamic and lively atmosphere. The balloons are scattered across the sky, some closer to the viewer, others further away, adding depth to the scene.  Below, the mountainous terrain is blanketed in a thick layer of snow, with a few patches of bare earth visible here and there. The snow-covered mountains provide a stark contrast to the colorful balloons, enhancing the visual appeal of the scene.
-A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell, is the main focus of the video, swimming gracefully towards the right side of the frame. The coral reef, teeming with life, is visible in the background, providing a vibrant and colorful backdrop to the turtle's journey. Several small fish, darting around the turtle, add a sense of movement and dynamism to the scene.
-A snowy forest landscape with a dirt road running through it. The road is flanked by trees covered in snow, and the ground is also covered in snow. The sun is shining, creating a bright and serene atmosphere. The road appears to be empty, and there are no people or animals visible in the video. The style of the video is a natural landscape shot, with a focus on the beauty of the snowy forest and the peacefulness of the road.
-The dynamic movement of tall, wispy grasses swaying in the wind. The sky above is filled with clouds, creating a dramatic backdrop. The sunlight pierces through the clouds, casting a warm glow on the scene. The grasses are a mix of green and brown, indicating a change in seasons. The overall style of the video is naturalistic, capturing the beauty of the landscape in a realistic manner. The focus is on the grasses and their movement, with the sky serving as a secondary element. The video does not contain any human or animal elements.
+a realistic 3d rendering of a female character with curly blonde hair and blue eyes. she is wearing a black tank top and has a neutral expression while facing the camera directly. the background is a plain blue sky, and the scene is devoid of any other objects or text. the character is detailed, with realistic textures and lighting, suitable for a video game or high-quality animation. there is no movement or additional action in the video. the focus is entirely on the character's appearance and realistic rendering.
+A panda strumming a guitar under a bamboo grove, its paws gently plucking the strings as a group of mesmerized rabbits watch, the music blending with the rustle of bamboo leaves. HD.
+A close-up of a woman with a vintage hairstyle and bright red lipstick, gazing seductively into the camera, the background blurred to keep the focus solely on her.
+In the jungle, a hidden temple stands guarded by statues of lions, their eyes glowing with emerald light, protecting secrets untold for millennia. 8K.
+A close-up of an old man’s weathered face, with deep wrinkles and a thick white mustache, looking out to sea, the wind gently blowing through his hair.
+An intense close-up of a soldier’s face, covered in dirt and sweat, his eyes filled with determination as he surveys the battlefield.
+A river that flows uphill, defying gravity as it returns lost treasures from the sea to the mountain top, each item telling a story of a voyage gone by. HD.
+A close-up of a man’s face, lit only by the glow of his computer screen, his eyes wide and unblinking as he discovers something shocking online.
+On a deserted island, palm trees sway to summon a rainstorm, their leaves conducting the wind like maestros, orchestrating a symphony of thunder and lightning. High Resolution.
+An extreme close-up of a middle-aged man’s face, with a five o’clock shadow, staring pensively into the distance as rain softly taps against the window beside him, his thoughts deep and contemplative.
+A close-up of a man’s face, his expression one of deep concentration as he works on a complex task.
+Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach.The crashing blue waters create white-tipped waves,while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green
+shrubbery covers the cliffs edge. The steep drop from the road down to the beach is adramatic feat, with the cliff's edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.
+a close-up shot of a woman standing in a dimly lit room. she is wearing a traditional chinese outfit, which includes a red and gold dress with intricate designs and a matching headpiece. the woman has her hair styled in an updo, adorned with a gold accessory. her makeup is done in a way that accentuates her features, with red lipstick and dark eyeshadow. she is looking directly at the camera with a neutral expression. the room has a rustic feel, with wooden beams and a stone wall visible in the background. the lighting in the room is soft and warm, creating a contrast with the woman's vibrant attire. there are no texts or other objects in the video. the style of the video is a portrait, focusing on the woman and her attire.