Bleeding of Subjects/styles

[Trauter1]

Since the updates that introduced shuffling and things like: instance_token and class_token { don't really understand those. I would love to have more detailed description and usage of them explained], I can't eliminate bleeding from my subjects/styles, like in the versions before, whille using the same exact settings. Maybe it is my fault, if so, please share your experience, so we all can learn from it!

{
    "adam_beta1": 0.9,
    "adam_beta2": 0.999,
    "adam_epsilon": 1e-08,
    "adam_weight_decay": 0.01,
    "attention": "default",
    "center_crop": true,
    "concepts_path": "",
    "gradient_accumulation_steps": 1,
    "gradient_checkpointing": true,
    "half_model": true,
    "hflip": false,
    "learning_rate": 2e-06,
    "lr_scheduler": "cosine",
    "lr_warmup_steps": 0,
    "max_grad_norm": 1,
    "max_token_length": 225,
    "max_train_steps": 20000,
    "mixed_precision": "fp16",
    "model_dir": "/notebooks/stable-diffusion-webui/models/dreambooth/ckuyeqaz",
    "model_name": "ckuyeqaz",
    "not_cache_latents": true,
    "num_train_epochs": 770,
    "pad_tokens": true,
    "pretrained_model_name_or_path": "/notebooks/stable-diffusion-webui/models/dreambooth/ckuyeqaz/working",
    "pretrained_vae_name_or_path": null,
    "prior_loss_weight": 1,
    "resolution": 512,
    "revision": 6000,
    "sample_batch_size": 1,
    "save_embedding_every": 2000,
    "save_preview_every": 20000,
    "scale_lr": false,
    "scheduler": "ddim",
    "src": "g-anonmix.ckpt [e93c3bf7]",
    "train_batch_size": 1,
    "train_text_encoder": false,
    "use_8bit_adam": true,
    "use_concepts": false,
    "use_cpu": false,
    "use_ema": true,
    "v2": false,
    "concepts_list": [
        {
            "max_steps": -1,
            "instance_data_dir": "/notebooks/Training datas/kuro",
            "class_data_dir": "/notebooks/Training datas/mybrain",
            "file_prompt_contents": "Description",
            "instance_prompt": "ckuyeqaz, [filewords]",
            "class_prompt": "[filewords]",
            "save_sample_prompt": "ckuyeqaz, masterpiece, best quality, 1girl,
            "save_sample_template": "",
            "instance_token": "",
            "class_token": "",
            "num_class_images": 1000,
            "class_negative_prompt": "lowres, bad anatomy, (bad hands), text, error, (missing fingers, extra digit, fewer digits), (extra arms), cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, censored, monochrome",
            "class_guidance_scale": 7.5,
            "class_infer_steps": 30,
            "save_sample_negative_prompt": "lowres, bad anatomy, (bad hands), text, error, (missing fingers, extra digit, fewer digits), (extra arms), cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, censored, monochrome",
            "n_save_sample": 1,
            "sample_seed": -1,
            "save_guidance_scale": 7.5,
            "save_infer_steps": 30
        }
    ],
    "tokenizer_name": null
}

[d8ahazard]

Since the updates that introduced shuffling and things like: instance_token and class_token { don't really understand those. I would love to have more detailed description and usage of them explained], I can't eliminate bleeding from my subjects/styles, like in the versions before, whille using the same exact settings. Maybe it is my fault, if so, please share your experience, so we all can learn from it!

{
    "adam_beta1": 0.9,
    "adam_beta2": 0.999,
    "adam_epsilon": 1e-08,
    "adam_weight_decay": 0.01,
    "attention": "default",
    "center_crop": true,
    "concepts_path": "",
    "gradient_accumulation_steps": 1,
    "gradient_checkpointing": true,
    "half_model": true,
    "hflip": false,
    "learning_rate": 2e-06,
    "lr_scheduler": "cosine",
    "lr_warmup_steps": 0,
    "max_grad_norm": 1,
    "max_token_length": 225,
    "max_train_steps": 20000,
    "mixed_precision": "fp16",
    "model_dir": "/notebooks/stable-diffusion-webui/models/dreambooth/ckuyeqaz",
    "model_name": "ckuyeqaz",
    "not_cache_latents": true,
    "num_train_epochs": 770,
    "pad_tokens": true,
    "pretrained_model_name_or_path": "/notebooks/stable-diffusion-webui/models/dreambooth/ckuyeqaz/working",
    "pretrained_vae_name_or_path": null,
    "prior_loss_weight": 1,
    "resolution": 512,
    "revision": 6000,
    "sample_batch_size": 1,
    "save_embedding_every": 2000,
    "save_preview_every": 20000,
    "scale_lr": false,
    "scheduler": "ddim",
    "src": "g-anonmix.ckpt [e93c3bf7]",
    "train_batch_size": 1,
    "train_text_encoder": false,
    "use_8bit_adam": true,
    "use_concepts": false,
    "use_cpu": false,
    "use_ema": true,
    "v2": false,
    "concepts_list": [
        {
            "max_steps": -1,
            "instance_data_dir": "/notebooks/Training datas/kuro",
            "class_data_dir": "/notebooks/Training datas/mybrain",
            "file_prompt_contents": "Description",
            "instance_prompt": "ckuyeqaz, [filewords]",
            "class_prompt": "[filewords]",
            "save_sample_prompt": "ckuyeqaz, masterpiece, best quality, 1girl,
            "save_sample_template": "",
            "instance_token": "",
            "class_token": "",
            "num_class_images": 1000,
            "class_negative_prompt": "lowres, bad anatomy, (bad hands), text, error, (missing fingers, extra digit, fewer digits), (extra arms), cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, censored, monochrome",
            "class_guidance_scale": 7.5,
            "class_infer_steps": 30,
            "save_sample_negative_prompt": "lowres, bad anatomy, (bad hands), text, error, (missing fingers, extra digit, fewer digits), (extra arms), cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, censored, monochrome",
            "n_save_sample": 1,
            "sample_seed": -1,
            "save_guidance_scale": 7.5,
            "save_infer_steps": 30
        }
    ],
    "tokenizer_name": null
}

So, I think you have two problems:

One, you probably need to change the "file prompt contents" value to tell the app what is in the tags for your instance images.

Based on what you have above, I'm guessing your instance images are generically captioned, so, like "a photo of a girl wearing a white dress" or "a girl standing in the sun" or something like that? As in, 'ckuyeqaz' is not present in any of the captions?

If so, then file prompt contents should be "Class Token + Description". If "girl" is not in any of the captions, and it just tags like 1girl, dress, white, sunlight" or something, then "Description" is fine.

And for completeness, if your instance photos were captioned "ckuyeqaz girl wearing white dreass", file prompt contents would be Instance + Class + Description.

Now the program knows how to parse instance images.

For instance and class prompt, you more than likely want to use just [filewords], assuming you have your classifier in the captions already. If there is no class OR subject in the captions, then instance prompt would stay how you have it, and class prompt would be ckuyeqaz girl [filewords].

LAST, you're missing your class and instance token, which are very much needed when using [filewords]. Instance token is just the subject, so 'ckuyeqaz', and class token would be what your subject is, so assuming 'girl' or 'woman'. You don't need to put "a girl", etc.

Once all configured, the end results should be like so:

Existing caption: "a photo of a girl in a white dress"
Created instance/sample prompt: "a photo of ckuyeqaz girl in a white dress"
Created class prompt: "A photo of a girl in a white dress"

Or, if description only:
Existing caption: "white dress, photo"
Instance prompt: "ckuyeqaz [filewords]"
Class prompt: "A photo of a girl [filewords]"
Instance Token: "ckuyeqaz"
Class token: "girl"

Created instance/sample: "ckuyeqaz girl white dress, photo"
Created class prompt: a photo of a girl white dress, photo"

It's complicated AF, I know. But it kinda has to be in order to support the myriad ways people can tag photos or want to generate prompts...

[dali99]

is class token no longer a comma separated list of class tokens?

[d8ahazard]

Nope. It was just too complex for no reason. Easier to set it to "girl" and then search and replace "a girl", "the girl", etc. versus trying to come up with every possible linguistic combination.

[dali99]

that explains why I'm seeing all the bleeding...

[Trauter1]

Thank you for answering! I'm training style. My txt. contains danbooru tags for ex. 1girl, long hair, white hair, smile, red eyes, happy, standing, arms behind back, simple background
But also images without subjects like: no humans, landscape, grass, tree, clouds, sky, sun
I guess those would count as a description.
I don't like using "artstyle" as my class so I always used only [filewords] as a class prompt. Instance prompt was my token, [filewords]
I see that you used my token [filewords] Is the comma in my example bad?
If I only use [filewords] as a class, what would my class token be? I guess leaving it empty is not a option anymore? My txt. don't really have any particular tag that is always present when I train for style, and as I said before, I don't use artstyle or something similar.