how to train a lora model of an avatar without body via dreambooth

[joey0922]

I wanna train a lora model of an avatar which doesn't have body via dreambooth, but the model often try to generate a body for the avatar which is only a head like a box. How to prevent it? Thanks.

[asomoza]

Hi, it's not clear to me, do you want to train something like this one but with just the faces?

You just need to train it with a lot of examples of what you want, since is more like a concept than a subject, you'll need a lot more images.

[joey0922]

Thanks, I will take a shot if I can. I still have a confusion that why the concept or subject I want to learn is broken or even missing in my generated images. I think if I got this situation, the model I trained is under-fitting? I found it is possible to generate images with concept broken no matter how well the model I trained. What can I do to decrease the ratio of broken concept in generated images?

[asomoza]

it can be by a number of reasons, without actually looking at your code and dataset it's really hard to tell.

I can tell you that what you want to train is not for beginners, most people that have success doing something like this have trained a lot of loras before and learned the in and outs of lora training.

You probably should start with something easier and gradually go for something more complicated, as I see it, this is the order of difficulty for training loras:

1. Human face
2. Human face and body or something like a dog, etc.
3. Similar style to what the model knows (like variations of anime style)
4. Completely new style that the model doesn't know
5. Completely new concept and style that the model doesn't know

You can also use a tested dataset and learn how to caption and how it does affect the training, for example an easy to start one is tuxemon, and if you want a style you can use the vintage ads one.

[joey0922]

@asomoza what i am trying to train is kind of a thing as below:

It is kind of an avatar which is a cube with single color and has two white rectangle eyes. And I have thousands of pictures for training. I want to train a model which can generate various backgrounds for it and make it possible to act like an avatar or human, such as wearing a hat or a pair of sunglasses or something else. I have two questions:
1. Is that possible? I tried so many times and it didn't work out. The figure I trained is often broken easily while the model can't understand the prompt well.
2. I use dreambooth to train a lora model based on sdxl in class plus identifier way. I am wondering what class word is good for this thing? avatar?emoji?any other good idea?
Many thanks.

[asomoza]

dreambooth is mostly for just one character/basic style with a low count of images (dataset), is not ideal to use it for something like you're trying to do. You probably need to finetune a normal lora or the whole model if you want better results.

The reason you're struggling with the class is the same, you are doing something more complex than just a class. If you already have thousands of images, just label them correctly in a normal way (describing each image) and finetune the model to see if you get better results. In this case you'll need to lower the LR and also do more epochs.

[joey0922]

I got it. But workload to label thousands of images is much heavy. How many labeled images is good enough for training at least?

[crapthings]

i've tested advanced dreambooth lora, have the same issue.
i use 1 ~ 10 images to test, but no success, there's nothing like my dataset, after training, what is wrong?

i have tried pivot tuning and training te with below.

rm -rf ./lora
rm -rf ./class_image_woman/.ipynb_checkpoints
rm -rf ./woman/.ipynb_checkpoints
rm -rf ./face/.ipynb_checkpoints

export MODEL_NAME="stable-diffusion-v1-5/stable-diffusion-v1-5"
export VAE_PATH="stabilityai/sd-vae-ft-mse"

export OUTPUT_DIR="./lora"
export DATASET_NAME="./face"

accelerate launch train_dreambooth_lora_sd15_advanced.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_PATH \
  --mixed_precision="bf16" \
  --instance_data_dir=$DATASET_NAME \
  --instance_prompt="ohwz" \
  --train_batch_size=1 \
  --resolution=512 \
  --rank=12 \
  --max_train_steps=100 \
  --checkpointing_steps=5 \
  --seed=42 \
  --snr_gamma=5.0 \
  --output_dir=$OUTPUT_DIR \
  --optimizer="prodigy" \
  --train_text_encoder \
  --learning_rate=1.0 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=5

import torch
from diffusers import StableDiffusionPAGPipeline, DPMSolverMultistepScheduler, UNet2DConditionModel, AutoencoderKL
from diffusers.utils import make_image_grid
from asdff import AdPipeline

vae = AutoencoderKL.from_pretrained('stabilityai/sd-vae-ft-mse', torch_dtype = torch.bfloat16,).to('cuda')

pipeline = StableDiffusionPAGPipeline.from_pretrained(
	'stable-diffusion-v1-5/stable-diffusion-v1-5',
    torch_dtype = torch.bfloat16,
    safety_checker = None,
    pag_applied_layers = 'mid',
    vae = vae
)

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, algorithm_type = 'sde-dpmsolver++', use_karras_sigmas = True)

pipeline.to('cuda')

# pipe2 = AdPipeline.from_pipe(pipeline, torch_dtype = torch.float16)

# pipe2.safety_checker = None

# pipe2.to('cuda')

pipeline.load_lora_weights('./lora', weight_name = 'pytorch_lora_weights.safetensors', adapter_name = 'face')

pipeline.set_adapters(adapter_names = ['face'], adapter_weights = [1.0])

# pipeline.load_textual_inversion('./lora/lora_emb.safetensors', token = 'TOK')

print('loaded')