The specific settings of Lora finetuning

[Toneyaya]

Thank you for your open-source work!
Can you share the specific settings used for the LoRA Futuning results reported in the paper? I tried training with LoRA but the results differed from those reported in the paper (InstructBLIP).

[farewellthree]

Hi. When we conducted experiments using LoRA, we had not yet integrated DeepSpeed and transformer.trainer into the code. Therefore, we used stllm/train/train.py, with the following run configuration (model config and dataset config are the same):
run:
task: video_text_it

optimizer

lr_sched: "linear_warmup_cosine_lr"
init_lr: 1e-5
min_lr: 3e-6
warmup_lr: 4e-7

weight_decay: 0.05
max_epoch: 3
#iters_per_epoch: 200
batch_size_train: 1
batch_size_eval: 1
num_workers: 4
warmup_steps: 10000

seed: 42
output_dir: "path/to/output/dir"

amp: True
resume_ckpt_path: null

evaluate: False
train_splits: ["train"]

device: "cuda"
world_size: 1
dist_url: "env://"
distributed: True

[Toneyaya]

Thank you for your reply! What are the settings of lora_r and lora_alpha?

[farewellthree]

64 and 16

[Toneyaya]

The results of training using the mentioned configuration are currently even worse, with a score of 44 achieved on the first attempt, and performance on MVbench being less than 40. Here's the complete configuration file.
Additionally, several errors are encountered when attempting to train directly using train.py, such as the following ValueError: "Attempting to unscale FP16 gradients."
Could you kindly provide the code and configuration for LoRa fine-tuning? Your assistance would be greatly appreciated.

model:
  arch: st_llm_hf
  model_type: instructblip_vicuna0_btadapter
  use_grad_checkpoint: True
  max_txt_len: 256
  end_sym: "###"
  video_input: "all"
  llama_model: 'vicuna-7b-v1.1'
  ckpt: instruct_blip_vicuna7b_trimmed.pth'
  q_former_model: instruct_blip_vicuna7b_trimmed.pth'
  qformer_text_input: True
  freeze_LLM: True
  use_mask : True
  mvm_decode: True
  lora_r: 64
  lora_alpha: 16

datasets:
  caption_videochatgpt:
    num_frames: 16
    #video_reader_type: 'rawframe'
  classification_k710:
    num_frames: 16
  classification_ssv2:
    num_frames: 16
  reasoning_next_qa:
    num_frames: 16
  reasoning_clevrer_qa:
    num_frames: 16
  reasoning_clevrer_mc:
    num_frames: 16
  vqa_webvid_qa:
    num_frames: 16

run:
  task: video_text_it
  output_dir: "./stllm/output/instructblipbase_stllm_qa_lora_v1.1"
  lr_sched: "linear_warmup_cosine_lr"
  init_lr: 1e-5
  min_lr: 3e-6
  warmup_lr: 4e-7

  weight_decay: 0.05
  max_epoch: 3
  #iters_per_epoch: 200
  batch_size_train: 1
  batch_size_eval: 1
  num_workers: 4
  warmup_steps: 10000

  seed: 42
  amp: True
  resume_ckpt_path: null

  evaluate: False
  train_splits: ["train"]

  device: "cuda"
  world_size: 1
  dist_url: "env://"
  distributed: True

[farewellthree]

Sorry for the delayed response. I've been busy with the rebuttal lately. We will rerun the LoRA experiments and then provide you with the LoRA parameters.