DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection

International Journal of Computer Vision (IJCV) 2024

Rui Shao^1,2 Tianxing Wu² Liqiang Nie¹ Ziwei Liu²

¹School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)
²S-Lab, Nanyang Technological University

[Paper]

🔥 Code has released. 👍

We propose a novel DeepFake-Adapter, which is a dual-level adapter composed of Globally-aware Bottleneck Adapters (GBA) and Locally-aware Spatial Adapters (LSA). DeepFake-Adapter can effectively adapt a pre-trained ViT by enabling high-level semantics from ViT to organically interact with global and local low-level forgeries from adapters. This contributes to more generalizable forgery representations for deepfake detection.

Updates

• [10/2024] Code has released.
• [10/2024] Accepted by International Journal of Computer Vision (IJCV) 2024.

Installation

Download

git clone https://github.com/rshaojimmy/DeepFake-Adapter.git
cd DeepFake-Adapter

Environment

We recommend using Anaconda to manage the python environment:

conda create -n deepfake python=3.8
conda activate deepfake
conda install -y -c pytorch pytorch=1.10.0 torchvision=0.11.1 cudatoolkit=11.3
pip install -r requirements.txt

Training

download the vit pretrain weights from https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_vit_base_patch16_224_in21k-e5005f0a.pth.

mkdir pre_trained
mv jx_vit_base_patch16_224_in21k-e5005f0a.pth pre_trained

! change the data path and result path in the train script

1. train model on FaceForensicspp RECCE c23/c40 all type

bash scripts/train_c23_all_type.sh
or
bash scripts/train_c40_all_type.sh

2. train model on FaceForensicspp RECCE c23 youtube FaceSwap or FaceForensicspp RECCE c40 youtube DeepFake, you can run:

bash scripts/train_c23_fs.sh
or
bash scripts/train_c40_df.sh

Feel free to modify the train script.

Inference

test the model trained on FaceForensicspp RECCE c23 youtube FaceSwap, you can run:

bash scripts/test_c23_fs.sh

Feel free to modify the test script.

Benchmark Results

Here we list the performance comparison of SOTA multi-modal and single-modal methods and our method. Please refer to our paper for more details.

Visualization Results

Grad-CAM visualizations in cross-dataset evaluation among DFDC, Celeb-DF and DF1.0 datasets.

t-SNE visualization of features encoded by (a) Xception and (b) DeepFake-Adapter in intra and crossmanipulation settings.

Citation

@article{shao2024deepfakeadapter,
  title={DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection},
  author={Shao, Rui and Wu, Tianxing and Nie, Liqiang and Liu, Ziwei},
  journal={International Journal of Computer Vision (IJCV)},
  year={2024},
}

Acknowledgements

The codebase is maintained by Rui Shao and Yuquan Xie.