Can I read Unlocking the Capabilities of Vision-Language Models for Generalizable and Explainable Deepfake Detection on EtoBox?

Unlocking the Capabilities of Vision-Language Models for Generalizable and Explainable Deepfake Detection by Yu, Peipeng; Fei, Jianwei; Gao, Hui; Feng, Xuan; Xia, Zhihua; Chang, Chip Hong is a scholarly article available to read on EtoBox.

What is Unlocking the Capabilities of Vision-Language Models for Generalizable and Explainable Deepfake Detection about?

Current vision-language models (VLMs) have demonstrated remarkable capabilities in understanding multimodal data, but their potential remains underexplored for deepfake detection due to the misaligned of their knowledge and forensics patterns. To this end, we present a novel paradigm that unlocks VLMs' potential capabilities through three components: (1) A knowledge-guided forgery adaptation module that aligns VLM's semantic space with forensic features through contrastive learning with external manipulation knowledge; (2) A multi-modal prompt tuning framework that jointly optimizes visual-textual embeddings for both localization and explainability; (3) An iterative refinement strategy enabling multi-turn dialog for evidence-based reasoning. Our framework includes a VLM-based Knowledge-guided Forgery Detector (KFD), a VLM image encoder, and a Large Language Model (LLM). The VLM image encoder extracts visual prompt embeddings from images, while the LLM receives visual and question prompt embeddings for inference. The KFD is used to calculate correlations between image features and pristine/deepfake class embeddings, enabling forgery classification and localization. The outputs from

Author

Yu, Peipeng; Fei, Jianwei; Gao, Hui; Feng, Xuan; Xia, Zhihua; Chang, Chip Hong

Published

2025

Language

EN