Statistics for AdaptPrompt with Diffusion Set: A Unified Framework for Generalizable Deepfake Detection