Add files via upload

README.md

@@ -1,2 +1,37 @@
-# DI-MML
-The code for ACM MM 2024 paper, Detached and Interactive Multimodal Learning
+## Download data
+- CREMAD: `https://crisisnlp.qcri.org/data/crisismmd/CrisisMMD_v2.0.tar.gz`
+- AVE: `https://sites.google.com/view/audiovisualresearch`
+- UCF:
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.001`
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.002`
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.003`
+`cat ucf101_jpegs_256.zip* > ucf101_jpegs_256.zip`
+`unzip ucf101_jpegs_256.zip`
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.001`
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.002`
+`wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.003`
+`cat ucf101_tvl1_flow.zip* > ucf101_tvl1_flow.zip`
+`unzip ucf101_tvl1_flow.zip`
+- ModelNet: `http://supermoe.cs.umass.edu/shape_recog/shaded_images.tar.gz`
+
+## Preprocess data
+- `video_preprocessing.py` in `./data/CREMAD` for CREMAD.
+- `video_preprocessing.py` in `./data/AVE` for AVE.
+
+## train
+- CREMAD: 
+`python main_CFT.py --dataset CREMAD --num_frame 3 --temperature 50 --train`
+`python main_CFT_stage2.py --dataset CREMAD --num_frame 3 --load_path path_model1 --load_path_other path_model2 --temperature 1 --train`
+
+- AVE: 
+`python main_CFT.py --dataset AVE --num_frame 4 --temperature 10 --train`
+`python main_CFT_stage2.py --dataset AVE --num_frame 4 --load_path path_model1 --load_path_other path_model2 --temperature 3 --train`
+
+- UCF: 
+`python main_CFT.py --dataset UCF --num_frame 1 --temperature 10 --train`
+`python main_CFT_stage2.py --dataset UCF --num_frame 1 --load_path path_model1 --load_path_other path_model2 --temperature 1 --train`
+
+- ModelNet: 
+`python main_CFT.py --dataset ModelNet --num_frame 1 --temperature 50 --train`
+`python main_CFT_stage2.py --dataset ModelNet --num_frame 1 --load_path path_model1 --load_path_other path_model2 --temperature 3 --train`
+

data/AVE/mp4_to_wav.py

@@ -0,0 +1,37 @@
+import os
+import moviepy.editor as mp
+
+def extract_audio(videos_file_path):
+    my_clip = mp.VideoFileClip(videos_file_path)
+    return my_clip
+
+
+
+
+
+
+all_videos = r'D:\data\AVE_Dataset\Annotations.txt'
+all_audio_dir = r'D:\data\AVE_Dataset\Audios'
+if not os.path.exists(all_audio_dir):
+    os.makedirs(all_audio_dir)
+
+# train set processing
+with open(all_videos, 'r') as f:
+    files = f.readlines()
+
+for i, item in enumerate(files[1:]):
+    if i % 500 == 0:
+        print('*******************************************')
+        print('{}/{}'.format(i, len(files)))
+        print('*******************************************')
+    item = item.split('&')
+    mp4_filename = os.path.join(r'D:\data\AVE_Dataset\AVE', item[1] + '.mp4')
+    wav_filename = os.path.join(all_audio_dir, item[1]+'.wav')
+    if os.path.exists(wav_filename):
+        pass
+    else:
+        my_clip = extract_audio(mp4_filename)
+        my_clip.audio.write_audiofile(wav_filename)
+
+        #os.system('ffmpeg -i {} -acodec pcm_s16le -ar 16000 {}'.format(mp4_filename, wav_filename))
+