Build software better, together

yuewang-cuhk / awesome-vision-language-pretraining-papers

Recent Advances in Vision and Language PreTrained Models (VL-PTMs)

bert vision-and-language multimodal-deep-learning pretraining vl-ptms

Updated Dec 23, 2021

jrzaurin / pytorch-widedeep

A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch

python deep-learning text images tabular-data pytorch pytorch-cv multimodal-deep-learning pytorch-nlp pytorch-transformers model-hub pytorch-tabular-data

Updated Apr 7, 2022
Python

TheShadow29 / awesome-grounding

Star

awesome grounding: A curated list of research papers in visual grounding

natural-language-processing computer-vision paper awesome-list arxiv papers video-understanding captioning-images captioning-videos phrase-grounding language-grounding multimodal-deep-learning grounding visual-grounding embodied-agent video-grounding image-grounding paper-roadmap

Updated Sep 28, 2021

declare-lab / multimodal-deep-learning

Star

This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.

multimodal-interactions multimodal-learning multimodal-sentiment-analysis multimodal-deep-learning

Updated Nov 29, 2021
OpenEdge ABL

declare-lab / awesome-emotion-recognition-in-conversations

Star

A comprehensive reading list for Emotion Recognition in Conversations

natural-language-processing dialogue-systems emotion-recognition conversational-ai multimodal-interactions multimodal-deep-learning emotion-recognition-in-conversation

Updated Aug 13, 2021

theislab / scarches

Star

Reference mapping for single-cell genomics

deep-learning scrna-seq data-integration single-cell rna-seq-analysis single-cell-genomics batch-correction multimodal-deep-learning multiomics human-cell-atlas

Updated Apr 2, 2022
Jupyter Notebook

david-yoon / multimodal-speech-emotion

Star

TensorFlow implementation of "Multimodal Speech Emotion Recognition using Audio and Text," IEEE SLT-18

speech-emotion-recognition multimodal-deep-learning paralinguistics

Updated Oct 19, 2020
Jupyter Notebook

This repository contains the code for a video captioning system inspired by Sequence to Sequence -- Video to Text. This system takes as input a video and generates a caption in English describing the video.

tensorflow seq2seq sequence-to-sequence video-captioning s2vt multimodal-deep-learning

Updated Oct 12, 2019
Python

fnzhan / MISE

Star

Multimodal Image Synthesis and Editing: A Survey

transformers gan nerf multimodal-deep-learning diffusion-models

Updated Apr 7, 2022

soujanyaporia / MUStARD

Star

Multimodal Sarcasm Detection Dataset

sarcasm multimodal-interactions sarcasm-detection multimodal-deep-learning

Updated Apr 5, 2022
OpenEdge ABL

cap-ntu / Video-to-Retail-Platform

Star

An intelligent multimodal-learning based system for video, product and ads analysis. Based on the system, people can build a lot of downstream applications such as product recommendation, video retrieval, etc.

machine-learning deep-neural-networks deep-learning multimedia network-server multimodal-deep-learning ai-system

Updated Jan 10, 2021
Python

DirtyHarryLYL / DJ-RN

Star

As a part of HAKE project (HAKE-3D). Code for our CVPR2020 paper "Detailed 2D-3D Joint Representation for Human-Object Interaction".

3d-reconstruction multimodal-deep-learning human-object-interaction

Updated Apr 4, 2022
Python

haamoon / mmtm

Star

Implementation of CVPR 2020 paper "MMTM: Multimodal Transfer Module for CNN Fusion"

pytorch action-recognition gesture-recognition multimodal-learning speech-enhancement multimodal-deep-learning cnn-fusion

Updated Jun 16, 2020
Python

idearibosome / embracenet

Star

Robust multimodal integration method implemented in PyTorch and TensorFlow

deep-learning tensorflow pytorch multimodal multimodal-deep-learning

Updated Mar 5, 2021
Python

declare-lab / Multimodal-Infomax

Star

This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

multimodal-sentiment-analysis multimodal-deep-learning multimodal-fusion

Updated Nov 9, 2021
Python

referit3d / referit3d

Star

Code accompanying our ECCV-2020 paper on 3D Neural Listeners.

computer-vision deep-learning computer-graphics language-modeling rgbd multimodal-deep-learning

Updated Jun 29, 2021
C++

imatge-upc / wav2pix

Star

Speech-conditioned face generation using Generative Adversarial Networks (ICASSP 2019)

deep-learning generative-adversarial-network multimodal-deep-learning

Updated Feb 12, 2022
Python

shamanez / Self-Supervised-Embedding-Fusion-Transformer

Star

The code for our IEEE ACCESS (2020) paper Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature Fusion.

bert emotion-recognition multimodal-sentiment-analysis multimodal-deep-learning self-supervised-learning multimodal-emotion-recognition

Updated Sep 20, 2021
Python

akashe / Multimodal-action-recognition

Star

Code on selecting an action based on multimodal inputs. Here in this case inputs are voice and text.

multimodality multimodal-learning multimodal-deep-learning multimodal-data multimodal-fusion multimodal-action-recognition

Updated Jun 7, 2021
Python

husseinmozannar / multimodal-deep-learning-for-disaster-response

Star

Damage Identification in Social Media Posts using Multimodal Deep Learning: code and dataset

dataset multimodal-deep-learning decision-fusion feature-fusion disaster-detection

Updated Sep 7, 2021
Python

ilaria-manco / muscaps

Star

Source code for "MusCaps: Generating Captions for Music Audio" (IJCNN 2021)

music-information-retrieval mir multimodal-deep-learning audio-captioning

Updated Dec 27, 2021
Jupyter Notebook

lmb-freiburg / Multimodal-Future-Prediction

Star

The official repository for the CVPR 2019 paper "Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction"

trajectory-prediction multimodal-deep-learning

Updated Jan 7, 2021
Python

A2Zadeh / Social-IQ

Star

[CVPR 2019 Oral] Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence

machine-learning deep-learning multimodal-deep-learning language-and-vision intelligent-qa

Updated Oct 17, 2019
Python

JunweiLiang / FVTA_MemexQA

Star

Real-world photo sequence question answering system (MemexQA). CVPR'18 and TPAMI'19

visual-question-answering vision-and-language multimodal-deep-learning multimodal-datasets multimodal-representation memex-question-answering memexqa-dataset

Updated Jul 1, 2019
Python

sk-aravind / 3D-Bounding-Boxes-From-Monocular-Images

Star

A two stage multi-modal loss model along with rigid body transformations to regress 3D bounding boxes

point-cloud pytorch object-detection 3d kitti multimodal-deep-learning yolov3 monocular-3d-detection monocular-images 3dboundingbox deep-manta

Updated Apr 21, 2019
Python

rizavelioglu / hateful_memes-hate_detectron

Star

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge. https://arxiv.org/abs/2012.12975

challenge multimodal-deep-learning hateful-memes hateful-memes-challenge