vision-and-language

Oscar and VinVL

X-modaler is a versatile and high-performance codebase for cross-modal analytics(e.g., image captioning, video captioning, vision-language pre-training, visual question answering, visual commonsense reasoning, and cross-modal retrieval).

Recent Advances in Vision and Language PreTrained Models (VL-PTMs)

My Reading Lists of Deep Learning and Natural Language Processing

Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".

Code for the ICML 2021 (long talk) paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"

[CVPR 2021 Best Student Paper Honorable Mention, Oral] Official PyTorch code for ClipBERT, an efficient framework for end-to-end learning on image-text and video-text tasks.

Code for ALBEF: a new vision-language pre-training method

Creating a software for automatic monitoring in online proctoring

AI Research Platform for Reinforcement Learning from Real Panoramic Images.

Implementation of 'X-Linear Attention Networks for Image Captioning' [CVPR 2020]

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

Conceptual 12M is a dataset containing (image-URL, caption) pairs collected for vision-and-language pre-training.

Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"

Awesome Resources for Advanced Computer Vision Topics

A curated list of research papers in Vision-Language Navigation (VLN)

Code for the ACL paper "No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling"

A curated list of awesome vision and language resources (still under construction... stay tuned!)

PyTorch code for CVPR 2019 paper: The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation

PyTorch code for ICLR 2019 paper: Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

X-VLM: Multi-Grained Vision Language Pre-Training

Research Code for NeurIPS 2020 Spotlight paper "Large-Scale Adversarial Training for Vision-and-Language Representation Learning": UNITER adversarial training part

code for TCL: Vision-Language Pre-Training with Triple Contrastive Learning, CVPR 2022

Code of the CVPR 2021 Oral paper: A Recurrent Vision-and-Language BERT for Navigation

A PyTorch implementation of VIOLET

🥶Vilio: State-of-the-art VL models in PyTorch & PaddlePaddle

[CVPR20] Video Object Grounding using Semantic Roles in Language Description (https://arxiv.org/abs/2003.10606)

This repository contains my solutions to the assignments for Stanford's CS231n "Convolutional Neural Networks for Visual Recognition" (Spring 2020).

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT