network-compression

Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research. https://intellabs.github.io/distiller

Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" (ICCV 2019)

Using ideas from product quantization for state-of-the-art neural network compression.

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (AAAI 2019)

Neural Network Quantization & Low-Bit Fixed Point Training For Hardware-Friendly Algorithm Design

Group Fisher Pruning for Practical Network Compression(ICML2021)

Knowledge Distillation with Adversarial Samples Supporting Decision Boundary (AAAI 2019)

MUSCO: MUlti-Stage COmpression of neural networks

Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression. CVPR2020.

This is the official implementation of "DHP: Differentiable Meta Pruning via HyperNetworks".

Pytorch implemenation of "Learning Filter Basis for Convolutional Neural Network Compression" ICCV2019

Deep Neural Network Compression based on Student-Teacher Network

Code for "Variational Depth Search in ResNets" (https://arxiv.org/abs/2002.02797)

MUSCO: Multi-Stage COmpression of neural networks

pruning

Homework for Machine Learning (2019, Spring) at NTU

2020 INTERSPEECH, "Sparse Mixture of Local Experts for Efficient Speech Enhancement".

Overparameterization and overfitting are common concerns when designing and training deep neural networks. Network pruning is an effective strategy used to reduce or limit the network complexity, but often suffers from time and computational intensive procedures to identify the most important connections and best performing hyperparameters. We suggest a pruning strategy which is completely integrated in the training process and which requires only marginal extra computational cost. The method relies on unstructured weight pruning which is re-interpreted in a multiobjective learning approach. A batchwise Pruning strategy is selected to be compared using different optimization methods, of which one is a multiobjective optimization algorithm. As it takes over the choice of the weighting of the objective functions, it has a great advantage in terms of reducing the time consuming hyperparameter search each neural network training suffers from. Without any a priori training, post training, or parameter fine tuning we achieve highly reductions of the dense layers of two commonly used convolution neural networks (CNNs) resulting in only a marginal loss of performance. Our results empirically demonstrate that dense layers are overparameterized as with reducing up to 98 % of its edges they provide almost the same results. We contradict the theory that retraining after pruning neural networks is of great importance and opens new insights into the usage of multiobjective optimization techniques in machine learning algorithms in a Keras framework. The Stochastic Multi Gradient Descent Algorithm implementation in Python3 is for usage with Keras and adopted from paper of S. Liu and L. N. Vicente: "The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning". It is combined with weight pruning strategies to reduce network complexity and inference time.

This repository consists of application of Deep Learning Models like DNN, CNN (1D and 2D), RNN (LSTM and GRU) and Variational Autoencoders written from scratch in tensorflow.

AIMET GitHub pages documentation

Deep Compression on AlexNet

collection of works aiming at reducing model sizes or the ASIC/FPGA accelerator for machine learning

李宏毅教授 ML 2020 機器學習課程筆記 & 實作