Matrix Factorization, Algorithms, Applications, and Avaliable packages
Matrix Decompositions has
a long history and generally centers around a set of known factorizations such as LU, QR, SVD and eigendecompositions. More recent
factorizations have seen the light of the day with work started with the advent of NMF, k-means and related algorithm [1].
However, with the advent of new methods based on random projections and convex optimization that started in part in the compressive
sensing literature, we are seeing another surge of very diverse algorithms dedicated to many different kinds of matrix factorizations with new constraints based on rank and/or positivity and/or sparsity,… As a result of this large increase in interest,
I have decided to keep a list of them here following the success of the big
picture in compressive sensing.
The sources for this list include the following most excellent sites: Stephen
Becker’s page, Raghunandan H. Keshavan‘ s page, Nuclear
Norm and Matrix Recovery through SDP by Christoph Helmberg, Arvind
Ganesh’s Low-Rank Matrix Recovery and Completion via Convex
Optimization who provide more in-depth additional information. Additional codes were featured also on Nuit
Blanche. The following people provided additional inputs: Olivier Grisel, Matthieu
Puigt.
Most of the algorithms listed below generally rely on using the nuclear norm as a proxy to the rank functional. It
may not be optimal. Currently, CVX ( Michael
Grant and Stephen Boyd) consistently allows one to explore other
proxies for the rank functional such as thelog-det as
found by Maryam Fazell, Haitham
Hindi, Stephen Boyd. ** is used to show that the algorithm uses
another heuristic than the nuclear norm.
In terms of notations, A refers to a matrix, L refers to a low rank matrix, S a sparse one and N to a noisy one. This page lists the different codes that implement the following matrix factorizations: Matrix Completion, Robust
PCA , Noisy Robust PCA, Sparse PCA, NMF, Dictionary Learning, MMV, Randomized Algorithms and other factorizations. Some of these toolboxes can sometimes implement several of these decompositions and are listed accordingly. Before I list algorithm here, I generally
feature them on Nuit Blanche under the MF tag: http://nuit-blanche.blogspot.com/search/label/MF or. you
can also subscribe to the Nuit Blanche feed,
Matrix Completion, A = H.*L with H a known mask, L unknown solve
for L lowest rank possible
The idea of this approach is to complete the unknown coefficients of a matrix based on the fact that the matrix is low rank:
- OptSpace: Matrix
Completion from a Few Entries by Raghunandan H. Keshavan, Andrea
Montanari, and Sewoong Oh - LMaFit: Low-Rank Matrix Fitting
- ** Penalty
Decomposition Methods for Rank Minimization by Zhaosong Lu and Yong
Zhang.The attendant MATLAB code is here. - Jellyfish: Parallel
Stochastic Gradient Algorithms for Large-Scale Matrix Completion, B. Recht, C. Re, Apr 2011 - GROUSE:
Online Identification and Tracking of Subspaces from Highly Incomplete Information, L. Balzano, R. Nowak, B. Recht, 2010 - SVP: Guaranteed
Rank Minimization via Singular Value Projection, R. Meka, P. Jain, I.S.Dhillon, 2009 - SET:
SET: an algorithm for consistent matrix completion, W. Dai, O. Milenkovic, 2009 - NNLS: An
accelerated proximal gradient algorithm for nuclear norm regularized least squares problems, K. Toh, S. Yun, 2009 - FPCA: Fixed point
and Bregman iterative methods for matrix rank minimization, S. Ma, D. Goldfard, L. Chen, 2009 - SVT: A singular value thresholding
algorithm for matrix completion, J-F Cai, E.J. Candes, Z. Shen, 2008
Noisy Robust PCA, A = L + S + N with L, S, N unknown, solve
for L low rank, S sparse, N noise
- GoDec :
Randomized Low-rank and Sparse Matrix Decomposition in Noisy Case - ReProCS: The Recursive
Projected Compressive Sensing code (example)
Robust PCA : A = L + S with L, S, N unknown, solve for L low
rank, S sparse
- Robust PCA :
Two Codes that go with the paper “Two
Proposals for Robust PCA Using Semidefinite Programming.” by MichaleI
Mccoy andJoel Tropp - SPAMS (SPArse
Modeling Software) - ADMM: Alternating
Direction Method of Multipliers ‘‘Fast Automatic
Background Extraction via Robust PCA’ by Ivan Papusha. The poster
is here. The matlab implementation is here. - PCP: Generalized
Principal Component Pursuit - Augmented Lagrange Multiplier (ALM) Method [exact ALM - MATLAB zip]
[inexact ALM - MATLABzip], Reference
- The Augmented Lagrange Multiplier Method
for Exact Recovery of Corrupted Low-Rank Matrices, Z. Lin, M. Chen, L. Wu, and Y. Ma (UIUC Technical Report UILU-ENG-09-2215, November 2009) - Accelerated Proximal Gradient , Reference - Fast
Convex Optimization Algorithms for Exact Recovery of a Corrupted Low-Rank Matrix, Z. Lin, A. Ganesh, J. Wright, L. Wu, M. Chen, and Y. Ma (UIUC Technical Report UILU-ENG-09-2214, August 2009)[full SVD version - MATLAB zip]
[partial SVD version - MATLAB zip] - Dual Method [MATLAB zip],
Reference - Fast Convex Optimization
Algorithms for Exact Recovery of a Corrupted Low-Rank Matrix, Z. Lin, A. Ganesh, J. Wright, L. Wu, M. Chen, and Y. Ma (UIUC Technical Report UILU-ENG-09-2214, August 2009). - Singular Value Thresholding [MATLAB zip].
Reference - A Singular Value Thresholding Algorithm
for Matrix Completion, J. -F. Cai, E. J. Candès, and Z. Shen (2008). - Alternating Direction Method [MATLAB zip]
, Reference - Sparse and Low-Rank Matrix
Decomposition via Alternating Direction Methods, X. Yuan, and J. Yang (2009). - LMaFit: Low-Rank Matrix Fitting
- Bayesian robust
PCA - Compressive-Projection
PCA (CPPCA)
Sparse PCA: A = DX with unknown D and X, solve for sparse
D
Sparse PCA on wikipedia
- R. Jenatton, G. Obozinski, F. Bach. Structured Sparse Principal Component Analysis. International Conference on Artificial Intelligence and Statistics (AISTATS). [pdf]
[code] - SPAMs
- DSPCA: Sparse
PCA using SDP . Code ishere. - PathPCA: A fast greedy algorithm for Sparse PCA. The code is here.
Dictionary Learning: A = DX with unknown D and X, solve for sparse
X
Some implementation of dictionary learning implement the NMF
- Online
Learning for Matrix Factorization and Sparse Coding by Julien Mairal, Francis
Bach, Jean Ponce,Guillermo
Sapiro [The code is released as SPArse Modeling Softwareor SPAMS] - Dictionary
Learning Algorithms for Sparse Representation (Matlab implementation of FOCUSS/FOCUSS-CNDL
is here) - Multiscale
sparse image representation with learned dictionaries [Matlab implementation of the K-SVD
algorithm is here, a newer implementation by Ron Rubinstein is here ] - Efficient
sparse coding algorithms [ Matlab code
is here ] url=http%3A%2F%2Fwww2.imm.dtu.dk%2Fpubdb%2Fviews%2Fedoc_download.php%2F4659%2Fpdf%2Fimm4659.pdf" style="color:rgb(41,112,166); text-decoration:none; margin:0px; padding:0px; word-wrap:break-word; border:none">Shift
. Matlab implemention is here
Invariant Sparse Coding of Image and Music Data- Shift-invariant
dictionary learning for sparse representations: extending K-SVD. - Thresholded
Smoothed-L0 (SL0) Dictionary Learning for Sparse Representations by Hadi Zayyani, Massoud
Babaie-Zadeh and Remi Gribonval. - Non-negative
Sparse Modeling of Textures (NMF) [Matlab implementation of NMF
(Non-negative Matrix Factorization) and NTF (Non-negative Tensor), a faster implementation of NMF can be found here,
here is a more recent Non-Negative Tensor Factorizations package]
NMF: A = DX with unknown D and X, solve for elements of D,X
> 0
Non-negative
Matrix Factorization (NMF) on wikipedia
- HALS: Accelerated
Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization by Nicolas
Gillis, François Glineur. - SPAMS (SPArse
Modeling Software) by Julien Mairal, Francis
Bach, Jean Ponce,Guillermo
Sapiro - NMF: C.-J. Lin. Projected
gradient methods for non-negative matrix factorization.issn=08997667" style="color:rgb(41,112,166); text-decoration:none; margin:0px; padding:0px; word-wrap:break-word; border:none">Neural
, 19(2007), 2756-2779.
Computation - Non-Negative Matrix Factorization: This
page contains an optimized C implementation of the Non-Negative Matrix Factorization (NMF) algorithm, described in [Lee
& Seung 2001]. We implement the update rules that minimize a weighted SSD error metric. A detailed description of weighted NMF can be found in[Peers
et al. 2006]. - NTFLAB for
Signal Processing, Toolboxes for NMF (Non-negative Matrix Factorization) and NTF (Non-negative Tensor Factorization) for BSS (Blind Source Separation) - Non-negative
Sparse Modeling of Textures (NMF) [Matlab implementation of NMF
(Non-negative Matrix Factorization) and NTF (Non-negative Tensor), a faster implementation of NMF can be found here,
here is a more recent Non-Negative Tensor Factorizations package]
Multiple Measurement Vector (MMV) Y = A X with unknown X and rows
of X are sparse.
- T-MSBL/T-SBL by Zhilin
Zhang - Compressive
MUSIC with optimized partial support for joint sparse recovery by Jong
Min Kim, Ok Kyun Lee, Jong
Chul Ye [no code] - The
REMBO Algorithm Accelerated Recovery of Jointly Sparse Vectorsby Moshe Mishali and Yonina C. Eldar [ no code]
Blind Source Separation (BSS) Y = A X with unknown A and X and
statistical independence between columns of X or subspaces of columns of X
Include Independent Component Analysis (ICA), Independent Subspace Analysis (ISA), and Sparse Component Analysis (SCA). There are many available codes for ICA and some for SCA. Here is a non-exhaustive list of some
famous ones (which are not limited to linear instantaneous mixtures). TBC
ICA:
- ICALab:
url=http%3A%2F%2Fwww%2Ebsp%2Ebrain%2Eriken%2Ejp%2FICALAB%2F&urlhash=8N_s&_t=tracking_disc" style="color:rgb(41,112,166); text-decoration:none; margin:0px; padding:0px; word-wrap:break-word; border:none">http://www.bsp.brain.riken.jp/ICALAB/
- BLISS softwares:
url=http%3A%2F%2Fwww%2Elis%2Einpg%2Efr%2Fpages_perso%2Fbliss%2Fdeliverables%2Fd20%2Ehtml&urlhash=5cMy&_t=tracking_disc" style="color:rgb(41,112,166); text-decoration:none; margin:0px; padding:0px; word-wrap:break-word; border:none">http://www.lis.inpg.fr/pages_perso/bliss/deliverables/d20.html
- MISEP: http://www.lx.it.pt/~lbalmeida/ica/mitoolbox.html
- Parra and Spence’s frequency-domain convolutive ICA:http://people.kyb.tuebingen.mpg.de/harmeling/code/convbss-0.1.tar
- C-FICA: http://www.ast.obs-mip.fr/c-fica
SCA:
- DUET:
url=http%3A%2F%2Fsparse%2Eucd%2Eie%2Fpublications%2Frickard07duet%2Epdf&urlhash=fZ9d&_t=tracking_disc" style="color:rgb(41,112,166); text-decoration:none; margin:0px; padding:0px; word-wrap:break-word; border:none">http://sparse.ucd.ie/publications/rickard07duet.pdf
(the
matlab code is given at the end of this pdf document) - LI-TIFROM: http://www.ast.obs-mip.fr/li-tifrom
Randomized Algorithms
These algorithms uses generally random projections to shrink very large problems into smaller ones that can be amenable to traditional matrix factorization methods.
Resource
Randomized algorithms for matrices and data by Michael W. Mahoney
Randomized Algorithms for Low-Rank Matrix
Decomposition
- Randomized PCA
- Randomized Least Squares: Blendenpik( http://pdos.csail.mit.edu/~petar/papers/blendenpik-v1.pdf )
Other factorization
D(T(.)) = L + E with unknown L, E and unknown transformation T and solve
for transformation T, Low Rank L and Noise E
- RASL:
Robust Batch Alignment of Images by Sparse and Low-Rank Decomposition - TILT:
Transform Invariant Low-rank Textures
Frameworks featuring advanced Matrix factorizations
For the time being, few have integrated the most recent factorizations.
- Scikit
Learn (Python) - Matlab
Toolbox for Dimensionality Reduction (Probabilistic PCA, Factor Analysis (FA)…) - Orange (Python)
- pcaMethods—a bioconductor package
providing PCA methods for incomplete data. R Language
GraphLab / Hadoop
- Danny Bickson keeps
a blog on GraphLab.
Books
Example of use
- CS:
Low Rank Compressive Spectral Imaging and a multishot CASSI - CS:
Heuristics for Rank Proxy and how it changes everything…. - Tennis
Players are Sparse !
Sources
Arvind Ganesh’s Low-Rank
Matrix Recovery and Completion via Convex Optimization
- Raghunandan H. Keshavan‘
s list - Stephen
Becker’s list - Nuclear
Norm and Matrix Recovery through SDP by Christoph Helmberg - Nuit Blanche
Relevant links
Reference:
A
Unified View of Matrix Factorization Models by Ajit P. Singh and Geoffrey J. Gordon
Matrix Factorization, Algorithms, Applications, and Avaliable packages的更多相关文章
- Matrix Factorization SVD 矩阵分解
Today we have learned the Matrix Factorization, and I want to record my study notes. Some kownledge ...
- 关于NMF(Non-negative Matrix Factorization )
著名的科学杂志<Nature>于1999年刊登了两位科学家D.D.Lee和H.S.Seung对数学中非负矩阵研究的突出成果.该文提出了一种新的矩阵分解思想――非负矩阵分解(Non-nega ...
- 机器学习技法:15 Matrix Factorization
Roadmap Linear Network Hypothesis Basic Matrix Factorization Stochastic Gradient Descent Summary of ...
- 《Non-Negative Matrix Factorization for Polyphonic Music Transcription》译文
NMF(非负矩阵分解),由于其分解出的矩阵是非负的,在一些实际问题中具有非常好的解释,因此用途很广.在此,我给大家介绍一下NMF在多声部音乐中的应用.要翻译的论文是利用NMF转录多声部音乐的开山之作, ...
- 机器学习技法笔记:15 Matrix Factorization
Roadmap Linear Network Hypothesis Basic Matrix Factorization Stochastic Gradient Descent Summary of ...
- Non-negative Matrix Factorization 非负矩阵分解
著名的科学杂志<Nature>于1999年刊登了两位科学家D.D.Lee和H.S.Seung对数学中非负矩阵研究的突出成果.该文提出了一种新的矩阵分解思想――非负矩阵分解(Non-nega ...
- 【RS】Sparse Probabilistic Matrix Factorization by Laplace Distribution for Collaborative Filtering - 基于拉普拉斯分布的稀疏概率矩阵分解协同过滤
[论文标题]Sparse Probabilistic Matrix Factorization by Laplace Distribution for Collaborative Filtering ...
- 【RS】List-wise learning to rank with matrix factorization for collaborative filtering - 结合列表启发排序和矩阵分解的协同过滤
[论文标题]List-wise learning to rank with matrix factorization for collaborative filtering (RecSys '10 ...
- 【RS】Matrix Factorization Techniques for Recommender Systems - 推荐系统的矩阵分解技术
[论文标题]Matrix Factorization Techniques for Recommender Systems(2009,Published by the IEEE Computer So ...
随机推荐
- sql表连接的几种方式
这里有两张表TableA和TableB,分别是姓名表和年龄表,用于我们例子的测试数据 TableA id name 1 t1 2 t2 4 t4 TableB id age 1 18 2 20 3 1 ...
- 深刻理解void,void*和sizeof关键字
void的字面值是“无类型”,void*则是"无类型指针".void*可以指向任何类型的数据.void几乎只有"注释"和限制程序的作用,因为从来没有人会定义一个 ...
- Set 与 Multiset
Set 与 Multiset 会根据待定的排序准则,自动将元素排序,两者不同之处在于前者不允许元素重复,后者允许,下面介绍一下set中的函数: 一.set 中的 begin.end.rbegin.re ...
- 关于CDH5.2+ 添加hive自定义UDAF函数的方法
- IOS 学习笔记(3) 视图UITabbarController
1.UITabbarViewController标签试图控制器.由于标签页本就起着分类的作用,所以往往呈现的视图内容之间,可以是毫不相关的功能. UITabbarViewController仍然继承自 ...
- NuGet安装软件包故障解决
今晚在家工作,使用NuGet安装 Install-Package Microsoft.AspNet.Web.Optimization 时,一直提示无法连接到nuget.org. 搜索后,可使用如下方法 ...
- APUE学习之---------------进程
离职了,交接期也有足够的时间了,可以在好好的再看一下APUE,想想上次详细的看还是在两年之前,虽然中间也偶尔会翻出来看看,但是由于工作上交集相对比较少一直没有去细读一下.现在正好是一段空挡期可以好好看 ...
- Eclipse创建新项目时无法输入项目名的解决方法
放假耍了那么久,也是该收心忙活了. 今天打开Eclipse新建项目时,发生了一个很奇怪的情况,就是在下面这个位置的输入框无法输入. 经过百度之后,发现解决方案是(原地址点我) Eclipse图标右键 ...
- Python生成随机数的方法
这篇文章主要介绍了Python生成随机数的方法,有需要的朋友可以参考一下 如果你对在Python生成随机数与random模块中最常用的几个函数的关系与不懂之处,下面的文章就是对Python生成随机数与 ...
- Objective-C浅拷贝和深拷贝
浅拷贝就是对内存地址的复制,让目标对象指针和源对象指向同一片内存空间 如: char* str = (char*)malloc(100);char* str2 = str; 浅拷贝只是对对象的简单拷贝 ...