The Institute of Statistical Mathematics

$Id: index.html,v 1.1 2013/06/28 13:02:38 daichi Exp $

rsm is a modified Python implementation of Replicated Softmax Model
of Salakhutdinov and Hinton (2009)
[PDF],

a simple single-layer "Deep Net" for documents.

This code is a modification to a Python implementation
by Joerg Landthaler,
http://www.fylance.de/rsm/,

in several aspects:

- Includes an easy-to-use command line interface (rsm.py) where many hyperparameters can be specified,
- Includes an n-step Contrastive Divergence (option -n) as opposed to the original code,
- Visiting data randomly through minibatches during optimization,
- Outputs perplexities rather than L2-norm of multinomial reconstruction,
- And some fixes.

% rsm.py -H 10 -N 20 -b 10 train model loading data.. done. number of documents = 100 number of lexicon = 1324 number of hidden variables = 10 number of learning epochs = 20 number of CD iterations = 1 minibatch size = 10 learning rate = 0.001 updates per epoch: 10 | total updates: 200 Epoch[ 0] : PPL = 994.07 [iter=1] Epoch[ 1] : PPL = 508.56 [iter=1] Epoch[ 2] : PPL = 400.47 [iter=1] Epoch[ 3] : PPL = 353.96 [iter=1] Epoch[ 4] : PPL = 334.94 [iter=1] :For detailed usage of rsm.py, type

% rsm.py -h rsm.py, modified python implementation of Replicated Softmax Model. $Id: rsm.py,v 1.7 2013/06/28 10:23:26 daichi Exp $ usage : rsm.py [options] train model options: -H hiddens number of hidden variables (default = 50) -N epochs number of learning epochs (default = 1) -n iter iterations of contrastive divergence (default = 1) -b batch number of batch size (default = 1) -r rate learning rate (default = 0.001) %or just execute rsm.py.

- To extract parameters, use hash references to the model
- model['w_vh'] = W_vh
- model['w_v'] = W_v
- model['w_h'] = W_h.

- For example, to map documents into hidden activations (binary coding),
use the accompanying script "rsmhidden.py" like this:
("*" means 1, "." means 0, "-" means 0 < h < 0.5, "+" means
0.5 < h < 1)
% rsmhidden.py model.nips test.dat .*..*....*..*..*......*...*........***....*....... .*..*.*.....+.........*........*.................. .**.*.-...............*.-.*......-................ .*..*..............+..*...+.........-............. .*-.*.*...............*.+.*....................... .*..*..............-..*........+-....+............ ..-.-........-........*.-......*.........-..+..... ...............-........*.-....*.........*........ ....*.................*........................... .*..*.-.....*..-......*.+........-..-.............

- To evaluate perplexities on test data, try using the script "rsmppl.py".

- rsm is originally written using Numpy ndarray (not sparse), thus runs fast but consumes pretty much memory.
- Replicated Softmax Model, or RBM in general, is very hard to optimize and
*quite sensitive to hyperparameters*for optimizing using stochastic gradients. Try many hyperparameter combinations that work well for the data at hand.

daichi<at>ism.ac.jp Last modified: Sat Jun 29 18:02:09 2013