Theano is this really pretty package for Python — it operates on top of Numpy to provide short hand and functionality useful for machine learning algorithms. I bumped into this when I was running some demo code available at deeplearning.net. The demo data used by this code is actually a Python Pickled +Gzipped version of the MNIST handwritten digit database. In this scenario, all I wanted to do was print the MNIST data after it had been already been loaded by Theano into the demo script. The data is accessible via instances of the theano.TensorVariable class.

The first attempt worked for the rows of data corresponding to the images stored as 2D arrays since Theano tensors of type matrix have a convenience method that lets one grab the data as a familiar 2D Numpy array using an undocumented get_value() method (it doesn’t appear in the API for theano.TensorVariable, but is mentioned briefly elsewhere). This convenience method disappears for the 1D arrays, since these were not constructed with a shared reference to an underlying Numpy array.

After spending a solid amount of time working away with Python’s dir(), searching for a way to coerce Theano into showing me its innards, I finally came to my senses and realized I should probably see what canonical solutions are. Completely new to Theano, and only having recently read about them — I simply wasn’t well versed enough with the notion of compiling a Theano function. Finally, peering into the loaded MNIST data came down to this …

datasets = load_data(dataset)

train_set_x, train_set_y = datasets[0]
valid_set_x, valid_set_y = datasets[1]
test_set_x , test_set_y  = datasets[2]

##### -- my code begins here ...

def print_set(data, name):
    with open("extract/" + name + ".txt", "w+") as handle:
        print >>handle, "# type(python)", type(data)
        print >>handle, "# type(theano)", data.type
        print >>handle, "# dimensions  ", data.ndim
        if data.ndim == 2:
            print >>handle, "# rows(numpy) ", len(data.get_value())
            print >>handle, "# cols(numpy) ", len(data.get_value()[0])
            for row in theano.function([], data)():
                for val in row:
                    print >>handle, "%0.3f" % val,
                print >>handle
        elif data.ndim == 1:
            for row in theano.function([], data)():
                print >>handle, "%d" % row
        else:
            return

print_set(train_set_x, "train_set_x")
print_set(train_set_y, "train_set_y")
print_set(valid_set_x, "valid_set_x")
print_set(valid_set_y, "valid_set_y")
print_set(test_set_x,  "test_set_x" )
print_set(test_set_y,  "test_set_y" )

return

Here, the magic is in the constructor theano.function([], data) (– Thanks: Olivier Delalleau) The great thing about working through this problem is that I should be able to extend this solution to building actual functions in Theano. The stumbling around did force me to understand more about Theano’s tensors and Theano’s compiled function graphs.

Updated thanks to Wyatt and Devon in the comments.

The assignment below …

int A = B == C ? D : E;

Will assign A to D if B is C and A to E if B is not C. This is equal to saying …

int A = (B == C) ? D : E; // Correct

An incorrect interpretation is …

int A = B == (C ? D : E); // Incorrect

Thanks to Wyatt for checking this and pointing it out in the comments — I had a typo in my original code which lead to an incorrect interpretation. Fittingly of a CogFault, the intro is still true (ternary has lower precedence). This post has been updated to reflect these changes.

Either way, as Devon points out in the comments — when the expression “B == C” gets even slightly more complicated, like “B == C-1″, then it’s better to use parentheses.

So, thanks everyone for making this CogFault better.

*Note that the middle expression between “?” and “:” of the ternary conditional operator is always evaluated first.

Solution: add this line to .bash_profile …

export MANPATH=/usr/share/man:$MANPATH

… as /usr/share/man is the default location of the system man pages.

Example — several functions execute a statement similar to the update line below.

static final int PRG = 100; // total progress that each thread contributes
static final int TOTAL_PRG = 1000; // total progress for all ten threads
static int progress = 0;

public double calculateSomething(double[] inputs) {
    double output = 0;
    int prev = 0;
    for(int iter = 0; iter < maximum; iter ++) {
        int diff = PRG * iter / maximum % PRG - prev;
        if(diff > 0) {
            progress += diff; // !!! update the progress bar integer !!!
            prev += diff;
        }
        // . . . math or function logic goes here --
        // calculate something with inputs, save in output . . .
    progress += PRG - prev; // finishes this bar
    return output;
}

Summary: Enclose the updating in a synchronized function as below.

protected synchronized void updateProgress(int amount) {
    progress += amount;
}

// ... OR -- make a thread safe mutator / accessor combo ...
// -- giving this function 0 will make it behave like a get.

protected synchronized int updateProgress(int amount) {
    progress += amount;
    return progress;
}

public double calculateSomething(double[] inputs) {
    double output = 0;
    int prev = 0;
    for(int iter = 0; iter < maximum; iter ++) {
        int diff = PRG * iter / maximum % PRG - prev;
        if(diff > 0) {
            updateProgress(diff); // update the progress bar integer (fixed)
            prev += diff;
        }
        // . . . math or function logic goes here --
        // calculate something with inputs, save in output . . .
    updateProgress(PRG - prev); // finishes this bar (fixed)
    return output;
}

Lesson: Always be thread safe — even on operations that look like they should be atomic (such as addition above).

Symptoms: Trying to archive 175MB of data over 7680 files resulted in this …

$ tar -cvzf exp_tin.tgz exp_tin
tar: exp_tin: Cannot savedir: Cannot allocate memory
tar: Error exit delayed from previous errors
Tin:CIS6650_Chicago eddiema$

Delving into this, the last reports of this issue that Google could dig up was all the way back in 2005 — that fits with this information …

$ tar --version
tar (GNU tar) 1.14
Copyright (C) 2004 Free Software Foundation, Inc.
This program comes with NO WARRANTY, to the extent permitted by law.
You may redistribute it under the terms of the GNU General Public License;
see the file named COPYING for details.
Written by John Gilmore and Jay Fenlason.

None of my other machines had this problem –

Finally, I did this …

$ which tar
/sw/bin/tar

Which made me realize that this version of tar came from an old fink update.

Solution: I moved /sw/bin/tar to /sw/bin/tar114 and the default version of tar given by OSX became the default …

$ tar --version
bsdtar 2.6.2 - libarchive 2.6.2

Example:

incorrect …

heat2.exe: heatmap/OtherHeat.cs heatmap/heatmap.cs
    gmcs -r:System.drawing -out:$@ $+ -main:$</HMOther

correct …

heat2.exe: heatmap/OtherHeat.cs heatmap/heatmap.cs
    gmcs -r:System.drawing -out:$@ $+ -main:heatmap.HMOther

This took a while to track down since I kept looking at the directory structure rather than the namespaces!

It turns out it was an added newline (rowbreak) immediately after a sideways multirow entry in a LaTeX table.

\begin{table}[h!tp]
\caption{\label{table:irisgamma1}
The allowed values of $\Gamma_1$ in the Iris Dataset cluster task given
$N, \alpha_0$.
}
\centering
\begin{tabular}{c | c | c c c c c c}
$\Gamma_1$ & \multicolumn{7}{c}{Epochs $N$} \\ \hline
\multirow{6}{*}
{
\begin{sideways}
Learning $\alpha_0$
\end{sideways}
} \\ % <-- EXTRA NEWLINE HERE
 &   & 1 & 10 & 50 & 100 & 500 & 1000\\
 \cline{2-8}
 & 0.01 & b & c & d & e & f & g\\
 & 0.05 & b & c & d & e & f & g\\
 & 0.1 & b & c & d & e & f & g\\
 & 0.5 & b & c & d & e & f & g\\
 & 1.0 & b & c & d & e & f & g\\
\end{tabular}
\end{table}

Which results in this ugliness …

The outermost item in that cell is not the sideways environment, rather it is the multirow — i.e. count the number of columns before adding a linebreak.

The above example with the rowbreak removed (correct).

goto STEP6;
# Some code happens here that we don't want to run ...
STEP6:

sub do_the {
}

Reason: PERL will not accept the subroutine declaration immediately after a goto label.

Solution: Add an empty statement at the goto label …

STEP6:; #BAM! Empty statement.

sub do_the {
}

Note — error messages will generally be ambiguous as in …

syntax error at make.pl line 279, near "sub do_the "
Execution of make.pl aborted due to compilation errors.

This was the PERL script I used …

while(<>) {
    $_ =~ s/[^[:ascii:]]+/-/;
    print $_;
}

Which only picked up the first non-ascii character per line — this is the correct script …

while(<>) {
    $_ =~ s/[^[:ascii:]]+/-/g; #BAM! 'g' flag!
    print $_;
}

Regular expressions do greedy matching for the Kleene Star (‘*’) and Kleene Plus (‘+’) operators; use the question mark (‘?’) to denote that we want the shortest match possible instead where appropriate.