Neural Network Basics

A previous post introduced ojAlgo's Artificial Neural Network feature. It did so by presenting a fully functional program that trained and evaluated a network model to categorise handwritten digits using the MNIST data set. That example then included all necessary pre and post processing as well as code that would generate the actual image (*.png) files. All of that may have obscured just how simple it is to work with ojAlgo's neural networks.

The code below is not complete – it simply outlines the basic steps involved to build, train and use a neural network with ojAlgo.

Now have a look at that fully functional example...

Introducing Artificial Neural Networks with ojAlgo

With v46 ojAlgo got support for building artificial neural networks. Here's an example of what you can do with it.

The MNIST database is a large image database of handwritten digits that is commonly used for training and testing in the field of machine learning / image categorisation. Information regarding this dataset and various results achieved is widely published.

• http://yann.lecun.com/exdb/mnist/
• https://en.wikipedia.org/wiki/MNIST_database

A correctly modelled/trained neural network should be able to achieve a 5% error rate on this dataset. Most/all published results are better than that. The largest, most advanced, models have managed 0,35%. That's almost unbelievably good. ojAlgo currently doesn't have all the features required to build that kind of model. The model in the program listed below gets about 2.2% error rate. Here are some sample digits/images from the data set.

The program below (with its dependency on ojAlgo) can do the following:

• Read/parse the files containing the image data and labels.
• Generate the actual images so that you can inspect them. The example images above are generated with that code.
• Print images to console (to sanity check results)
• Model and train feedforward neural networks:
• Any number of layers
• Any number of input/output nodes per layer
• Choose between 5 different activator and 2 different error/loss functions 

The main benefit of using ojAlgo is how easy it is to do this and get good results. Download the example code below (you also need ojAlgo v46.1.1 or later) and run it, and start modifying the network structure, learning rate and other things. (You also need to download the data files, and update the various paths in the programs.)

ojAlgo Examples

All ojAlgo example code previously published on the ojAlgo wiki has now been moved to a GitHub Gist. On this page you see the complete contents of that gist. (Never mind the eclipse and git files at the top. Scroll down the page.) When/if anything is added to that gist this page is automatically updated.

Code

New Java Matrix Benchmark Results Coming

I'm in the process of executing the Java Matrix Benchmark on a Google Cloud Platform Compute Engine (n1-highmem-4 (4 vCPUs, 26 GB memory) with Intel Skylake processors). Actually I'm doing 3 separate benchmark executions using 3 different JVM:s on otherwise identical machines:

1. Java(TM) SE Runtime Environment (build 1.8.0_161-b12) 
2. Java(TM) SE Runtime Environment (build 9.0.4+11) 
3. Zing Runtime Environment for Java Applications (build 1.8.0-zing_18.04.0.0-b2) 

When all is done it'll be very interesting to compare the results between the JVM:s.

The results for all pure Java libraries running on Oracle 8 are already complete. Here's the summary performance chart for that:

It'll be another 2 weeks or so before everything is done. By then there will be results for these library and JVM combinations:

		Oracle 8	Oracle 9	Zing 8
Colt	1.2.0	X	X	X
Apache Commons Math	3.6.1	X	X	X
EJML	0.33	X	X	X
JAMA	1.0.3	X	X	X
jblas (using native code)	1.2.4		X
la4j	0.6.0	X	X	X
MTJ (pure Java)	1.0.7	X	X	X
MTJ-N (using native code)	1.0.7	X	X
ojAlgo	45.0.0	X	X	X
Parallel Colt	0.9.4	X	X	X
UJMP (pure Java)	0.3.0	X	X	X
UJMP-N (using native code)	0.3.0	X	X

The jblas library (using native code) crashed the benchmark twice running on different JVM:s. As part of the plan for how to recover and resume the executions it was decided to not re-run the libraries using native code on the all the instances – to save time. The native code libraries are not the most interesting when comparing JVM:s.

Further, and more detailed, results will be published here, at the ojAlgo wiki and/or at the Java Matrix Benchmark web site:

• https://ojalgo.blogspot.com
• https://github.com/optimatika/ojAlgo/wiki/Java-Matrix-Benchmark
• https://lessthanoptimal.github.io/Java-Matrix-Benchmark/

Matrix multiplication on different JVM:s

Most developers are aware of the possibility to use either a JVM from Oracle or "just" OpenJDK, but few consider using any of the other alternatives. There are more alternatives!

https://en.wikipedia.org/wiki/List_of_Java_virtual_machines

What if choosing one of the alternatives could significantly improve the performance of your application?

I frequently see discussions and benchmarks comparing different libraries (that solve the same problem). Comparing Java performance to that of native code is also common, but I don't think I've ever seen a benchmark comapring different JVM:s. I decided to do this - I decided to run the Java Matrix Benchmark using both the Oracle JVM and Zing from Azul Systems.

I haven't done it yet, but as part of the preparations, I've executed some smaller tests. Below you'll find the results from benchmarking matrix multiplication using 3 different pure Java linear algebra libraries on both the Oracle JVM and Azul's Zing.

This chart shows the relative performance on matrix multiplication of 3 different Java linear algebra libraries, using 2 different JVM:s. The x-axis shows the (square) matrix size, and the y-axis the relative performance.

Which 3 Java linear algebra libraries?

• ojAlgo v45.0.0-SNAPSHOT 
• EJML v0.33 
• Apache Commons Math (ACM) v3.6.1

ojAlgo is the only multi-threaded library among these.

Which 2 JVM:s?

• JDK 1.8.0_161, VM 25.161-b12 (Oracle 8) 
• JDK 1.8.0-zing_17.12.1.0, VM 1.8.0-zing_17.12.1.0-b1-product-azlinuxM-X86_64 (Zing 8) 

Relative to what?

Actually a 4:th linear algebra library was benchmarked – Matrix Toolkits Java (MTJ) v1.0.7. It is based on netlib-java which use native code libraries if available. The machine in question had some plain/default version of ATLAS installed which was used. (just did 'sudo yum install atlas'). Since the actual matrix multiplication is done in native code the performance was essentially identical between the different JVM:s. This was used as a reference.

"1.0" on the y-axis represents the speed of MTJ/netlib-java/ATLAS. Then for the other library and JVM combinations you can see how much faster or slower they where.

Interpreting the results

These are my conclusions:

1. The choice of JVM does make a difference (it did in this case) and Oracle is not necessarily the best option.
2. Performance differences between the JVM:s are of the same magnitude as between different code libraries.
3. Depending on which code library you use, the effects of changing JVM:s vary – you need to look at the individual combinations.
4. ojAlgo is the only multi-threaded library here. This allows it to be more than 4 times faster than the other libraries (with the larger matrices). In those cases there is also a significant difference between Oracle and Zing – it's another 4x speed improvement. This would be vectorisation happening in the JVM. Modern JVM:s have support to automagically utilise SIMD instructions. This feature is brittle, it doesn't always happen! It seems Zing agreed with ojAlgo and managed to vectorise the code, where Oracle did not.
5. Java is fast enough to sometimes be faster than native. In this case the native code library is not the fastest possible. I believe ATLAS, that was used in this case, can be tuned to perform better than it did here, and there are other alternatives (perhaps to purchase an Intel MKL license). But, unless you do something to ensure the best possible native performance, a good Java library may be faster (as shown here).

Look at that chart again. For the larger matrix sizes there is almost a 100x speed increase between the slowest and the fastest library/JVM combination.

What kind of machine was this run on?

The benchmark was executed on a Google Cloud Platform n1-standard-8 (8 vCPUs, 30 GB memory) with Intel Skylake processors.

Full execution output logs

The same benchmark was executed 3 times using different JVM:s.

• Oracle 8
• Zing 8
• Oracle 9

The benchmark software

https://github.com/optimatika/ojAlgo-linear-algebra-benchmark