This page present some benchmarks of Nieme. For the moment it only concerns binary classification on moderate-size datasets.

Binary Classification - Moderate-size datasets

The experiment described below compares several learning machines from Nieme with a baseline linear SVM.

Datasets

The datasets are classical binary classification datasets that can be found in the libSVM datasets. When a train/test split was provided we kept it, otherwise we did a random split of 50% training and 50% of testing data. Datasets with binary sparse features are processed directly, whereas we use the scaled version (features in [-1,1]) for other datasets.

Baseline computation

The baseline is a linear SVM with the libSVM implementation. For each dataset, we tried the values 0.001, 0.01, 0.1, 1.0, 10.0 and 100.0 for the parameter C, and kept the best result.

Nieme machines

The experiment compares several learning machines in Nieme:

• Online machines:
  • Losses: Perceptron, LargeMargin, LogBinomial
  • Learners: Stochastic Descent, MiniBatch Descent (mini batch sizes: 2, 5, 10, 20, 50, 100, 200, 500)
• Batch machines:
  • Losses: LogBinomial, Exponential
  • Learners: OWLQN, LBFGS, RProp
  • Regularizers: None / Only L1 / Only L2 / Combined L1 and L2. For each regularizers we tried the weights 1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3 and 0.01.

In all cases, the machines are based on a linear Architecture. These choices lead to a bit more than 400 differently parameterized learning machines.

Results

Each row of the table below corresponds to a dataset. We give the name of the dataset, the number of training and testing examples, the number of features, the best libSVM generalization accuracy with the corresponding parameter C, and the best Nieme's model generalization accuracy with its description.

Since we try much more models from Nieme than libSVM models, the former most-of-time outperforms the latter in accuracy. These results are only preliminary results and the aim is only to illustrate the variety of models that can be handled by Nieme.

More than 10.000 training examples

More than 1.000 training examples

Small DataSets