Commit e39c3842 authored by Konstantinos Chatzilygeroudis's avatar Konstantinos Chatzilygeroudis
Browse files

Fixes in benchmarks [ci skip]

parent 26ff63e3
This page presents benchmarks in which we compare the Bayesian optimization performance of LIMBO against BayesOpt (https://github.com/rmcantin/bayesopt , a state-of-the-art Bayesian Optimization library in C++).
This page presents benchmarks in which we compare the Bayesian optimization performance of **Limbo** against BayesOpt (https://github.com/rmcantin/bayesopt , a state-of-the-art Bayesian Optimization library in C++).
Each library is given 200 evaluations (10 random samples + 190 function evaluations) to find the optimum of the hidden function. We compare both the accuracy of the obtained solution (difference with the actual optimum solution) and the time (wall clock time) required by the library to run the optimization process. The results show that while the libraries generate solutions with similar accuracy (they are based on the same algorithm), LIMBO generates these solutions significantly faster than BayesOpt.
Each library is given 200 evaluations (10 random samples + 190 function evaluations) to find the optimum of the hidden function. We compare both the accuracy of the obtained solution (difference with the actual optimum solution) and the time (wall clock time) required by the library to run the optimization process. The results show that while the libraries generate solutions with similar accuracy (they are based on the same algorithm), **Limbo** generates these solutions significantly faster than BayesOpt.
In addition to comparing the performance of the libraries with their default parameter values (and evaluating LIMBO with the same parameters than BayesOpt, see variant: limbo/bench_bayes_def), we also evaluate the performance of multiple variants of LIMBO, including different acquisition functions (UCB or EI), different inner-optimizers (CMAES or DIRECT) and whether optimizing or not the hyper-parameters of the model. In all the these comparisons, LIMBO is faster than BayesOpt (for similar results), even when BayesOpt is not optimizing the hyper-parameters of the Gaussian processes.
In addition to comparing the performance of the libraries with their default parameter values (and evaluating **Limbo** with the same parameters than BayesOpt, see variant: limbo/bench_bayes_def), we also evaluate the performance of multiple variants of **Limbo**, including different acquisition functions (UCB or EI), different inner-optimizers (CMAES or DIRECT) and whether optimizing or not the hyper-parameters of the model. In all the these comparisons, **Limbo** is faster than BayesOpt (for similar results), even when BayesOpt is not optimizing the hyper-parameters of the Gaussian processes.
Details
......
This page presents benchmarks in which we compare the performance of the Gaussian Process regression in **Limbo** against two other libraries: GPy (https://github.com/SheffieldML/GPy) and libGP (https://github.com/mblum/libgp).
The quality of the produced model is evaluated according to the Mean Squared Error (lower is better) with respect to the ground truth function. We also quantify the amount of time required by the different libraries to learn the model and to query it. In both cases, lower is better. The evaluations are replicated 30 times and for each replicate, all the variants (see below for the available variants) are using exactly the same data. The data are uniformly sampled and some noise is added (according to the variance of the data).
The comparison is done on 11 tasks to evaluate the performance of the libraries on functions of different complexity, and input/output spaces. The results show that the query time of Limbo's Gaussian processes is several orders of magnitude better than the one of GPy and around twice better than libGP for a similar accuracy. The learning time of Limbo, which highly depends on the optimization algorithm chosen to optimize the hyper-parameters, is either equivalent or faster than the compared libraries.
It is important to note that the objective of the compared libraries are not necessarily the performance, but to provide baselines so that users know what to expect from **Limbo** and how it compares to other GP libraries. For instance, GPy is a python library with much more feature and designed to be easy to use. Moreover, GPy can achieve comparable performance with C++ libraries in the hyper-parameters optimization part because it utilizes numpy and scipy that is basically calling C code with MKL bindings (which is almost identical to what we are doing in **Limbo**).
Variants
-------------------
- **GP-SE-Full-Rprop**: Limbo with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (default for limbo)
- **GP-SE-Rprop**: Limbo with Squared Exponential kernel where the signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (default for limbo) and where the signal noise is not optimized but set to a default value: 0.01
- **libGP-SE-Full**: libGP with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (the only one that libGP has)
- **GPy**: GPy with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation (with the L-BFGS-B optimizer --- `check scipy.optimize.fmin_l_bfgs_b= <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html>`_)
......@@ -66,7 +66,7 @@ except:
try:
import matplotlib
matplotlib.use('Agg') # for headless generation
matplotlib.use('Agg') # for headless generation
from pylab import *
pylab_found = True
except:
......@@ -148,7 +148,7 @@ def clean_labels(k1, k2):
'bayesopt_default':'BayesOpt defaults, ',
'bench_bayesopt_def':'BayesOpt defaults, ',
'bench_': '',
}
m2 = { 'limbo': 'Limbo',
'bayesopt' : 'BayesOpt'}
......@@ -168,7 +168,7 @@ def get_notes():
'rastrigin':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/rastr.html>`_ (note, the input space has been rescaled and shifted to have the minimum at [1,1,1,1], instead of [0,0,0,0]',
'sixhumpcamel':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/camel6.html>`_.',
'sphere':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/spheref.html>`_.'};
return notes
# plot a single function
......@@ -184,7 +184,7 @@ def plot(func_name, data, rst_file):
'figure.figsize' : [11, 2.5]
}
rcParams.update(params)
# plot
d = data[func_name]
da_acc = []
......@@ -200,7 +200,7 @@ def plot(func_name, data, rst_file):
if 'def' in y and 'def' not in x:
return -1
return x < y
for k in sorted(d.iterkeys()):
for k2 in sorted(d[k].iterkeys(), sort_fun):
da_acc.append(d[k][k2][:, 0])
......@@ -223,12 +223,13 @@ def plot(func_name, data, rst_file):
ax.set_title("Wall clock time (s)")
notes = get_notes()
name = func_name.split('.')[0]
fig.savefig("benchmark_results/fig_benchmarks/" + name + ".png")
fig.savefig("benchmark_results/fig_benchmarks/" + name + ".png")
rst_file.write(name.title() + " function\n")
rst_file.write("-----------------\n\n")
rst_file.write(notes[name] + " \n\n")
if name in notes:
rst_file.write(notes[name] + " \n\n")
rst_file.write(str(len(da_acc[0])) + " replicates \n\n")
rst_file.write(".. figure:: fig_benchmarks/" + name + ".png\n\n")
......@@ -237,7 +238,7 @@ def plot(func_name, data, rst_file):
def include(src_file, dst_file):
for i in open(src_file):
dst_file.write(i)
def plot_all():
if not plot_ok:
print_log('YELLOW', "No plot")
......@@ -257,8 +258,7 @@ def plot_all():
rst_file.write("*" + date + "* -- " + node + " (" + str(multiprocessing.cpu_count()) + " cores)\n\n")
include("docs/benchmark_res_bo.inc", rst_file)
print('loading data...')
data = load_data()
print('data loaded')
......
......@@ -81,7 +81,7 @@ params = {
rcParams.update(params)
def load_data():
files = glob("regression_benchmark_results/*/*/*.dat")
files = sorted(glob("regression_benchmark_results/*/*/*.dat"))
points = defaultdict(lambda : defaultdict(lambda : defaultdict(dict)))
times_learn = defaultdict(lambda : defaultdict(lambda : defaultdict(dict)))
times_query = defaultdict(lambda : defaultdict(lambda : defaultdict(dict)))
......@@ -159,7 +159,7 @@ def plot_ax(ax, data, points, labely, disp_legend=True, disp_xaxis=False):
if var_p[i] not in pp:
pp[var_p[i]] = []
pp[var_p[i]].append(var_data[i])
pp = OrderedDict(sorted(pp.items()))
x_axis = pp.keys()
......@@ -178,7 +178,7 @@ def plot_ax(ax, data, points, labely, disp_legend=True, disp_xaxis=False):
ax.plot(x_axis, y_axis, '-o', color=c_kk, linewidth=3, markersize=5)
ax.fill_between(x_axis, y_axis_75, y_axis_25, color=c_kk, alpha=0.15, linewidth=2)
kk = kk + 1
if disp_legend:
ax.legend(labels)
if disp_xaxis:
......@@ -189,19 +189,27 @@ def plot_ax(ax, data, points, labely, disp_legend=True, disp_xaxis=False):
return replicates
def get_notes():
notes={'Rastrigin':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/rastr.html>`_.',
'Pistonsimulation':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/piston.html>`_.',
'Step':'Step function: for :math:`x\in[-2; 2]` :\n.. math::\n f(x) = \\begin{cases} 0, &\\mbox{if x<=0} \\\\ 1, &\\mbox{otherwise} \\end{cases}',
'Robotarm':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/robot.html>`_.',
'Gramacylee':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/grlee12.html>`_.',
'Planarinversedynamicsii':'Approximation of the second motor\'s torque in the inverse dynamics of a Planar 2D Arm. Details are given at the bottom of this page.',
'Planarinversedynamicsi':'Approximation of the first motor\'s torque in the inverse dynamics of a Planar 2D Arm. Details are given at the bottom of this page.',
'Otlcircuit':'Details about the function can be found `here <https://www.sfu.ca/~ssurjano/otlcircuit.html>`_.'};
notes={'Rastrigin':"Details about the function can be found `here <https://www.sfu.ca/~ssurjano/rastr.html>`_.",
'Pistonsimulation':"Details about the function can be found `here <https://www.sfu.ca/~ssurjano/piston.html>`_.",
'Step':"Step function: for :math:`x\in[-2; 2]` :\n\n.. math::\n f(x) = \\begin{cases} 0, &\\mbox{if x<=0} \\\\ 1, &\\mbox{otherwise} \\end{cases}",
'Robotarm':"Details about the function can be found `here <https://www.sfu.ca/~ssurjano/robot.html>`_.",
'Gramacylee':"Details about the function can be found `here <https://www.sfu.ca/~ssurjano/grlee12.html>`_.",
'Planarinversedynamicsii':"Approximation of the second motor\'s torque in the inverse dynamics of a Planar 2D Arm. Details are given at the bottom of this page.",
'Planarinversedynamicsi':"Approximation of the first motor\'s torque in the inverse dynamics of a Planar 2D Arm. Details are given at the bottom of this page.",
'Otlcircuit':"Details about the function can be found `here <https://www.sfu.ca/~ssurjano/otlcircuit.html>`_."};
return notes
def get_names():
names={'Pistonsimulation':'Piston Simulation',
'Gramacylee':'Gramacy-Lee',
'Robotarm':'Robot Arm',
'Planarinversedynamicsi':'Planar Inverse Dynamics I',
'Planarinversedynamicsii':'Planar Inverse Dynamics II',
'Otlcircuit':'OTL Circuit'}
return names
def planarinversedynamics_math():
res="""Inverse Dynamics of a Planar 2D Arm (I \& II): for :math:`\ddot{q}\in[-2\pi; 2\pi]^2`; :math:`\dot{q}\in[-2\pi; 2\pi]^2`; :math:`q\in[-pi; pi]^2`
.. math::
res="""Inverse Dynamics of a Planar 2D Arm (I \& II): for :math:`\ddot{q}\in[-2\pi; 2\pi]^2`; :math:`\dot{q}\in[-2\pi; 2\pi]^2`; :math:`q\in[-pi; pi]^2`\n\n.. math::
\\begin{gather}
\\tau (q,\dot{q},\\ddot{q})=\\textbf{M}(q)\\ddot{q}+ \\textbf{C}(q,\\dot{q})\\dot{q}\\\\
\\textrm{where:}\\\\
......@@ -230,9 +238,14 @@ def plot_data(bench, func, dim, mses, query_times, learning_times, points, rst_f
fig.savefig('regression_benchmark_results/'+bench+'_figs/'+name+'.png')
close()
notes=get_notes()
rst_file.write(func.title() + " in " + str(dim) + "D\n")
rst_file.write("-----------------\n\n")
rst_file.write(notes[func.title()] + " \n\n")
func_name = func.title()
names = get_names()
if func_name in names:
func_name = names[func_name]
rst_file.write(func_name + " in " + str(dim) + "D\n")
rst_file.write("----------------------------------\n\n")
if func.title() in notes:
rst_file.write(notes[func.title()] + " \n\n")
rst_file.write(str(replicates) + " replicates \n\n")
rst_file.write(".. figure:: fig_benchmarks/" + bench + "_figs/" + name + ".png\n\n")
......@@ -246,14 +259,21 @@ def plot(points,times_learn,times_query,mses,rst_file):
except:
print('WARNING: directory \'%s\' could not be created! (it probably exists already)' % fig_dir)
# for each function
for func in points[bench].keys():
functions = sorted(points[bench].keys())
for func in functions:
# for each dimension
for dim in points[bench][func].keys():
dims = sorted(points[bench][func].keys())
for dim in dims:
print('plotting for benchmark: ' + bench + ', the function: ' + func + ' for dimension: ' + str(dim))
name = bench+'_'+func+'_'+str(dim)
plot_data(bench, func, dim, mses[bench][func][dim], times_query[bench][func][dim], times_learn[bench][func][dim], points[bench][func][dim], rst_file)
# dst file is already open
def include(src_file, dst_file):
for i in open(src_file):
dst_file.write(i)
def plot_all():
if not plot_ok:
print_log('YELLOW', "No plot")
......@@ -266,20 +286,7 @@ def plot_all():
node = platform.node()
rst_file.write("*" + date + "* -- " + node + " (" + str(multiprocessing.cpu_count()) + " cores)\n\n")
rst_file.write("- We compare to GPy (https://github.com/SheffieldML/GPy) and libGP (https://github.com/mblum/libgp)\n")
rst_file.write("- Mean Squared Error: lower is better\n")
rst_file.write("- Learning time: lower is better\n")
rst_file.write("- Querying time (for 1 by 1 query points): lower is better\n\n")
rst_file.write("- In each replicate, all the variants (see below for the variants available) are using exactly the same data\n\n")
rst_file.write("- The data are uniformly sampled and some noise is added (according to the variance of the data).\n\n")
rst_file.write("Naming convention\n")
rst_file.write("------------------\n\n")
rst_file.write("- GP-SE-Full-Rprop: Limbo with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (default for limbo)\n")
rst_file.write("- GP-SE-Rprop: Limbo with Squared Exponential kernel where the signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (default for limbo) and where the signal noise is not optimized but set to a default value: 0.01\n")
rst_file.write("- libGP-SE-Full: libGP with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation with the Rprop optimizer (the only one that libGP has)\n")
rst_file.write("- GPy: GPy with Squared Exponential kernel where the signal noise, signal variance and kernel lengthscales are optimized via Maximum Likelihood Estimation (with the L-BFGS-B optimizer --- `check scipy.optimize.fmin_l_bfgs_b= <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html>`_)\n\n")
include("docs/benchmark_res_reg.inc", rst_file)
print('loading data...')
points,times_learn,times_query,mses = load_data()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment