Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
Matthias Mayr
limbo
Commits
c0949673
Commit
c0949673
authored
Apr 06, 2018
by
Jean-Baptiste Mouret
Browse files
include the text from a different file
parent
a586f13b
Changes
2
Hide whitespace changes
Inline
Side-by-side
docs/benchmark_res_bo.inc
0 → 100644
View file @
c0949673
This
page
presents
benchmarks
in
which
we
compare
the
Bayesian
optimization
performance
of
LIMBO
against
BayesOpt
(
https
://
github
.
com
/
rmcantin
/
bayesopt
,
a
state
-
of
-
the
-
art
Bayesian
Optimization
library
in
C
++
)
.
Each
library
is
given
200
evaluations
(
10
random
samples
+
190
function
evaluations
)
to
find
the
optimum
of
the
hidden
function
.
We
compare
both
the
accuracy
of
the
obtained
solution
(
difference
with
the
actual
optimum
solution
)
and
the
time
(
wall
clock
time
)
required
by
the
library
to
run
the
optimization
process
.
The
results
show
that
while
the
libraries
generate
solutions
with
similar
accuracy
(
they
are
based
on
the
same
algorithm
),
LIMBO
generates
these
solutions
significantly
faster
than
BayesOpt
.
In
addition
to
comparing
the
performance
of
the
libraries
with
their
default
parameter
values
(
and
evaluating
LIMBO
with
the
same
parameters
than
BayesOpt
,
see
variant
:
limbo
/
bench_bayes_def
),
we
also
evaluate
the
performance
of
multiple
variants
of
LIMBO
,
including
different
acquisition
functions
(
UCB
or
EI
),
different
inner
-
optimizers
(
CMAES
or
DIRECT
)
and
whether
optimizing
or
not
the
hyper
-
parameters
of
the
model
.
In
all
the
these
comparisons
,
we
can
observe
that
LIMBO
is
faster
than
BayesOpt
,
ebven
when
BayesOpt
is
not
optimizing
its
hyper
-
parameters
.
Details
-------
-
We
compare
to
BayesOpt
(
https
://
github
.
com
/
rmcantin
/
bayesopt
)
-
Accuracy
:
lower
is
better
(
difference
with
the
optimum
)
-
Wall
time
:
lower
is
better
-
In
each
replicate
,
10
random
samples
+
190
function
evaluations
-
see
`src/benchmarks/limbo/bench.cpp`
and
`src/benchmarks/bayesopt/bench.cpp`
waf_tools/plot_bo_benchmarks.py
View file @
c0949673
...
...
@@ -217,6 +217,12 @@ def plot(func_name, data, rst_file):
rst_file
.
write
(
str
(
len
(
da_acc
[
0
]))
+
" replicates
\n\n
"
)
rst_file
.
write
(
".. figure:: fig_benchmarks/"
+
name
+
".png
\n\n
"
)
# dst file is already open
def
include
(
src_file
,
dst_file
):
for
i
in
open
(
src_file
):
dst_file
.
write
(
i
)
def
plot_all
():
if
not
plot_ok
:
print_log
(
'YELLOW'
,
"No plot"
)
...
...
@@ -235,23 +241,7 @@ def plot_all():
node
=
platform
.
node
()
rst_file
.
write
(
"*"
+
date
+
"* -- "
+
node
+
" ("
+
str
(
multiprocessing
.
cpu_count
())
+
" cores)
\n\n
"
)
rst_file
.
write
(
"- We compare to BayesOpt (https://github.com/rmcantin/bayesopt)
\n
"
)
rst_file
.
write
(
"- Accuracy: lower is better (difference with the optimum)
\n
"
)
rst_file
.
write
(
"- Wall time: lower is better
\n\n
"
)
rst_file
.
write
(
"- In each replicate, 10 random samples + 190 function evaluations
\n
"
)
rst_file
.
write
(
"- see `src/benchmarks/limbo/bench.cpp` and `src/benchmarks/bayesopt/bench.cpp`
\n\n
"
)
rst_file
.
write
(
"Naming convention
\n
"
)
rst_file
.
write
(
"------------------
\n\n
"
)
rst_file
.
write
(
"- limbo_def: default Limbo parameters
\n\n
"
)
rst_file
.
write
(
"- opt_cmaes: use CMA-ES (from libcmaes) to optimize the acquisition function
\n
"
)
rst_file
.
write
(
"- opt_direct: use DIRECT (from NLopt) to optimize the acquisition function
\n
"
)
rst_file
.
write
(
"- acq_ucb: use UCB for the acquisition function
\n
"
)
rst_file
.
write
(
"- acq_ei: use EI for the acquisition function
\n
"
)
rst_file
.
write
(
"- hp_opt: use hyper-parameter optimization
\n
"
)
rst_file
.
write
(
"- bayesopt_def: same parameters as default parameters in BayesOpt
\n
"
)
include
(
"docs/benchmark_res_bo.inc"
,
rst_file
)
print
(
'loading data...'
)
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment