So in practice, you have to invest a lot of computing time at the beginning to find a good configuration. In the use phase of the algorithm, you benefit from each runtime so that overall, resources can be saved.
Is it possible to shorten the computing time at the beginning?
Britta Heymann: Parallelization is the most important thing! We can either execute parallel evaluations at the same time on one computer or we even use several computers in one computing network. This is what I would recommend – in connection with Cloud-Computing, for example.
As far as the runtime is concerned with tuning, we can even terminate evaluations early on if parallel results show us that the associated configuration may not be one of the best.
It’s also important to ensure that you can terminate an evaluation after a certain time. Otherwise the tuner might spend too much time on a run which has, quite by chance, started with a very poor configuration and where it is actually clear early on that it will produce a very bad result.
Which algorithms is tuning particularly suitable for? Or are there any algorithms for which it isn’t suitable?
Jannick Lange: It’s always a good idea for algorithms which have a relatively long runtime or if you have so-called run-time critical algorithms. For example, when you have a model that needs to be solved within a specific runtime. Sometimes this is only possible with tuning.
Hmm, what isn’t suitable? Well basically, an algorithm naturally has to provide parameters which the behavior can be influenced with.
Tuning is also diffcult if only very few instances are available or if individual evaluations require several hours or days. In the first case there’s a risk of overfitting, that means a parameter combination which is too specific for the training instances and generally offers no benefits. In the second case it could well be that the time required to perform tuning increases astronomically. One possible solution to this problem would be to use “smaller” instances from the same problem class for tuning – assuming that the configurations found are transferrable based on their structural similarity.
Britta Heymann: Basically, you have to decide whether you want to invest the time in tuning at the beginning. I think it’s the most useful if the algorithm fulfills a short runtime or if you use the algorithm very often. Then it’s worth it because you benefit from every run.
The faster the algorithm is from the start, the faster the tuner finds good configurations. That means that even algorithms which are realtively fast can benefit from tuning because tuning itself then works more quickly.