RATS offers several optimization methods which do not require a differentiable function. In general, continuity is the only requirement for these. Most are used only as a preliminary method (PMETHOD option) prior to using a hill-climbing method to do a final optimization.

Simplex Method

The simplex method (METHOD/PMETHOD=SIMPLEX) is a search procedure which requires only function evaluations, not derivatives. It works using a geometrical object (called a simplex) connecting K+1 points in K-space, where K is the number of parameters. At each stage, it attempts to replace the worst of the K+1 points with something better. So it "climbs" the hill by eliminating directions in which the function is lower, rather than directly trying to go up.

 

It is by far the most commonly used of these, as it is reasonably efficient at moving off guess values into a region where the hill-climbing methods can work better. It does assume that you start on the correct "hill" and so is not as useful for a function which is known or suspected to have multiple modes.

Genetic algorithm

Genetic algorithms (METHOD/PMETHOD=GENETIC) are designed to handle difficult optimization problems by applying lots of compute power and a stylized model of “evolution,” with random “mutations.” RATS applies a variation of this called differential evolution. Given a “population” of parameter vectors, at each iteration each vector is compared with a possible successor. Whichever is the “fitter” of the two (the one with the better function value) is retained, the other discarded. It is more robust to oddities in the surface than simplex, at a significant computational cost. However, because it is always trying to move "uphill" it may not work as well on a multi-modal surface as annealing and genetic annealing.

Simulated Annealing

Simulated annealing (option METHOD/PMETHOD=ANNEALING) is an algorithm originally proposed to solve (or at least approximately solve) certain combinatorial problems such as the traveling salesman problem (minimizing the total distance required to hit each of a set of stops exactly once), however, it can also be applied to functions which are continuous in their parameter space.

 

The key to simulated annealing is that it can make changes that can either increase or decrease the function. The name is based upon the annealing process in metallurgy, where the energy state generally goes down, but can occasionally go up, as metal cools. At a high “temperature” the moves in the “wrong” direction are more likely, and become less likely as the “temperature” declines. Because of this, it has the possibility of working with surfaces that have multiple modes. It also can work with discontinuous functions.

Genetic Annealing

Genetic Annealing (option METHOD/PMETHOD=GA) is a cross between the genetic algorithm and simulated annealing. It’s mainly the genetic algorithm (thus tracks a population of parameter sets) which are mutated at each point, but, instead of only picking the “fittest” (that is, replacing one parameter set only if the mutated one produces a better function value) uses an annealing schedule so it might replace it with one that’s not as good. This has (by far) the greatest computational cost of all four, but does the broadest search of the parameter space.

 

Copyright © 2025 Thomas A. Doan