Many real-world optimisation problems look simple on paper but become difficult in practice. You may be trying to minimise a cost function, maximise a score, or find the best configuration among millions of possibilities. In machine learning and analytics, this shows up in feature selection, hyperparameter tuning, clustering, and scheduling. A key challenge is that many objective functions contain multiple “good” regions, including local optima that trap straightforward search methods. Simulated annealing (SA) is a classic technique designed to escape these traps by occasionally accepting worse solutions early on, then gradually becoming more selective. In many data science classes in Bangalore, simulated annealing is introduced as an intuitive bridge between greedy optimisation and more advanced stochastic methods.
Why Global Optimisation Is Hard
If an objective function is convex, simple methods like gradient descent can reliably move toward a single best solution. But many functions are not convex. They may be noisy, discontinuous, or defined over discrete choices rather than continuous values. For example:
- Selecting a subset of features from hundreds of candidates is combinatorial.
- Assigning points to clusters can create many competing partitions.
- Choosing model hyperparameters often produces a “bumpy” validation surface.
Greedy approaches typically improve the objective at each step, but that can lead to a dead end. Once a greedy method reaches a local optimum, no small change improves the score, even if much better solutions exist elsewhere. Simulated annealing tackles this by allowing controlled randomness: it sometimes takes a step that looks worse now to potentially reach a better region later.
How Simulated Annealing Works
Simulated annealing is inspired by metallurgy, where materials are heated and slowly cooled to settle into a low-energy structure. In optimisation terms:
- A state is a candidate solution.
- An energy (or cost) is the objective value to minimise.
- A neighbour is a small change to the current solution.
- Temperature (T) controls how willing the algorithm is to accept worse moves.
The algorithm iterates as follows:
- Start with an initial solution SSS and compute its cost E(S)E(S)E(S).
- Propose a neighbouring solution S′S’S′ (a small modification of SSS).
- Compute the change in cost: ΔE=E(S′)−E(S)\Delta E = E(S’) – E(S)ΔE=E(S′)−E(S).
- If ΔE≤0\Delta E \le 0ΔE≤0, accept S′S’S′ (it is better or equal).
- If ΔE>0\Delta E > 0ΔE>0, accept S′S’S′ with probability:
- p=exp?(−ΔET)p = \exp\left(-\frac{\Delta E}{T}\right)p=exp(−TΔE)
- Reduce the temperature according to a cooling schedule and repeat.
This acceptance rule is the core idea. When TTT is high, even worse solutions may be accepted, enabling exploration. As TTT decreases, the probability of accepting worse moves drops, turning the search into a more cautious, improvement-focused process.
Cooling Schedules and Key Design Choices
Simulated annealing is simple to describe, but performance depends heavily on a few practical decisions.
1) Neighbourhood Design
The “neighbour” step must be meaningful. If the change is too small, the search moves slowly. If it is too large, the search becomes unstable. Examples:
- In feature selection, a neighbour might flip one feature (add/remove).
- In clustering, it might reassign one point to a different cluster.
- In route optimisation, it might swap two stops.
2) Cooling Schedule
Cooling defines how temperature decreases. Common options include:
- Geometric cooling: T←αTT \leftarrow \alpha TT←αT where α\alphaα is often 0.90–0.99.
- Linear cooling: T←T−βT \leftarrow T – \betaT←T−β.
- Adaptive cooling: reduce temperature based on acceptance rate.
Cooling too fast can cause premature convergence (getting stuck). Cooling too slowly may be accurate but costly in runtime. A practical approach is to start with a temperature that accepts a reasonable number of worse moves (for exploration), then gradually tighten.
3) Stopping Criteria
Typical stopping rules include:
- Reaching a minimum temperature threshold
- No improvement after N iterations
- A fixed time or iteration budget
In applied analytics, fixed budgets are common because compute time matters.
Where Simulated Annealing Helps in Data Science
Simulated annealing is especially useful when gradients are unavailable or unreliable, and when the search space is large and discrete.
Hyperparameter Tuning Beyond Grid Search
Grid search is expensive when there are many parameters. Random search is often better, but it can still miss structured improvements. Simulated annealing can explore hyperparameter settings while occasionally stepping away from a good configuration to escape a local “sweet spot” and find a better one later. This is one reason data science classes in Bangalore may position SA as a conceptual tool before introducing Bayesian optimisation.
Feature Selection with a Clear Trade-off
Suppose you want a model that is accurate but uses fewer features for interpretability and cost. You can define an objective such as:
cost=(1−validation score)+λ×(number of features)\text{cost} = (1 – \text{validation score}) + \lambda \times (\text{number of features})cost=(1−validation score)+λ×(number of features)
Simulated annealing can explore subsets efficiently by flipping features on and off, accepting occasional short-term drops in score to discover better combinations.
Combinatorial Optimisation in Operations
Even outside modelling, analysts face scheduling and allocation problems. For instance, assigning instructors to sessions, minimising travel time, or balancing workloads can be formulated as optimisation tasks with constraints. Simulated annealing often performs well as a flexible baseline when exact methods are too slow.
Conclusion
Simulated annealing is a practical global optimisation method built on a simple but powerful idea: accept occasional worse moves early, then gradually become more selective. Its effectiveness depends on good neighbourhood moves, a sensible cooling schedule, and clear stopping rules. While it does not guarantee the absolute best solution in every run, it often finds high-quality solutions in complex, non-convex, or discrete search spaces where greedy methods fail. For many learners in data science classes in Bangalore, simulated annealing serves as a valuable mental model for balancing exploration and exploitation,an idea that appears repeatedly across modern optimisation and machine learning.
