Every machine learning model comes with two kinds of settings: the ones it learns on its own from data, and the ones you have to set before training even starts. That second group — learning rates, tree depths, regularization strengths, batch sizes — are hyperparameters, and choosing them well is often the difference between a model that’s mediocre and one that’s genuinely useful. The problem? The number of possible combinations explodes fast, and brute-forcing all of them is rarely an option.
That’s where tuning methods come in. The three you’ll run into most often are grid search, random search, and Bayesian search, and each one makes a different trade-off between thoroughness, speed, and compute cost. Get the choice right and you’ll find strong settings in a fraction of the time. Get it wrong and you’ll spend a weekend watching progress bars crawl.
The following sections cover what each method actually does under the hood, how they compare head-to-head, when to reach for one over the others, and the practical habits that separate a tuning run that works from one that just burns budget.
What Hyperparameters Actually Are
A quick grounding before the comparisons. Hyperparameters control how a model learns, not what it learns:
- Capacity controls — how complex the model is allowed to get, such as the number of layers, tree depth, or number of clusters.
- Learning dynamics — how aggressively the model updates, such as learning rate, momentum, or decay schedules.
- Regularization — how much the model is penalized for complexity, which directly influences overfitting.
- Training logistics — batch size, number of training epochs, and similar knobs.
These settings interact. A learning rate that’s perfect at one batch size can be disastrous at another. That interaction is exactly why exhaustive search becomes painful so quickly.
Grid Search: Exhaustive and Predictable
Grid search is the most intuitive approach. You define a list of candidate values for each hyperparameter, then train and evaluate a model for every possible combination.
If you’re tuning two hyperparameters with five values each, you get 25 training runs. Add a third with five values and you’re at 125. A fourth pushes you past 600. This is the combinatorial explosion, and it’s the core weakness of the method.
Where grid search wins
- Small search spaces with two or three hyperparameters.
- Categorical settings with a handful of sensible options.
- Cases where you need a fully reproducible, easy-to-explain sweep.
- Runs that parallelize cleanly, since every trial is independent.
Where it falls apart
Continuous hyperparameters are the killer. If a learning rate could reasonably be anything from 0.0001 to 0.1, a grid forces you to pick a few fixed points — and the best value is probably sitting between two of them. Worse, most of your compute goes to hyperparameters that barely matter, while the few that dominate the result get the same handful of values as everything else.
Random Search: Surprisingly Strong
Random search throws out the grid entirely. Instead of fixed values, you define a distribution for each hyperparameter — uniform, log-uniform, or a categorical list — and sample combinations at random for a fixed number of trials.
It sounds sloppy. In practice, it’s often better than grid search, and the reason is worth internalizing: most hyperparameters have a small effective influence. If only two of your eight hyperparameters meaningfully change performance, grid search wastes enormous effort varying the other six, while random search gives every trial a fresh shot at landing on a good value for the two that count.
Why random search earns its place
- Better anytime behavior — you can stop whenever your budget runs out and keep the best result so far.
- Handles continuous spaces properly — sampling from a log-uniform range covers orders of magnitude naturally.
- Scales to many hyperparameters — it degrades gracefully instead of exponentially.
- Trivially parallel — no coordination needed between trials.
The trade-off is that random search never learns from what it has already tried. Trial 90 is just as blind as trial one.
Bayesian Search: Learning From Every Trial
Bayesian optimization takes the opposite philosophical stance. Rather than ignoring results, it uses every completed trial to build a surrogate model — a cheap approximation of how your model’s performance responds to hyperparameter settings.
The loop looks like this:
- Evaluate a few random configurations to get initial data.
- Fit the surrogate model to all results so far.
- Use an acquisition function to pick the next configuration — balancing exploration (testing uncertain regions) against exploitation (refining near known good points).
- Train, measure, update the surrogate, repeat.
Because each decision uses everything learned so far, Bayesian search typically reaches strong results in noticeably fewer trials than random search — which matters enormously when a single training run costs hours or real money.
The honest downsides
- Sequential by nature — the next trial depends on the last, so parallelizing is harder.
- Overhead per step — fitting the surrogate and optimizing the acquisition function takes time, which is wasted if trials are cheap.
- Struggles in very high dimensions — surrogate quality degrades as the number of hyperparameters climbs.
- Results can be noisy — if run-to-run variance is high, the surrogate chases noise.
Several practical variants ease these limits, including tree-based surrogates that handle mixed categorical and continuous spaces, bandit-style early stopping that kills weak trials partway through, and evolutionary schemes that adapt hyperparameters during training rather than before it.
Head-to-Head Comparison
- Grid search — exhaustive, deterministic, simple. Best for tiny spaces and categorical options. Worst for continuous or high-dimensional spaces.
- Random search — budget-driven, highly parallel, embarrassingly simple. Best for wide spaces and limited time. Worst when every trial is extremely expensive.
- Bayesian search — adaptive, sample-efficient, more complex. Best when trials are costly and the budget is small. Worst when trials are cheap or the space is enormous.
Practical Habits That Make Tuning Work
- Use log-uniform ranges for scale-sensitive values. Learning rates and regularization strengths should be sampled across orders of magnitude, not linearly.
- Never tune against your test set. Tune on a validation split, then touch the test set once. Tuning against test data is the fastest way to build a model that looks brilliant and performs badly.
- Start coarse, then refine. Run a wide random sweep to locate promising regions, then narrow the ranges and apply a smarter method.
- Set a budget before you start. Decide how many trials or how many hours you’re willing to spend, then respect it.
- Track everything. Log every configuration and score. Patterns across failed trials are often more informative than the single best result.
- Re-run top candidates with different random seeds. If a configuration only wins under one seed, it didn’t really win.
- Watch for diminishing returns. Tuning typically delivers a big early jump and then flattens. Knowing when to stop is a skill.
How to Choose in Practice
Most strong workflows don’t pick one method — they chain them. A common pattern: run a random search with a generous range to map the landscape cheaply, shrink the ranges around the best-performing regions, then switch to Bayesian optimization for the final push where every trial counts.
If your space is genuinely small and discrete, grid search is still perfectly fine and easy to reason about. If it’s wide, continuous, and your budget is the binding constraint, random search will usually beat a grid. If each training run is expensive enough that you wince when you hit go, Bayesian search is worth the extra machinery.
The bigger point is that tuning is a budget allocation problem. You’re deciding where to spend compute in exchange for performance, and the right method is simply the one that spends it most efficiently for your specific setup.
The Bottom Line
Grid search is thorough but wasteful, random search is simple but surprisingly effective, and Bayesian search is smart but demands more from your pipeline. Understanding why each behaves the way it does turns hyperparameter tuning from a guessing game into a deliberate engineering decision — one that regularly buys you meaningful accuracy for a fraction of the compute.
If you want more breakdowns like this, covering how models actually learn, where modern AI tooling is heading, and which techniques are worth your time, keep exploring the rest of what we publish. There’s always another layer of the stack worth understanding, and we’re here to make it click.