The boosting ladder

We have a coarse picture and want the fine one

Weather models are run on grids. A coarse grid with cells of 27 km is cheap; a fine grid with points 3 km apart shows much more detail but costs far more. Downscaling means estimating, from the coarse run alone, what the fine run would show.

This page belongs to a project that downscales real weather forecasts (“the real project” below). There, both grids belong to one weather model, WRF (Weather Research and Forecasting model): a coarse grid (“d01”) and a fine grid nested inside it (“d03”). Each run is one 72-hour forecast started from one initial state, with six variables: rain rate, surface pressure, two wind components, temperature and humidity.

Here the example is synthetic: one made-up quantity without units, along a line of 864 km that wraps around (the right end joins the left end). Its fine truth has three parts: big waves the coarse grid also sees, a response that comes hours after the waves that drive it, and small wiggles finer than any coarse cell. The lower plot follows one point: the response (blue) peaks hours after the waves that drive it (orange), while the coarse input (gray) has no delay.

Fix what is still wrong, a little at a time

Gradient boosting (J. H. Friedman, “Greedy function approximation: a gradient boosting machine”, Annals of Statistics 29, 1189–1232, 2001) builds a fit by adding many small corrections. Each round computes what the current fit still gets wrong, fits a very simple model to that, and adds a fraction ν (the Greek letter nu; the step size) of it. “Gradient” refers to fitting each piece to the direction that lowers the error fastest; for squared error that direction is simply what is still wrong.

F_m is the fit after round m, and y here are the noisy data points of the toy curve. The small model is a regression tree with two levels of yes/no splits.

This step uses a separate toy, a noisy curve, because the effect is easiest to see there. The lower plot reproduces the paper's main result: smaller steps need more rounds and reach a lower error on data the fit has not seen, down to about ν = 0.1; smaller steps gain nothing more. Every curve turns up again at the end, so the number of rounds must be chosen on data not used for fitting.

The ladder: a first guess, then corrections of corrections

Each row is one rung. Rung 0 is a first guess of the truth. Rung 1 estimates the error that rung 0 leaves. Rung 2 estimates the error that rungs 0 and 1 leave together, and so on. A hat marks an estimate: ŷ estimates the truth y, and ê estimates an error. In the diagram a superscript is the rung and a subscript the hour: ê¹₃ is rung 1's estimate at hour 3. The final estimate is the sum of the rows: rung 0 plus the corrections of rungs 1, 2, …. The weights A and b of rung 0 are explained in step 4, validation error (the error on runs held out from fitting) in step 4, the stopping rule in step 8 and the random rung in step 9.

Two kinds of arrows matter. Red curved arrows pass down what is still wrong: the difference between the truth and the estimate so far, which the next rung learns to predict. The truth is needed only while fitting; in use, every rung works from the coarse input (gray arrows; later rungs also read it) and the rungs above it. Green arrows carry each rung's estimate forward in time: the error at one hour helps predict the error at the next. This carrying forward is called feedback.

A straight-line first guess

The simplest map from coarse to fine is linear: each fine point is a weighted sum of the coarse cells (the weights form the matrix A) plus a constant (b). The weights come from how coarse and fine values vary together, their covariances C, in the 240 training runs, the runs used for fitting. In the real project a more elaborate statistical estimate of the same linear kind, built from a large collection of past cases, plays this role.

C_yu is the covariance between fine and coarse values, C_uu the covariance among the coarse values (its inverse removes their overlap), and the means are taken over all hours of the training runs.

Errors on this page are RMSE, the root-mean-square error: the square root of the average squared difference from the truth, over all points, hours and runs. They are measured on 60 validation runs, held out from fitting and used for every choice. Rung 0 is better than stretching the coarse cells onto the fine grid by linear interpolation, but the gap shaded in red is not random: it follows the lagged response and the small wiggles.

Learn the error, and let it carry from hour to hour

The error of the first guess has a slow part, which persists: if the guess is too low now, it is probably too low next hour too. It also has a fast part, the wiggles, which nobody can predict. Rung 1 goes after the slow part with a model that keeps a memory of its own last estimate:

z_t is the coarse input near the point at hour t, β its fitted weights, and φ (phi), the feedback gain, is the fraction of last hour's estimate carried into this hour. The plot shows the validation RMSE hour by hour. A second straight-line model of the same input, without memory, adds nothing. A model fitted with the true past error in place of its own leans on information it will never have when used. The kept model is fitted the way it is used, running forward on its own estimates (a rollout), and the data choose φ = 0.90; the model fitted with the true past error ends up with a gain of 0.87. At hour 0 each rung starts as if its inputs had been constant before, and the lagged response starts in balance with the waves, so the solid lines start lower at hour 0 and rise over the first few hours.

Feedback must forget, or errors grow without limit

Feedback has a cost: a mistake at one hour is carried to the next. With gain φ it shrinks to φ^h of itself after h hours. If φ is below 1 the mistake fades and the total error stays bounded:

Here ε is the largest new error any single hour adds. If the gain were above 1, the same mistake would grow every hour. The plot adds one unit of error at hour 10 and follows it. The red line is a what-if with gain 1.05; the two fitted models both have gains below 1.

Correct the correction, again and again

A straight-line model cannot capture a response that bends. So each further rung is a small curved model of what all rungs before it still get wrong: 32 fixed curved building blocks, each max(0, weighted sum) with weights drawn once at random and never changed, applied to the local coarse input and the current estimate and combined with fitted weights. Each rung has its own feedback gain φ, chosen by the data. The estimate after rung n is rung 0 plus rung 1 plus ν times the sum of rungs 2 to n; the map uses ν = 1, the value chosen in step 8.

Each rung is fitted out-of-fold: the training runs are split into five groups, and the correction for each group comes from a model fitted on the other four. The next rung therefore learns from errors that were not themselves fitted, as on new data.

The map shows one validation run: distance along the line across, hours upward, colour = what is still wrong. The wide red and blue bands are the lagged response. Watch them fade as rungs are added. What remains is fine-grained noise: the wiggles, whose strength follows the big waves, so they still trace a slanted path, but whose sign is random.

Corrections shrink only with small steps; validation data say when to stop

A Cauchy sequence is a sequence whose later terms get arbitrarily close to one another, so that it settles on a limit; corrections that shrink geometrically give one. Do the rungs' corrections behave like that, and what is the limit? The left plot shows the size (root mean square) of each rung's correction; the right plot shows the validation RMSE as rungs are added, for two step sizes.

Why keep ν = 1 when step 2 favored small steps? On this problem the two step sizes reach the same accuracy, and the rule picks the lowest validation error. Step 2's lesson still shows: only the small step gives corrections that keep shrinking.

What a converging ladder settles on is not the truth itself. It is the part of the truth the coarse input can predict. What is left is noise, and that needs a different kind of rung.

What cannot be predicted is drawn at random, with the right memory

The left-over wiggles have a predictable size (bigger where the wave is high) and a predictable texture, but no predictable position. So the last rung draws them at random several times: noise whose size follows the value of the estimate and whose texture copies the spatial pattern of the left-over error in the training runs. The result is an ensemble: several equally likely fine fields, called its members. The hour-to-hour correlation below measures how alike the small scales are at consecutive hours (1 identical, 0 unrelated).

How the random part changes from hour to hour matters: in an animation, frozen or flickering detail looks wrong, and anything added up over hours, such as accumulated rain, depends on it. The panels play the small scales of one validation run. The truth's wiggles change gradually. Noise frozen in time does not move, noise redrawn every hour flickers, and noise with a memory ρ (rho) behaves like the truth. ρ plays for the random part the role φ plays for a rung: the fraction of last hour's noise kept.

Each rung lowers the error; the random rung adds a spread that matches the error

Sixty test runs were set aside at the start, used for nothing else, and scored once at the end. The bars show CRPS (continuous ranked probability score), a score for ensembles that rewards both accuracy and a spread (how much the members differ) that matches the actual error; lower is better, and for a single, non-random estimate it equals the average absolute error. The lines show RMSE at each of the 73 hourly times, 0 to 72; after the first few hours none of them grows, so the error does not build up. Numbers here differ slightly from earlier steps because these are the test runs, not the validation runs, and because the bars are CRPS while earlier cards are RMSE. Deterministic means without the random rung.

The same ladder on the real weather model

The real work follows the same order, one step each. What changes: six variables instead of one (rain rate, surface pressure, two wind components, temperature, humidity), two-dimensional maps instead of a line, rungs that are small neural networks (flexible fitted functions built from many simple units) reading local patches of the maps and carrying a summary of past hours forward (a state), and an error measure that is less sensitive to a few very large rain errors than squared error is, such as the absolute error or Huber's loss (squared for small errors, absolute for large ones).

One check comes first, with no counterpart in the 1-D example: whether the fine run, averaged back onto the coarse grid, equals the coarse run. If it does, the coarse run tells us at every hour how far the averaged estimate is from it, an error signal available without the fine truth that the ladder could feed back in addition to its own past estimate.

Terms used on this page

Terms are listed in the order they first appear; the number in brackets is the step. Symbols: y fine truth, u coarse input, a hat (^) marks an estimate, t the hour.