How the scores are made.
- DWD stations in Germany (30)
- National services: Austria 31, Netherlands 27, Denmark 13, Switzerland 5, Finland 1 (77)
- NOAA hourly reports, 11 further countries (61)
- Cabauw tower, Netherlands (1)
What the numbers mean.
- Skill is the share of a reference forecast's error that a model removes. A perfect forecast would score 100%, 0% means the same error as the reference, and a negative score means a higher error.
- Persistence assumes the weather from the lead time earlier repeats. Climatology is the average for that date and time of day, and it becomes the harder reference after the first few days.
- Each bar is a 95% interval. A table calls AIFS lower or higher only when the paired difference passes the correction for multiple testing and the sample is long enough. Otherwise the models cannot be separated. Green and red, with the marks ▼ and ▲, appear only on such tested differences. Any other difference shows its sign only: in plain ink, or in violet (lower) and yellow-brown (higher).
- Raw forecasts are scored as issued. Bias-corrected forecasts first have their average error from the previous 30 days removed, using only days already observed when the forecast was issued. Both references get the same correction.
- RMSE rewards smooth forecasts, and it says little about extremes. Activity measures the smoothing: the spread of a forecast's departures from climatology divided by that of the measurements, with each station centred on its own mean at each valid hour first, so that neither differences between the station means nor an error in the mean daily cycle counts as variability. Three days ahead at the German stations, AIFS 2's wind forecasts carry 89% of the measured variability around climatology, against 94% to 98% for the physics models; its temperature forecasts carry 97% of the measured variability around climatology, against 94% for the physics models. Activity leaves out a model's error in the mean difference between day and night. Over these days the measured wind at the German stations was on average 1.49 m/s higher at 12:00 than at 00:00 UTC; three days ahead the forecasts made that difference +0.81 m/s (AIFS), +0.91 m/s (IFS) and +1.39 m/s (GFS). The tables carry MAE as well.
- Only models with public forecast archives can appear. The models differ in resolution, so a station sits closer to some grid cells than to others.
- Both AIFS versions were trained on ERA5 for 1979 to 2022 and then tuned on IFS analyses: AIFS 1.1 on those of 2016 to 2022, AIFS 2 on those of 2018 to 2024 and on test analyses of IFS Cycle 50r1 (ECMWF's model cards for AIFS Single 1.1 and 2.0). Every forecast scored here was issued after those years. AIFS starts from the IFS analysis, as IFS does. IFS itself changed to Cycle 50r1 on 12 May 2026, the day AIFS 2 came in, so the two periods differ in IFS as well as in AIFS.
What this sample cannot show
- Skill across a full year. Neither AIFS version has one yet, so these numbers describe only the seasons scored.
- Whether one AIFS version has lower error than the other, because they cover different seasons.
- Skill in storms, ramps or other extremes. The scores average over every day.
- Skill outside the countries scored here, or for global weather. Stations are weighted equally: the German result covers 30 stations, 9 of them in Bavaria.
- Skill at wind or solar farms. These are measurements at weather stations, plus one tower.
- Scores on final measurements. DWD's observations from January 2026 onwards are provisional, so every AIFS 2 score rests on data that DWD has not yet quality-checked.
- How AIFS compares with IFS at its native 9 km or with regional models such as ICON-D2, which forecasters use for the first days. The physics models here are the global files as openly distributed, and their publishers keep no open archive of the finer ones.
Method, sources and credits.
Stations
30 DWD stations, frozen before any forecast was downloaded. To qualify, a station had to report both wind and solar, stand no higher than 800 m, carry its anemometer 8 to 12 m above ground, and be at least 97% complete over the scoring period. It also had to lie within 100 m of every model's grid-cell height and 30 km of its grid point. Offshore platforms were excluded. One station was kept per square of 0.5° by 0.75°: the most complete one with a usable climatology.
The height test was run at selection with the grid heights Open-Meteo reported. Recomputed from each model's own surface height for AIFS, IFS and GFS: at the nearest land point the largest gap is 95 m and no station exceeds 100 m; under interpolation it is 137 m and three stations exceed 100 m for at least one model. The station list was not changed. Temperature is moved to the station's height before scoring, which removes most of the effect of such a gap but not all of it.
The 30 stations, north to south
| Station | State | Height | Lat, lon |
|---|---|---|---|
| Sankt Peter-Ording | Schleswig-Holstein | 5 m | 54.33, 8.60 |
| Dörnick | Schleswig-Holstein | 27 m | 54.17, 10.35 |
| Ueckermünde | Mecklenburg-Vorpommern | 1 m | 53.74, 14.07 |
| Norderney | Niedersachsen | 12 m | 53.71, 7.15 |
| Hamburg-Fuhlsbüttel | Hamburg | 11 m | 53.63, 9.99 |
| Bremen | Bremen | 4 m | 53.05, 8.80 |
| Diepholz | Niedersachsen | 38 m | 52.59, 8.34 |
| Braunschweig | Niedersachsen | 81 m | 52.29, 10.45 |
| Lindenberg | Brandenburg | 98 m | 52.21, 14.12 |
| Lügde-Paenbruch | Nordrhein-Westfalen | 258 m | 51.87, 9.27 |
| Leinefelde | Thüringen | 356 m | 51.39, 10.31 |
| Görlitz | Sachsen | 239 m | 51.16, 14.95 |
| Dresden-Klotzsche | Sachsen | 228 m | 51.13, 13.75 |
| Köln/Bonn | Nordrhein-Westfalen | 91 m | 50.86, 7.16 |
| Aachen-Orsbach | Nordrhein-Westfalen | 231 m | 50.80, 6.02 |
| Gießen/Wettenberg | Hessen | 203 m | 50.60, 8.64 |
| Nürburg-Barweiler | Rheinland-Pfalz | 485 m | 50.36, 6.87 |
| Lautertal-Oberlauter | Bayern | 344 m | 50.31, 10.97 |
| Würzburg | Bayern | 268 m | 49.77, 9.96 |
| Mannheim | Baden-Württemberg | 98 m | 49.51, 8.56 |
| Nürnberg | Bayern | 314 m | 49.50, 11.05 |
| Waldmünchen | Bayern | 499 m | 49.39, 12.68 |
| Saarbrücken-Ensheim | Saarland | 319 m | 49.21, 7.11 |
| Stuttgart (Schnarrenberg) | Baden-Württemberg | 314 m | 48.83, 9.20 |
| Fürstenzell | Bayern | 476 m | 48.55, 13.35 |
| Augsburg | Bayern | 462 m | 48.43, 10.94 |
| Weihenstephan-Dürnast | Bayern | 477 m | 48.40, 11.69 |
| Mühldorf | Bayern | 406 m | 48.28, 12.50 |
| Chieming | Bayern | 551 m | 47.88, 12.54 |
| Konstanz | Baden-Württemberg | 428 m | 47.70, 9.13 |
Forecasts and sample
AIFS, IFS and GFS forecasts at leads of 1, 3, 5, 7 and 10 days are decoded from the models' original files: ECMWF open data and NOAA's GFS, all at 0.25°. ICON came from the Open-Meteo Previous Runs archive, as the publisher keeps no open archive of past runs, and is scored in a set of its own that ends on 7 November 2026. Every model decoded here is read the same way: by bilinear interpolation over the four grid points around the station, using land points only where any of the four is land. Each model's temperature refers to its own surface height, interpolated the same way; for the re-run AI models, which share AIFS's grid, that is ERA5's, on which they were trained. AIGFS publishes no surface height, so it takes GFS's, since it starts from the GFS analysis on the same grid. ICON is read at the cell Open-Meteo assigned. Wind is scored at 00:00 and 12:00 UTC against the centred hour of measurements, because at those times every model's value comes from the run issued exactly that many days earlier. Solar is daily energy in kWh/m², summed from four 6-hour windows, each taken from the run issued that many days before the window starts. Open-Meteo assembles the day differently, so on solar AIFS is compared only with models taken from the original files, and ICON is scored without a comparison. Temperature is scored at the same two times as wind, after moving each forecast to the station's height with a lapse rate of 6.5 °C per km. ICON at day 5 is left out of solar, because part of its day comes from a run six hours older. IFS has no solar at days 7 and 10, because its 06:00 and 18:00 UTC runs stop at day 6. ICON's archived forecasts stop before day 7, so it has no day 7 and no day 10. Open-Meteo's archive repeats one run under more than one lead label for ICON's 2 m temperature on 6 to 10 April 2026; those days are left out at the leads affected. Each version, variable and track uses one sample, the same at every lead. A station-time counts when the measurement, the climatology and every model's forecast exist at every lead. A date counts when at least 80% of its possible station-times can be scored, and every model and climatology is then scored on exactly the same pairs. Persistence needs an earlier measurement too, so persistence and the skill against it are scored on the station-times of the sample where it exists at every lead; a chart or table of skill against persistence states its own days where they are fewer. Changeover days between versions are excluded.
Baselines and correction
Persistence is the measurement from the lead time earlier. For solar it is the energy of the day before that, because a day's energy is complete only at 24:00 UTC, after the forecasts of that day were issued. Climatology is the average for that date and time of day over 2015 to 2024, smoothed over 15 days and needing at least 4 years. 2 stations with a sensor or position change in those years use only the years after it. Bias correction removes each series' mean error over the 30 days before issue, per station and time of day, from at least 20 pairs already observed at issue and never across a version change. Models and baselines are corrected alike. Skill is 1 minus the ratio of RMSEs.
Intervals and verdicts
Intervals come from a stationary block bootstrap over forecast dates, with all stations kept together on each date and 20,000 resamples. The mean block is 10 days: at least 10 days, and longer if a model's error or a paired difference stays correlated for longer. Blocks follow the calendar, so a gap in the data stays a gap. Slow seasonal drift in the error is removed before measuring that correlation. The effective sample is about 28 to 93 days per comparison, and about 43 to 247 for the paired differences themselves.
The samples at the German stations hold 14.3 to 24.8 blocks. A percentile interval from so few blocks is too narrow, so each interval is widened about its point estimate by a t factor for the number of blocks. For a paired difference, the wider of that interval and one that allows for serial correlation in the daily differences is used, with the correlation measured on the series as it is, seasonal drift left in. The p-value is taken from the same interval, so an interval excludes zero exactly when p is below 0.05. Event scores carry intervals from the first 5,000 of the same resamples.
Comparisons are paired differences in RMSE, AIFS minus each other model. Only comparisons of AIFS with another model are tested; no other pair is, so the page gives no verdict between, say, IFS and GFS. A difference counts only if its 95% interval excludes zero, its Holm-adjusted p-value is below 0.01, the sample holds at least 15 blocks, and the difference has the same sign in the first and second half of the period. It must also hold when the models are read at the nearest land point instead of by interpolation; otherwise it is shown as depending on how the station is read. A temperature verdict must also hold when the height correction of 6.5 K/km, which is a convention, is replaced by none or by 4 K/km; otherwise it is shown as depending on the height correction. The intervals shown are not adjusted for the number of comparisons; the Holm correction applies to the p-values.
The verdicts are tested again at every weekly update, on a sample one week longer, and a test repeated like that gives a false verdict more often than its level says. In a simulation of 3,000 comparisons with no real difference, tested with this method every week from day 150 to day 365, 16% got a verdict at some update when the level was 0.05, and 4.4% at 0.01. In families of 15 independent comparisons with the Holm correction, at least one comparison got a verdict at some update in 27% of 200 families at 0.05 and in 10% at 0.01; comparisons in a real family share their days, which makes the correction stricter than that. That is why a verdict needs a p-value below 0.01; until 16 November 2026 the level was 0.05. An interval can therefore exclude zero without a verdict. Every verdict is interim until a version covers 365 days and every season: the test on that sample is the one to rely on.
The level holds within one family of comparisons at one update, not across families. At the stations and the tower there are 28 families, one per station set, AIFS version and track, across variables and leads; 18 of them hold enough days to give verdicts. With ICON, a family covers the comparisons with ICON only, which are the ones shown. Against ERA5 there are 6, one per variable and AIFS version, across model sets and leads; 2 of them, for 500 hPa geopotential height, are judged. Against ERA5 no verdict is given at day 1, whatever the test says: the models start from different analyses, and ERA5 is an ECMWF product. Station-level winners are not published.
Robustness checks
- At the 30 German stations, both AIFS versions together: 56 raw comparisons were repeated with each station or month removed in turn. Removing a station changed the sign of 4 differences. Removing a month changed the sign of 3.
- Doubling the block length changed 8 of the 56 verdicts. Using blocks up to 39 days, without removing the seasonal drift, changed 7. Scoring on MAE instead of RMSE changed 4.
- Bias correction lowered AIFS's RMSE, with an interval that excludes zero, in 10 of 30 bias-corrected scores. The CSV also carries skill computed on MSE.
- Scoring wind and temperature at all four valid times, at days 1 and 3 only, gave a lower error for AIFS in 11 of 16 comparisons.
- Height correction: in every station set, raw temperature was scored again with no correction and with 4 K/km in place of 6.5 K/km. All 64 temperature verdicts held under all three.
- Coast: the European set was scored again without the 18 stations that at least one model reads from sea points or from a mix of sea and land points, which leaves 89. 3 of the 112 verdicts differed: AIFS 1.1 against IFS on 10 m wind at day 10, from cannot be separated to AIFS lower error, AIFS 1.1 against IFS on daily solar energy at day 5, from AIFS lower error to cannot be separated and AIFS 1.1 against GFS on daily solar energy (bias-corrected) at day 7, from cannot be separated to AIFS lower error.
- Calm-wind screen: at stations in Denmark, a wind value under 0.1 m/s is left out when the median of at least 8 other stations of the country is 3 m/s or more, a sign of a stuck anemometer. This blanked 280 values at 12 stations, between 1 February 2015 and 20 October 2026, the climatology years included. The rule reads observations only.
- In every station set, each verdict was also scored with the models read at the nearest of the four surrounding grid points that the model counts as land. One of 215 verdicts did not hold and is shown as depending on how the station is read: Europe with ICON, AIFS 1.1 against ICON on 10 m wind at day 1.
Cabauw uses the same code on one site. Its climatology covers the validated months between 2015 and 2024.
Changes to the method
The rules were not all fixed before the first results were seen. Each change that altered a published figure or a verdict is listed here with its date, newest first, so that a figure which moved without new data can be traced. The full record, with the numbers before and after, is the file CHANGELOG.md in the repository.
- 16 November 2026
- A verdict needs a Holm-adjusted p-value below 0.01; it was 0.05. The verdicts are tested again at every weekly update, and a test repeated like that gives a false verdict more often than its level says. At the stations and the tower 214 of 612 comparisons carried a verdict afterwards, where 247 had; against ERA5, 20 of the 72 that are judged, where 24 had.
- Persistence is needed only for persistence's own scores and for the skill against it. Before, a station-time counted only if persistence could be scored at every lead, so one missing measurement removed six station-times from every score. For AIFS 2, temperature at the Cabauw tower went from 166 to 172 days, and the NOAA stations from 61 days to 94 for wind and 101 for temperature.
- Activity is centred on the mean of each station and valid hour, and of each grid cell and valid hour against ERA5; it was centred per station and per grid cell. An error in the mean difference between 00:00 and 12:00 UTC no longer counts as variability. At the German stations the values moved by up to 0.04: three days ahead, AIFS 2's wind activity went from 0.91 to 0.89.
- Solar persistence is the energy of the day before the issue day. It was the issue day's own total, which is complete only after that day's forecasts were issued. At the German stations AIFS 2's solar skill against persistence one day ahead went from 63% to 66%.
- 11 November 2026
- Models decoded from the original files are read by interpolation over the four grid points around a station, and the nearest land point is the check a verdict must also pass. Before, it was the other way round.
- Day 10 was added as a lead time. Every sample starts three days later as a result, so earlier figures moved slightly.
- Intervals and p-values were widened for the small number of blocks, after a simulation in which the intervals labelled 95% covered 87 to 91%. Point estimates did not move; some verdicts were withdrawn.
- No verdict is given at day 1 for 500 hPa height against ERA5. A temperature verdict must hold with no height correction and with 4 K/km as well as with 6.5 K/km.
- Activity is centred per station. The climatology window crosses New Year. Temperature at a valid time is the mean of seven 10-minute values centred on the hour.
- The NOAA set went from 68 to 61 stations and gained quality control. An event score needs the event on at least 10 dates.
- ICON was frozen at 7 November 2026 in a set of its own, the 9 km IFS was retired, and stations in five more countries were pooled with the German ones into the European set.
- 10 November 2026
- AIFS, IFS and GFS are read from the models' original files and no longer from Open-Meteo's archive, after a cross-check found archived runs that did not match those files.
- A verdict must also hold under a second reading of the grid. A sample needs 15 blocks, not 8. Days on which Open-Meteo repeats one run under two lead times are left out.
- 9 November 2026
- Before any result was shown: wind was limited to 00:00 and 12:00 UTC, one sample was used across all lead times, and the baselines were bias-corrected alongside the models.
About
MeteoSignal is run by Dr. Iman Soltanzadeh, a Data & AI Scientist with nearly 20 years of experience in weather, climate and energy data. He holds a PhD in meteorology from the University of Tehran.
From 2011 to 2013 he was a postdoctoral researcher at the University of Canterbury in New Zealand. There he worked on the climate of the McMurdo Dry Valleys in Antarctica, in particular their winds and energy fluxes, and on the transport of snow over Antarctic sea ice. He also took part in a project that modelled the weather and climate of vineyard regions in New Zealand and South Africa at high resolution, to help growers adapt to climate variability, and was first author of its assessment of the WRF model over the Stellenbosch wine district (2017).
He then spent nine years at MetService, New Zealand's national weather service, working on numerical weather prediction and forecast post-processing before leading its research and development. In that role he was a core expert on the Expert Team on Aeronautical Meteorological Science of the World Meteorological Organization (WMO). He now leads AI and data science projects for Europe's energy networks, with a focus on AI weather models and renewable energy forecasting.
MeteoSignal is a personal, non-commercial project, separate from his employment. It has no funding and no advertising. More about the author is on soltanzadeh.com, and his publications are on Google Scholar. Errors and questions are welcome through the contact form.
Licence
Results and text on this site are licensed under Creative Commons Attribution 4.0. You may copy, share and adapt them if you give credit: "MeteoSignal, Iman Soltanzadeh, meteosignal.com, CC BY 4.0". The code is under the MIT licence. The source data keep their own terms, listed under Sources.
How to cite
Soltanzadeh, I. (2026). MeteoSignal: verification of open weather forecasts over Europe. Data version 16 November 2026, run 20261116T194902Z. https://meteosignal.com/ Please name the data version, because the results change with each update.
Credits
Analysis, code and site by Iman Soltanzadeh.
MeteoSignal is independent. It is not affiliated with or endorsed by any organisation whose models or data appear here. All source data were processed to produce these scores.
Sources
Forecasts
- AIFS and IFS forecasts at the stations and for the area section: © 2025–2026 European Centre for Medium-Range Weather Forecasts (ECMWF), www.ecmwf.int, published under CC BY 4.0; decoded and reduced to station and area values of wind speed, temperature and solar energy. ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use. AIFS is described in Lang et al. 2024, arXiv:2406.01465.
- GFS forecasts: NOAA/NCEP, from the NOAA Open Data Dissemination archive on AWS, public domain. NOAA does not endorse this site.
- AIGFS forecasts: NOAA/NCEP, from the NOAA Open Data Dissemination archive on AWS, public domain. NOAA does not endorse this site.
- GraphCast, FourCastNet, Pangu-Weather and Aurora, re-run by NOAA: AI Weather Prediction Model Reforecasts, CIRA and NOAA GSL (Radford et al. 2025, doi:10.1175/BAMS-D-24-0057.1). The models behind those re-runs: GraphCast (Lam et al. 2023, Google DeepMind), FourCastNet v2-small (Bonev et al. 2023, NVIDIA), Pangu-Weather (Bi et al. 2023, Huawei) and Aurora (Bodnar et al. 2025, Microsoft). The scores here describe NOAA's re-runs, not the developers' own systems.
- ICON in the closed set: Weather data by Open-Meteo.com, Previous Runs API, CC BY 4.0, for ICON (DWD), up to 7 November 2026.
- ICON and AICON from DWD's own files, kept since 8 November 2026 for a later series: Deutscher Wetterdienst, open data server, CC BY 4.0. They are not in any score yet.
Measurements
- Germany: Source: Deutscher Wetterdienst, 10-minute station observations of wind, of solar and sunshine, and of air temperature for Germany, Version v24.03, CC BY 4.0. Observations from January 2026 onwards are provisional.
- Austria, Denmark, the Netherlands, Switzerland and Finland: GeoSphere Austria, CC BY 4.0. Danish Meteorological Institute (DMI), CC BY 4.0; its 10-minute records were reduced to hourly wind and temperature and daily solar energy. Royal Netherlands Meteorological Institute (KNMI), 10-minute in-situ observations and hourly station data, CC BY 4.0; reduced to hourly wind and temperature and daily solar energy. Source: MeteoSwiss, CC BY 4.0. Finnish Meteorological Institute, CC BY 4.0.
- Further countries: NOAA National Centers for Environmental Information, Global Historical Climatology Network hourly (GHCNh), from the NOAA Open Data Dissemination archive on AWS, public domain. NOAA does not endorse this site.
- Cabauw tower: KNMI, CESAR Cabauw tower profiles, validated (cesar_tower_meteo_lb1_t10, v1.3 and v1.2) and unvalidated (cesar_tower_meteo_la1_t10, v1.3), CC BY 4.0.
Reanalysis and climatology
- ERA5: Contains modified Copernicus Climate Change Service data (ERA5 reanalysis, the latest months preliminary), CC BY 4.0. Neither the European Commission nor ECMWF is responsible for any use made of this information. Described in Hersbach et al. 2020, doi:10.1002/qj.3803. Climatology for the area scores: WeatherBench 2, 1990 to 2019, itself derived from ERA5 (Rasp et al. 2024, doi:10.1029/2023MS004019).
Maps
- Country outlines on the maps: Natural Earth, public domain.
Method references
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
- Politis, D. N. and Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. doi:10.1080/01621459.1994.10476870
- Jolliffe, I. T. and Stephenson, D. B., eds. (2012). Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd ed. Wiley. doi:10.1002/9781119960003
Download
The scores and the paired comparisons are on the data access page, as four CSV files with their columns explained, and on a summary sheet to print.