Who has the better forecast? It depends...

This week's tornadoes in Oklahoma and Arkansas have gotten a lot of press coverage, as this was the first real tornadic event of the season and there was one fatality in the Tulsa tornado.  The environment for tornadoes, particularly in central Oklahoma, was somewhat marginal, and this contributed to a slow reaction from the National Weather Service in issuing a tornado warning for a storm that produced a brief EF1 tornado in Moore, Oklahoma.  On the other hand, private forecast provider Accuweather issued a tornado warning for its private clients several minutes in advance of the tornado happening.  Accuweather released a "brochure" shortly thereafter touting the success of its forecast in the face of the late warning from the National Weather Service.

There has been a fair bit of debate in the meteorological community following this direct competitive criticism of the NWS by Accuweather.  Several bloggers (such as Jason Samenow at Capital Weather Gang, Dennis Mersereau at The Vane and Mike Smith at Accuweather) have covered this issue in depth from a variety of viewpoints, so I'm not going to do so here.  But this brings up the age-old question of who actually produces the best forecast, something that I've long been interested in.

Let's keep it simple.  We're not going to look at forecasting tornadoes, nor are we going to look at multi-week forecasts of questionable origin.  Instead, let's look at something that any good forecasting service should provide--- the forecast for the next day's high and low temperature.  Easy enough numbers to analyze and understand.

To get a variety of locations, I'm going to follow the WxChallenge weather forecasting competition from this year.  Hundreds of meteorology undergraduates, graduates, and faculty participate in the WxChallenge.  The competition spends two weeks forecasting the next day's high, low, maximum wind speed and total precipitation at a random city somewhere in the US before switching to a new city.  I took a sample of several of the cities used in this year's competition and, for the two week periods where the contest was forecasting for each city, recorded the next-day high and low temperature forecasts from a variety of forecast providers and some model output.  All of these forecasts were taken at 2300 UTC the day before.  You can see this year's schedule of WxChallenge cities here.

To examine the quality of the forecasts, I evaluate skill scores for each city's two-week forecasting period.  A skill score measures your error relative to some baseline for comparison.  We're going to use two baselines here---a climatology forecast (where you would simply forecast the climatological average high and low temperature for tomorrow) and a persistence forecast (where you just forecast that whatever happened today will happen again tomorrow).  A skill score of 1 means you had a perfect forecast---no error.  A skill score of 0 means that you had exactly the same skill as the baseline forecast used for comparison.  A negative skill score means you actually did worse than the baseline.  Any good forecast should be able to consistently beat the climatology and persistence baselines, so we're looking for scores between 0 and 1.

So, without further ado, here's a big graph of the skill scores:

The blue bars represent the skill scores for the low temperature (lighter bar is the skill against persistence, darker bar is the skill against climatology) and the red bars represent the skill scores for high temperature.  Each city has its own plot.  The forecast sources are grouped together and they are sorted from left to right with the forecast source that had the highest average skill score to the left.  By this definition, the first forecast source you see in each row was the "best" forecaster for that particular two week period at that city.

Some background on these forecast sources---WUTWC is Weather Underground/The Weather Channel; ACUWX is Accuweather; NWS is the National Weather Service; HAMWX, FCSTIO, WWO and METGRP are other private forecasting companies; GFS and NAM are the GFS and NAM MOS forecasts (automated model-generated forecasts) and the USL12Z and USL22Z are another automated model-generated forecast.

So what does the plot show us?  Of the six cities looked at, Weather Underground/The Weather Channel (WUTWC) was the best at two of them, the automated USL22Z model was the best at two, and FCSTIO (forecast.io) and Accuweather (ACUWX) were the best at the remaining two.  All of these are either private or automated forecasts.

Some cities were quite challenging--a lot of negative skills for low temperature at Butte, Montana (KBTM) show that in some cases forecasts were too wild and didn't capture local changes in low temperature well.  In fact, the automated forecasts (USLs, NAM and GFS) all beat out the other forecasts at Butte.  It's interesting that for a city with so varied of weather and such a complex forecast as Butte, forecasts that included human meteorologists were unable to have higher skill than any of the automated forecasts...

Where was the National Weather Service in all of this?  Decidedly in the middle of the pack.  The best they did was 4th at Long Beach, and at all cities either ACUWX or WUTWC (or both) had higher average skill.

So what can we learn from this?  Forecast quality depends a lot on when and where you're looking.  In some cases, automatic or raw model forecasts can outperform human forecasters.  It also would be a mistake to think of the NWS as a "gold standard" when it comes to forecasts for next-day high and low temperature---companies like The Weather Company (which owns Weather Underground and The Weather Channel) and Accuweather routinely provide better high and low temperature forecasts.  That being said, this is a very small sample size and there are a ton of other events and variables to forecast, many of which the Weather Service excels at.  It's also impressive how many other companies there are that provide forecasting services (see the plethora of other companies on that chart that I didn't mention) that really aren't very good at all.  They routinely do worse than freely-available model guidance (like the NAM or the GFS) or the NWS forecast.  So choose your forecast provider wisely!

Cyclone Pam, Seattle rain records and the MJO

Last week the small island nation of Vanuatu was devastated by a major tropical cyclone: the Category-5-equivalent Cyclone Pam.  There has been terrible damage to the infrastructure on the islands, with estimates of up to 90% of the buildings in the capital of Port Vila.  Power was lost early on in the storm and is just now being restored in some places.  We can actually see this looking at the weather observations from Port Vila, here from Weather Underground:

You can see that before they lost power late on the 13th, the pressure was plummeting (down to 965 hPa) with wind speeds rapidly increasing to 45 mph.

The CIMSS Satellite blog had a post showing several satellite animations of the cyclone as it hit; I encourage you to take a look at it here.

The cyclone continued south and is giving a glancing blow to New Zealand today, where it has caused some power outages and a lot of rainfall.  Remember that in the southern hemisphere low pressure centers rotate clockwise.  Here's a satellite image from the New Zealand Met Service this morning, showing that the cyclone has basically been sheared apart, but there is still a lot of moisture with strong winds.

As The Weather Channel pointed out in an earlier blog post, Pam isn't the only cyclone we had in the western Pacific over the past week.  There were actually four active at once: Pam, which hit Vanuatu; Olwy, which is affecting the west coast of Australia; Nathan, which is hitting the York Peninsula of Austrlia, and Bavi, which is headed west towards the Philippines, but is expected to weaken before it arrives there.

Is there some uniting factor behind this burst of tropical cyclone formation?  It would appear so---the Madden-Julian Oscillation (MJO) is currently, by many measures, the strongest we have seen it in a very long time.

What is the Madden-Julian Oscillation?  The exact nature of it is still a somewhat poorly-understood topic in meteorology, despite intense research activity in recent years.  Basically, it's an area of enhanced convection (thunderstorm activity) that propagates around the equator every 30-90 days.  (See the Wikipedia article for a decent overview).  We can see this looking at a global satellite loop over the past week.  There's been a lot of cloud cover and convection moving out from Papua New Guinea and northern Australia and into the central Pacific.  You can see that as it moves east, it looks like there are tropical cyclones (Bavi and Pam) that are being "shed" from the convection.

It turns out that the basic structure of the MJO has an equatorial "heat source" being trailed by flanking "gyres" on either side of the equator.  If strong enough, these "gyres" can break away and form tropical cyclones, which is exactly what we saw last week.

How strong is the MJO right now?  Here's a diagram that tracks the strength and position of the MJO as it goes around the globe, called the Wheeler-Hendon diagram.  

You can see that there are different "zones" on the diagram, labeled with different geographical regions (the "Maritime Continent" (Indonesia, New Guinea, etc.) , "Western Pacific", "Western Hemisphere and Africa" and the "Indian Ocean".  So as the MJO moves around the globe, it's supposed to make a big circle around the diagram.  The magnitude of the MJO (how strong the convection associated with it is) is given by how far from the center you are.  The line on the plot shows where the MJO has been for the past 40 days.  You can see that for much of February (the purple part of the line, the MJO was very weak (close to the middle of the diagram) and not clearly being tracked around the globe.  But in March (the red part of the line), the MJO exploded in strength over the Maritime Continent and is now moving out over the western Pacific, getting even stronger.  This is WAY stronger than the MJO has been for a long time!  No wonder we are seeing powerful cyclones.

It turns out that an "active" MJO in the western Pacific can also contribute to heavy rain ("pineapple express") events on the west coast.  Here's a diagram from the MJO wikipedia article describing this connection:

So as the active area of convection on the equator associated with the MJO moves out over the Pacific, the moisture associated with that gets drawn northward to the west coast of North America, causing heavy precipitation.

Well, over the past two days we've had just that here in Seattle.  Here's an animated map of the total column water vapor over the past two days.  You can see the plume of moisture drawn northward from the tropics that brought heavy rain to Seattle.

So much rain fell (2.2") that Sunday was actually the second wettest March day on record in Seattle, and the single wettest day we've had since 2009 (courtesy of Scott Sistek).  It's amazing how a single weather anomaly like the MJO can contribute to both tremendous rain in Seattle and an extreme Cyclone in Vanuatu.

Today's classic case of isentropic lift

An incredible band of snow and rain currently stretches across the eastern  half of the country, as seen here on the radar composite from around 0100 UTC tonight:

Widespread precipitation from Texas all the way up through New York.  You'll notice in the middle of that band there is an area of relatively strong reflectivity.  This doesn't necessarily mean it is precipitating heavier in that area; rather it's a symptom of what we call the "bright band" effect that occurs when the radar is sampling mixed-phase precipitation.  It turns out that melting snow is more reflective to our WSR-88D radar beams than either pure snow or pure rain.  Usually this effect is somewhat localized and doesn't always show up well on radar composites, but here we have a very clear "bright band" that shows the separation between all snow to the north and all rain to the south.

You'll also notice that the character of the reflectivity changes across the band.  To the north the reflectivity looks a lot smoother and the northern edge of the precipitation looks almost "wispy".  That's a good indication that the radar is seeing snow.  This is opposed to the precipitation to the south which is much more "blobby" and irregular---characteristic of the radar seeing rain.  A striking example of how valuable our network of radars can be for determining precipitation types (even without looking at dual-pol products!)

But there's a bit more I wanted to talk about with respect to this band.  And it's going to get a little technical, so stop now if you just want to enjoy the fun radar images.  Here we overlay the Storm Prediction Center's mesoanalysis for 0100 UTC:

There's a lot going on on this map, but I want to focus on the surface temperature contours (the red solid and blue dashed lines).  You can see where the strong temperature gradient associated with a frontal zone is located--the temperature contours are really "squashed" together in a band from central Mississippi through northern Alabama and eastern Tennessee.  But notice that the precipitation itself is actually well behind the surface front---it doesn't begin in earnest until central Tennessee into eastern Kentucky.  Why the separation?  Why is the "lift" so far behind the surface front?

This is a classic case of what we call isentropic lift.   The surface front is only the leading edge of a dome of colder air.  This edge slopes back to the north and rises in height the further north you go.  Let's draw a cross-section through an analysis of what's going on now.  Below is a cross section (from the College of DuPage site) from New Orleans, Louisiana (on the left) to Green Bay, WI (on the right):

The red lines you see there are called "isentropes", lines of equal potential temperature.  You can see that there is a "wedge" of colder potential temperatures, and this wedge slopes up to the right (to the north).  That's describing the structure of the dome-like cold air mass sitting to the north.  If you look at the wind barbs in this dome of cold air (particularly between the surface and 850 hPa), you'll notice that they all have a strong northerly component (note: even though in this cross section "north" is technically to the right, with wind barbs "north" is still oriented "up").  This makes sense---cold air advecting out of the north.  If we look on the left end of the diagram outside of the cold air (down near New Orleans) the winds have a strong southerly component---warm air advecting out of the south.

It turns out that, as long as air remains unsaturated, as it is advected along it will maintain the same potential temperature.  In other words, if air starts at a potential temperature, it will follow that same potential temperature line wherever it goes.  So let's take air just above the ground at New Orleans.  It has a potential temperature on that map of just under 300 K. As that air moves northward, it is going to stay on the 300 K isentrope.  As we said before, all the isentropes are tilted upwards as we head north.  So,  warm air moving in from the south will be forced to rise following its isentropes as it moves northward.

The air that rises will keep following the isentrope until it's saturated.  We can see from the radar images that the air must be lifted for quite a ways before it hits saturation, as the precipitation band is so far behind the surface front.  But it's all laid out for us in that cross-section!  You can even see in the cross section above that the moist air (to the south; green contours are moisture) has a lobe lifted up and over the cold air, just like we'd expect if this were happening!

We can also look at a single isentrope and see what is happening along that particular isentrope.  Let's look at the 296 K isentrope.  We see in the cross section above that the height of that isentrope above the ground changes quite a bit as you move around horizontally.  We can make a map showing the height of that isentrope above the ground, and the moisture and winds along that isentrope.  Here is such a map, again from College of DuPage:

The heights are the black contours on this map and they are given in pressure levels.  Remember that pressure decreases with actual height.  So, we see lower pressures to the north (some of the black contours get below 400 hPa up in Canada) and higher pressures to the south (around 850 hPa in Louisiana).  This agrees with that cross section---as we go north, the 296K isentrope gets higher above the ground.  Notice the winds on this map; they are mostly southerly over the southern US. Combine this with the fact that this surface is higher off the ground as you move north and we can again conclude that the air moving northward will be rising along with this isentropic surface.  We also see how much moisture is being brought northward with this air---relative humidities over 85% for much of that region! So we have moist air being forced to rise by this flow, a classic isentropic lift setup.