Degrees of Freedom

How many degrees of freedom are evident in a physical process represented by f(s, t)? In some form questions about “degrees of freedom” (d.o.f.) are common in mathematics, physics, statistics, and geophysics. This would mean, for instance, in how many independent directions a weight suspended from the ceiling could move. Dofs are important for three reasons that will become apparent in the remaining chapters. First, dofs are critically important in understanding why natural analogues can (or cannot) be applied as a forecast method in a particular problem (Chapter 7). Secondly, understanding dofs leads to ideas about truncating data sets efficiently, which is very important for just about any empirical prediction method (Chapters 7 and 8). Lastly, the number of dofs retained is one aspect that has a bearing on how nonlinear prediction methods can be (Chapter 10). In view of Chapter 5 one might think that the total number of orthogonal directions required to reproduce a data set is the dof. However, this is impractical as the dimension would increase (to infinity) with ever denser and slightly imperfect observations. Rather we need a measure that takes into account the amount of variance represented by each orthogonal direction, because some directions are more important than others. This allows truncation in EOF space without lowering the “effective” dof very much. We here think schematically of the total atmospheric or oceanic variance about the mean state as being made up by N equal additive variance processes. N can be thought of as the dimension of a phase space in which the atmospheric state at one moment in time is a point. This point moves around over time in the N-dimensional phase space. The climatology is the origin of the phase space. The trajectory of a sequence of atmospheric states is thus a complicated Lissajous figure in N dimensions, where, importantly, the range of the excursions in each of the N dimensions is the same in the long run. The phase space is a hypersphere with an equal probability radius in all N directions.

Introduction

This is first and foremost a book about short-term climate prediction. The predictions we have in mind are for weather/climate elements, mainly temperature (T) and precipitation (P), at lead times longer than two weeks, beyond the realm of detailed Numerical Weather Prediction (NWP), i.e. predictions for the next month and the next seasons out to at most a few years. call this short-term climate so as to distinguish it from long-term climate change which is not the main subject of this book. A few decades ago “short-term climate prediction” was known as “longrange weather prediction”. In order to understand short-term climate predictions, their skill and what they reveal about the atmosphere, ocean and land, several chapters are devoted to constructing prediction methods. The approach taken is mainly empirical, which means literally that it is based in experience. We will use global data sets to represent the climate and weather humanity experienced (and measured!) in the past several decades. The idea is to use these existing data sets in order to construct prediction methods. In doing so we want to acknowledge that every measurement (with error bars) is a monument about the workings of Nature. We thought about using the word “statistical” instead of “empirical” in the title of the book. These two notions overlap, obviously, but we prefer the word “empirical” because we are driven more by intuition than by a desire to apply existing or developing new statistical theory. While constructing prediction methods we want to discover to the greatest extent possible how the physical system works from observations. While not mentioned in the title, diagnostics of the physical system will thus be an important part of the book as well. We use a variety of classical tools to diagnose the geophysical system. Some of these tools have been developed further and/or old tools are applied in novel ways. We do not intend to cover all diagnostics methods, only those that relate closely to prediction. There will be an emphasis on methods used in operational prediction. It is quite difficult to gain a comprehensive idea from existing literature about methods used in operational short-term climate prediction.

Conclusion

In this book we have reviewed empirical methods in short-term climate prediction. We devoted a whole chapter to the design of two of these methods, Empirical Wave Propagation (EWP, Chapter 3) and Constructed Analogue (CA Chapter 7). Other methods of empirical prediction were listed in Chapter 8, with brief descriptions and examples and references. One chapter is devoted to EOFs, as such a diagnostic topic, but widely used in both prediction and diagnostics, and thoroughly debated for a few decades. Two brief chapters, written in support of the subsequent chapter, Teleconnections (Chapter 4), should make the discussion on EOFs more interesting, and the topic of effective degrees of freedom (Chapter 6) is indispensable when one wants to understand why and when natural analogues would work (or not), or how an analogue is constructed, or how any method using truncation works. Most chapters can be read largely in isolation, but connections can be made of course between chapters. EWP is claimed to be useful, if not essential, in understanding teleconnections. Dispersion experiments, featuring day-by-day time-scales, link the CA and EWP methods. Examples of El Nino boreal winter behavior can be found in (a) the examples of EOFs on global SST and 500 mb streamfunction (Chapter 5), (b) specification of surface weather from 500 mb streamfunction (Chapter 7), and (c) the ENSO correlation and compositing approach (Chapter 8). The noble pursuit of knowledge may have been as important in the choice of some material as any immediate prediction application. Chapter 9 is different, less research oriented, and more an eyewitness description of what goes on in the making of a seasonal prediction. This eyewitness account style spills over into Chapter 8 here and there, because in order to understand why certain methods have survived to this day some practicalities have to be understood. The closeness to real-time prediction throughout the book creates a sense of application. However, the application in this book does not go beyond the making of the forecast itself; we completely shied away from such topics as a cost/benefit analysis or decision-making process by, for example, a climate sensitive potato farmer or reservoir operator.

Background on Orthogonal Functions and Covariance

The purpose of this chapter is to present some basic mathematics and statistics that will be used heavily in subsequent chapters. The organization of the material and the emphasis on some important details peculiar to the geophysical discipline should help the reader.

Empirical Wave Propagation

The purpose of this chapter is to demonstrate that, given a long data set of global extent, one can design a simple forecast method called Empirical Wave Propagation (EWP), which has modest forecast skill and allows us to explore aspects of atmospheric dynamics empirically, most notably aspects that help to explain mechanisms of teleconnection. The highlight of this chapter are dispersion experiments where we ask the question what happens to an isolated source at t = 0? Even though Nature has never done such an experiment, we will address this question empirically. In case the reader does not need/want to know the technical details of deriving wavespeeds he/she can skip to page 22 (EWP diagnostics sct 3.2) of this chapter. We will also discuss the skill of one-day EWP forecasts, in comparison to skill controls like “persistence”, as a function of season, hemisphere, level and variable. While short-range (1 day) forecasts are certainly not the topic of this book, we note that the short-term wave propagation features described here do nourish and maintain the teleconnection patterns thought to be important for longer range forecasts. EWP uses either zonal harmonic waves (sin/cos pairs) along each latitude circle separately, or global domain spherical harmonics (see Parkinson and Washington (1986) for the basics on spherical harmonics). The orthogonal functions used here are thus analytical. The atmosphere is to first order rotation-symmetric and obviously periodic in the east–west direction, which makes the zonal Fourier transform a natural. Moreover, many weather systems, wave-like in the upper levels, are seen to move from west to east (east to west) in the mid-latitudes (tropics), so a decomposition in sin/ cos functions should inform us about phase propagation and energy dispersion on the sphere. For any initial time we decompose the state of the atmosphere into harmonic waves. If we knew the wave speed, and made an assumption about the future amplitude, we could make forecasts by analytical means. But how do we know the phase speed? One way to proceed, with data alone, is to calculate from a large data set the climatological speeds of anomaly waves. This is where the empirical aspects come in.

The Practice of Short-Term Climate Prediction

While previous chapters were about methods and their formal backgrounds, we here present a description of the process of making a forecast and the protocol surrounding it. A look in the kitchen. It is difficult to find literature on the subject, presumably because a real-time forecast is not a research project and potential authors (the forecasters) work in an ever-changing environment and may never feel the time is right to write an overview of what they are doing. Moreover, it may be very difficult to describe real-time forecasts and present a complete picture. Nearly all of the material presented here specifically applies to the seasonal prediction made at the NWS in the USA, but should be relevant elsewhere. A real-time operational forecast setting lacks the logic and methodical approach one should strive for in science. This is for many reasons. There is pressure, time schedules are to be met, input data sets could be missing or incorrect, and one can feel the suspense, excitement and disappointment associated with a forecast in real time. There are habits that are carried over from years past—forecasters are partly set in their ways or find it difficult to make major changes in mid-stream. The interaction with the user influences the forecast, and/or the way the information is conveyed. Psychology enters the forecast. Assumptions about what users want or understand do play a role. Generally speaking a forecast is thus a mix of what is scientifically possible on the one hand and what is presumably useful to the customer on the other. The CPC/NWS forecasts are moreover for the general user, not one user specifically. Users for short-term climate forecasts range from the highly sophisticated (energy traders, selling of weather derivatives, hydrologists) via the (wo)man in the street to entertainment. The seasonal forecast has been around a long time in the USA. Jerome Namias started in-house seasonal forecasts at the NWS in 1958. After 15 years of testing, his successor Donald Gilman made the step to public release in 1973.

Empirical Orthogonal Functions

The purpose of this chapter is to discuss Empirical Orthogonal Functions (EOF), both in method and application. When dealing with teleconnections in the previous chapter we came very close to EOF, so it will be a natural extension of that theme. However, EOF opens the way to an alternative point of view about space–time relationships, especially correlation across distant times as in analogues. EOFs have been treated in book-size texts, most recently in Jolliffe (2002), a principal older reference being Preisendorfer (1988). The subject is extremely interdisciplinary, and each field has its own nomenclature, habits and notation. Jolliffe’s book is probably the best attempt to unify various fields. The term EOF appeared first in meteorology in Lorenz (1956). Zwiers and von Storch (1999) and Wilks (1995) devote lengthy single chapters to the topic. Here we will only briefly treat EOF or PCA (Principal Component Analysis) as it is called in most fields. Specifically we discuss how to set up the covariance matrix, how to calculate the EOF, what are their properties, advantages, disadvantages etc. We will do this in both space–time set-ups already alluded to in Equations (2.14) and (2.14a). There are no concrete rules as to how one constructs the covariance matrix. Hence there are in the literature matrices based on correlation, based on covariance, etc. Here we follow the conventions laid out in Chapter 2. The post-processing and display conventions of EOFs can also be quite confusing. Examples will be shown, for both daily and seasonal mean data, for both the Northern and Southern Hemisphere. EOF may or may not look like teleconnections. Therefore, as a diagnostic tool, EOFs may not always allow the interpretation some would wish. This has led to many proposed “simplifications” of the EOFs, which hopefully are more like teleconnections. However, regardless of physical interpretation, since EOFs are maximally efficient in retaining as much of the data set’s information as possible for as few degrees of freedom as possible they are ideally suited for empirical modeling. Indeed EOFs are an extremely popular tool these days.

Teleconnections

Mankind has long been intrigued by the possibility that weather in one location is related to weather somewhere else, especially somewhere very far away. The fascination may be mostly related to possible predictions that could be based on such relationships. The severe weather that harmed the British Army in the Crimea in November 1854 (Lindgrén and Neumann 1980) was due to a weather system moving across Europe, suggesting it could have been anticipated from observations upstream. It took analyses of many surface weathermaps, an activity starting around 1850, to see how weather systems have certain horizontal dimensions, thousands of kilometers in fact, and move around in semisystematic ways. It thus followed that, in a transient sense, the weather at two places can be related, and in a time-lagged sense that weather observed at one (or more) places serves as a predictor for weather at other locations. The other reason for fascination with teleconnection might be called “system analysis”. The idea that given an impulse at some location (“input”) a reaction can be expected thousands of miles away (the “output”) through a chain of events, is intriguing and should tell us about the workings of the system. It is akin to an engineer testing electronic equipment. Unfortunately, Nature is not a laboratory experiment where we can organize these impulses. Only by systematically observing what Nature presents us with, may we dare to search for teleconnections in some aggregate way. The word teleconnection suggests a connection at long distance, but a stricter definition requires some thought and pruning down of endless possibilities. We need to make choices about (a) simultaneous vs time-lagged teleconnections, (b) correlations vs other measures of “connection”, (c) transient vs standing teleconnections, (d) teleconnections in filtered data (e.g. seasonal means) vs unfiltered instantaneous (e.g. daily) data, and (e) one or more variables. On (a), (b) and (e) our choice in this chapter is simultaneous, use of linear correlation (except in section 4.3 where other measures of teleconnection are discussed), and a single variable respectively. On possibilities (c) and (d) we keep our options open.

Methods in Short-Term Climate Prediction

The purpose of this chapter is to list the more common accepted methods used in short-term climate prediction, explain how they are designed, how they are supposed to work, what level of skill can be expected and the references to find more about them. The emphasis is on methodology but aspects of verification and cross-validation will be mentioned as well. Most methods will be accompanied by an example. We will also mention some of the less common methods, but with less detail. We even list some methods that are not used, to delineate which are acceptable and which are not. Sections 8.1–8.6 and 8.8 are easy to read, but Sections 8.7 and 8.9 are more difficult. It will become clear by the end of the climatology section (8.1), that only the departure from climatology, the so-called anomalies, are considered worthy forecast targets. The climatology itself, including such empirically established facts as “days are warmer than nights”, and “winters are colder than summer”, is considered too obvious to be a forecast target. This is not to say that a quantitative explanation of the Earth’s climate, including daily and annual cycle, is easy. But in professionally honest verification no points are given for forecasting a correct climatology. This chapter is thus about forecasting aspects of the geophysical system that are not so obvious and more difficult. The daily and annual cycle are periodic variations controlled by external forcings such as the solar heating. Implicit in identifying a periodic phenomenon as such is that the forecast of the phenomenon is easy out to infinity. This explains a widespread search for “cycles” in early meteorological research, but very little has been found other than the obvious daily and annual cycles. By removing a climatology that accounts for daily and annual variations we in effect remove the known easy periodic part of the system. In the absence of any other information climatology is the best information available. As many travelers can attest, somebody visiting an unfamiliar location 6 months from now is well served by inspecting climatological tables.

Analogues

In 1999 the Earth’s atmosphere was gearing up for a special event. Towards the end of July, the 500 mb flow in the extra-tropical SH started to look more and more like a flow pattern observed some 22 years earlier in May 1977. Two trajectories in the N-dimensional phase space, N as defined in Chapter 6, were coming closer together. Figure 7.1 shows the two states at the moment of closest encounter, with the appropriate climatology subtracted. These two states are, for a domain of this size, the most similar looking patterns in recorded history. But are these good analogues? They do look alike, nearly every anomaly center has its counterpart, but they are certainly not close enough to be indistinguishable within observational error, the anomaly correlation being only 0.81. The rms difference between the two states in Figure 7.1 is 71.6 gpm, far above observational error (<10 gpm). The close encounter did not make it to the newspapers and, more telling, not even to a meteorological journal. The idea of situations in geophysical flow that are analogues to each other has always had tremendous appeal, at least in meteorology. Even lay people may comment that the weather today or this season reminds them of the weather in some year past. The implications of true analogues would be enormous. If two states many years apart were nearly identical in all variables on the whole domain (of presumed relevance) , including boundary conditions, then their subsequent behavior should be similar for some time to come. In fact one could make forecasts that way, if only it was easy to find analogues from a “large enough” data set. The analogue method was fairly widely used for weather forecasting at one time (Schuurmans 1973) but currently is rarely used for forecasts per sé (for all the reasons explained in Section 7.1). Rather analogues are used to specify one field given another, a process called “specification” or downscaling, or to learn about predictability (Lorenz 1969). In Section 7.1 we review the idea and limitations of naturally occurring analogues, and explain why/when it is (un)likely to find analogues.