Epilogue A Game of Theories

The Quantum Cookbook shows that whilst quantum mechanics is mathematically challenging, some basic knowledge and a bit of effort will carry you a long way. It also explains how quantum mechanics was derived from the physics. The abstract formalism based on state vectors in Hilbert space was introduced only when it was deemed desirable to lend the theory greater mathematical consistency, and to reject some of its historical baggage. The best way to come to terms with this formalism is to understand how and why it came about. Debates about interpretation continue to this day and, by providing some historical context, you should get the impression that any lack of comprehension of its meaning on your part is absolutely not your fault. Quantum mechanics challenges our comprehension of what any (and all) scientific theories are meant to be telling us about the nature of reality. It’s okay to have doubts.

Einstein, Bohm, Bell, and the Derivation of Bell’s Inequality

By 1935, the Copenhagen interpretation had become the orthodoxy. Einstein needed to find a situation in which it is possible in principle to acquire knowledge of the state of a quantum system without disturbing it in any way. Working with two young theorists, Boris Podolsky and Nathan Rosen, Einstein devised an extraordinarily cunning challenge based on entangled particles. We can discover the state of one particle with certainty by making measurements on its entangled partner. All we have to assume is that the particles are local: any measurement we make on one in no way affects or disturbs the other. Through the work of David Bohm and John Bell, the challenge posed by EPR became accessible to experiment, and Bell devised a simple test for all locally realistic theories. All the experiments performed to date suggest that the standard quantum formalism is correct: in any realistic interpretation, quantum particles are non-local.

Born’s Interpretation of the Wavefunction

Schrödinger hoped that his wave mechanics would help to re-establish some sense of ‘visualizability’ of the physics going on inside the atom. In searching for a suitable interpretation of the wavefunction, he focused on the density of electrical charge, which he associated with the wavefunction ψ‎ multiplied by its complex conjugate. Hidden in his words is the interpretation that would eventually come to dominate our understanding of the wavefunction. Max Born had no hesitation in concluding that the only way to reconcile wave mechanics with the particle description is to interpret the modulus-square of the wavefunction as a probability density. It was Wolfgang Pauli who proposed to interpret this not only as a transition probability or as the probability for the system to be in a specific state, as Born had done, but as the probability of ‘finding’ the electron at a specific position in its orbit inside an atom.

Planck’s Derivation of E=hν

Planck was unwilling to accept Boltzmann’s statistical interpretation of the second law. He therefore needed to find a way to show how irreversible processes could result from continuous matter. Planck chose the physics of black body radiation as a battleground. After discovering his radiation law in October 1900, he sought a deeper theoretical interpretation for it. Now thermodynamics is not the most obvious place to look for evidence of the quantum nature of radiation and, in an ‘act of desperation’, Planck had to torture the theory in a way that would eventually allow this conclusion to emerge from an entirely classical structure. Planck’s derivation heralded the very beginning of the quantum revolution, but only in promise, not in deed. The revolution began in earnest in 1905 with the help of Albert Einstein.

Von Neumann and the Problem of Quantum Measurement

Heisenberg was an ‘anti-realist’. Although Bohr was infamously obscure in many of his writings, it seems that he adopted a generally anti-realist interpretation, too. As their debate became more bitter, in early June 1927 Pauli was called in to mediate. With Pauli’s help, they forged an uneasy consensus, which became known as the Copenhagen interpretation. Einstein didn’t like it at all, setting the stage for a great debate about the quantum representation of reality. Although von Neumann’s formalism broadly conforms to the Copenhagen interpretation, he saw no need to introduce an arbitrary split between the classical and quantum worlds. But eliminating the split poses the problem of quantum measurement: when scaled to classical dimensions, a superposition of different measurement outcomes appears contrary to our experience, exemplified by the famous paradox of Schrödinger’s cat. Von Neumann was obliged to break the infinite regress by postulating the ‘collapse of the wavefunction’.

Dirac, Von Neumann, and the Derivation of the Quantum Formalism

The evolution of quantum mechanics through the 1920s was profoundly messy. Some physicists believed that it was necessary to throw out much of the conceptual baggage that early quantum mechanics tended to carry around with it and re-establish the theory on much firmer ground. It was at this critical stage that the search for deeper insights into the underlying reality was set aside in favour of mathematical expediency. All the conceptual problems appeared to be coming from the wavefunctions. But whatever was to replace them needed to retain all the properties and relationships that had so far been discovered. Dirac and von Neumann chose to derive a new quantum formalism by replacing the wavefunctions with state vectors operating in an abstract Hilbert space, and formally embedding all the most important definitions and relations within a system of axioms.

Heisenberg’s Derivation of the Pauli Exclusion Principle

Despite the success of Schrödinger’s description of the H-atom, it became apparent that the spectrum of the simplest multi-electron atom—helium—could not be so readily explained. And the spectra of other atoms showed ‘anomalous’ splitting in a magnetic field. In 1920 Sommerfeld introduced a fourth quantum number. A few years later Pauli was led to the inspired conclusion that the electron must have a curious ‘two-valuedness’ characterized by a quantum number of ½, and went on to discover the exclusion principle. Perhaps this is because the electron possesses a self-rotation, leading to the notion of electron spin, potentially explaining why each orbital can accommodate only two electrons. Heisenberg traced this behaviour back to the symmetry properties of the wavefunctions. By observing which transitions in the spectrum of helium are allowed and which are forbidden, we can deduce the generalized Pauli principle, from which the exclusion principle follows.

Heisenberg, Bohr, Robertson, and the Uncertainty Principle

From the outset, Heisenberg had resolved to eliminate classical space-time pictures involving particles and waves from the quantum mechanics of the atom. He had wanted to focus instead on the properties actually observed and recorded in laboratory experiments, such as the positions and intensities of spectral lines. Alone in Copenhagen in February 1927, he now pondered on the significance and meaning of such experimental observables. Feeling the need to introduce at least some form of ‘visualizability’, he asked himself some fundamental questions, such as: What do we actually mean when we talk about the position of an electron? He went on to discover the uncertainty principle: the product of the ‘uncertainties’ in certain pairs of variables—called complementary variables—such as position and momentum cannot be smaller than Planck’s constant h (now h / 4π‎).

What’s Wrong with This Picture?

Despite its intuitive appeal, classical mechanics is just as fraught with conceptual difficulties and problems of interpretation as its quantum replacement. The problems just happen to be rather less obvious, and so more easily overlooked or ignored. Quantum mechanics was born not only from the failure wrought by trying to extend classical physical principles into the microscopic world of atoms and molecules, but also from the failure of some of its most familiar and cherished concepts. To set the scene and prepare for what follows, this Prologue highlights some of the worst offenders, including: space and time; force and energy; the troublesome concept of mass; light waves and the ether; and atoms and the second law of thermodynamics.

Einstein’s Derivation of E=mc2

Hamiltonian mechanics did not rid the structure of its dependence on Newton’s absolute space and time. And Maxwell’s electrodynamics demanded an ether that could not be found, no matter how hard the physicists looked for it. Einstein judged that the solution to these thorny problems demanded a firmly pragmatic approach in which the ‘observer’ takes centre-stage. To understand the physics correctly, we must accept that this is physics as seen from the perspective of someone or something inside the reality that is being observed or measured. Such an observer is implicit in the physics of Newton. But Newton’s laws are formulated as though the observer can be imagined to sit outside of the reality in which all the action is taking place. The result is Einstein’s special theory of relativity, which he published in June 1905. In September, he published an addendum in which he derived the iconic formula E=mc2.