We develop a topology optimization approach to design two- and three-dimensional phononic materials/structures. Whereas most phononic crystals are based on ideas of periodicity, our approach relieves us of this constraint and allows for the creation of much more complex and efficient designs. It also enables us to go beyond simple filters and waveguides to the point of creating phononic devices. We focus on surface wave devices which carry the energy of the wave near and along the surface of the device. The design of surface wave devices is particularly attractive given recent advances in nano- and micro-manufacturing processes, such as thin-film deposition, etching, and lithography, which make it possible to precisely place thin film materials on a substrate with sub-micron feature resolution. The thin films can be made thick enough to affect most of the energy propagating in the surface wave and therefore a patterned thin film is all that is needed to create a surface wave device. It is the role of topology optimization to determine this pattern.