In this paper, we look at the key forecasting algorithms and optimization strategies for the building energy management and demand response management. By conducting a combined and critical review of forecast learning algorithms and optimization models/algorithms, current research gaps and future research directions and potential technical routes are identified. To be more specific, ensemble/hybrid machine learning algorithms and deep machine learning algorithms are promising in solving challenging energy forecasting problems while large-scale and distributed optimization algorithms are the future research directions for energy optimization in the context of smart buildings and smart grids.