The Northern European countries largely rely on poplars and their hybrids that are transferred northward from more southern regions, therefore, facing a trade-off between the use of full growth potential and sufficient tolerance of low temperatures. We characterized frost damage and its link to early growth and survival of 33 poplar genotypes at the age of one, three, and four years in a clonal collection on agricultural land. The survival after the first season varied from 42% to 99%. Half of the clones had autumn frost injuries with a maximum of 20% of trees damaged. The autumn frost-damaged trees were significantly (p < 0.001) higher before the injury than the undamaged trees with site mean 105.3 ± 7.9 and 72.0 ± 2.5 cm, respectively, but, at the clone mean level, the proportion of autumn frost-damaged trees had no link to survival (rho = 0.53, p < 0.001). The observed winter frost damage showed no relation (p < 0.05) regarding the proportion of trees with autumn frost damage. Trees with winter frost damage tended to be shorter than undamaged trees (p < 0.001) and had lower survival (rho = 0.47, p < 0.01). The maximum height difference of the clones among the studied years was in a magnitude from 2.4 to 2.9, and the stability of a clonal ranking increased with the age. The results emphasize the need for further monitoring of the clone performance under the local climatic conditions before recommendations of commercial use of particular clones in Latvia.