The soil properties, climate, type of management, and fermentation process critically affect the productivity and quality of tea. In this study, tender tea leaves were collected from central Taiwan, and organic components in their infusions as well as physical and chemical soil properties differentiated using aerial photographs where good (G) and bad (B) growth exhibitions were determined. Eleven physical and chemical soil properties as well as five compounds in tea infusions were analyzed to determine the main factor that affects the growth of these tea trees. The Fleiss’ kappa statistic results revealed that the wet aggregate stability, pH, and exchangeable potassium content exhibit the most significant effect, with scores of 0.86, 0.64, and 0.62, respectively. Soil quality calculated using the mean weight diameter based on 11 soil properties revealed that ~67% of the total score of G is greater than that of B. Generally, contents of total polyphenols (51.67%) and catechins (51.76%) in the infusions of B were greater than those of G. In addition, significant positive correlations between the free amino acids content and soil properties, including pH and copper content, were observed. However, a negative correlation between the free amino acids and flavone contents and most of the soil properties was observed. The survey data set obtained from this study can provide useful information for the improved management of tea plantations.