Organoids are three-dimensional in vitro tissue models that closely represent the native heterogeneity, microanatomy, and functionality of an organ or diseased tissue. Analysis of organoid morphology, growth, and drug response is challenging due to the diversity in shape and size of organoids, movement through focal planes, and limited options for live-cell staining. Here, we present OrganoID, an open-source image analysis platform that automatically recognizes, labels, and tracks single organoids in brightfield and phase-contrast microscopy. The platform identifies organoid morphology pixel by pixel without the need for fluorescence or transgenic labeling and accurately analyzes a wide range of organoid types in time-lapse microscopy experiments. OrganoID uses a modified u-net neural network with minimal feature depth to encourage model generalization and allow fast execution. The network was trained on images of human pancreatic cancer organoids and was validated on images from pancreatic, lung, colon, and adenoid cystic carcinoma organoids with a mean intersection-over-union of 0.76. OrganoID measurements of organoid count and individual area concurred with manual measurements at 96% and 95% agreement respectively. Tracking accuracy remained above 89% over the duration of a four-day validation experiment. Automated single-organoid morphology analysis of a dose-response experiment identified significantly different organoid circularity after exposure to different concentrations of gemcitabine. The OrganoID platform enables straightforward, detailed, and accurate analysis of organoid images to accelerate the use of organoids as physiologically relevant models in high-throughput research.