BackgroundExamining Hematoxylin & Eosin (H&E) images using brightfield microscopes is the gold standard of pathological diagnosis as it is an inexpensive method and provides basic information of tumors and other nuclei. Complementary to H&E-stained images, Immunohistochemical (IHC) images are crucial in identifying tumor subtypes and efficacy of treatment response. Other newer technologies such as Multiplex Immunofluorescence (mIF) in particular, identifies cells such as tumor infiltrating lymphocytes (TILs) which can be augmented via immunotherapy, an evolving form of cancer treatment. Immunotherapy helps in the manipulation of the host immune response and overcome limitations like the PD-1 (Programmed Cell Death-1) receptor induced restrictions on TIL production. If the same biopsy specimen is used for inspection, the higher order features in H&E images can be used to obtain information usually found in mIF images using Convolutional Neural Networks (CNNs), widely used in object detection and image segmentation tasks.MethodsAs shown in (figure 1), firstly, a novel optical flow-based image registration paradigm is prepared to co-register H&E and mIF image pairs, aided by adaptive color thresholding and automated color clustering. Secondly, generative adversarial networks (GANs) are adapted to predict TIL (CD3, CD45) regions. For this purpose, a unique dataset is ideated and used in which a given single channel mIF image, e.g., a CD3 channel mIF image is superimposed on the corresponding H&E image. Primarily, the Pix2Pix GAN model is used to predict CD3 and/or CD45 regions.ResultsThe intensity-based image registration workflow is fast and fully compatible with the given dataset, with an increase in evaluation metric scores after alignment (table 1). Furthermore, this study would be the first implementation of optical flow as the registration algorithm for pathological images. Next, the use of the special dataset not only reduces penalization during the training of the Pix2Pix model, but also helped in gaining repeatable results with high scores in metrics like structural similarity index measure and peak-signal to noise ratio, with minimal effects on location accuracy (table 2 and table 3).ConclusionsThis multi-modal pathological image transformation study could potentially reduce dependence on mIF and IHC images for TILs scoring, reducing the amount of tissue and cost needed for examination, as its information is derived directly from inexpensive H&E images automatically – ultimately develop into a pathologist-assisted tool for TILs scoring. This would be highly beneficial in facilities where resources are relatively limited.Ethics ApprovalThe Agency of Science, Technology and Research, Singapore, provided approval for the use of control tissue materials in this study IRB: 2020 112Abstract 818 Figure 1Proposed workflowAbstract 818 Table 1Image registration metricsAbstract 818 Table 2CD3 negative regions examplesAbstract 818 Table 3CD3 positive regions examples
Precise Image Registration with Structural Similarity Error Measurement Applied to Superresolution
