Cluster analysis of cell nuclei in h&e‐stained histological sections of prostate cancer and classification based on traditional and modern artificial intelligence techniques

Biomarker identification is very important to differentiate the grade groups in the histopathological sections of prostate cancer (PCa). Assessing the cluster of cell nuclei is essential for pathological investigation. In this study, we present a computer‐based method for cluster analyses of cell nuclei and performed traditional (i.e., unsupervised method) and modern (i.e., supervised method) artificial intelligence (AI) techniques for distinguishing the grade groups of PCa. Two da-tasets on PCa were collected to carry out this research. Histopathology samples were obtained from whole slides stained with hematoxylin and eosin (H&E). In this research, state‐of‐the‐art approaches were proposed for color normalization, cell nuclei segmentation, feature selection, and classifica-tion. A traditional minimum spanning tree (MST) algorithm was employed to identify the clusters and better capture the proliferation and community structure of cell nuclei. K‐medoids clustering and stacked ensemble machine learning (ML) approaches were used to perform traditional and modern AI‐based classification. The binary and multiclass classification was derived to compare the model quality and results between the grades of PCa. Furthermore, a comparative analysis was carried out between traditional and modern AI techniques using different performance metrics (i.e., statistical parameters). Cluster features of the cell nuclei can be useful information for cancer grad-ing. However, further validation of cluster analysis is required to accomplish astounding classification results.