Scientists from the Center for Diagnostics and Telemedicine of the Moscow Health Department have published a study guide, Assessing the Quality and Performance Parameters of Artificial Intelligence Technologies in Healthcare. The guide is based on the scientific and practical results of the Moscow Experiment (2020–2025), which focused on implementing computer vision for medical image analysis. It examines the future application of these technologies in healthcare. Based on the Experiment, the authors developed a methodology for monitoring AI quality and performance parameters throughout all stages of the technology lifecycle. A separate section of the guide is devoted to the national standards entitled Artificial Intelligence Systems in Clinical Medicine.
“Over the past five years, as part of the Moscow Experiment, we have progressed from test implementations to the creation of a comprehensive quality-control system for medical AI. We have compiled all this experience in a new study guide. It is not about abstract ‘artificial intelligence,’ but rather about specific figures, checklists, thresholds, and action algorithms: how to verify a system’s security and accuracy, and whether it can be trusted in a real clinical setting,” said Yuri Vasiliev, Medical Director of the Center for Diagnostics and Telemedicine of the Moscow Health Department. “I am confident that this manual will be a valuable resource for cyberneticists, healthcare administrators, and anyone responsible for implementing AI in medicine.”
The authors have summarized everything they have learned over the years: how to verify that a neural network operates accurately, how quickly it performs, and whether it genuinely helps physicians make diagnoses. The book presents official Russian National standards (GOSTs) governing medical AI, as well as the authors’ own research. Special attention is given to the evaluation of modern LLMs that can analyze patient records and generate summaries from medical documents. The manual is based not on abstract theories, but on real-world practice, validated methods, and ready-to-use checklists for quality control of artificial intelligence in a hospital setting.
The publication is intended for students and residents specializing in General Medicine, Pediatrics, and Medical Cybernetics, as well as programmers and testers working with medical information systems. The material is structured to help develop competencies in the critical analysis of AI performance, the management of digital transformation projects, and monitoring.
“We have thoroughly examined not only classic metrics but also modern tools for large-scale generative models, as well as a unique ‘maturity matrix’ that clearly distinguishes reliable solutions from early-stage prototypes. This manual gives us an opportunity to share with colleagues and future generations of cyberneticists the experience we have accumulated while analyzing millions of medical images. Many of the methods and metrics were developed independently, based on the experience of the Experiment. We want students and practicing physicians not to be afraid of AI, but to be able to assess its strengths and weaknesses professionally. After all, patient safety is always the result of continuous monitoring,” said Anton Vladzimirsky, Deputy Director for Research at the Center for Diagnostics and Telemedicine of the Moscow Department of Health. This textbook is recommended for use in higher education institutions offering programs in Medical Informatics (Protocol No. 095 dated December 18, 2025). The leading experts who served as reviewers were Professor Nikolai Nudnov, MD, PhD, of the Russian Scientific Center of Roentgenology and Radiology of the Ministry of Health of the Russian Federation, and Professor Georgy Lebedev, ScD, PhD, of Sechenov University.
The Center for Diagnostics and Telemedicine is a leading scientific and practical organization within the Moscow Health Department. It organizes and develops the work of radiology and instrumental diagnostic departments, digitally transforms healthcare, implements AI technologies in clinical practice, conducts scientific research, and trains healthcare professionals.