At the present stage of society's development, there is an increasing need for highly qualified specialists who are ready for continuous updating of knowledge and skills, solving research problems in their professional activities. In this regard, one of the important tasks of the implementation of educational programs is the achievement of a high level of research competence by graduates. Teaching mathematics opens up unlimited opportunities for the development of the intellectual sphere and research skills of schoolchildren. However, the insufficient development of pedagogical technologies focused on the implementation of psychodidactic patterns of research activity development is a negative factor that prevents the realization of the developing potential of mathematical disciplines. The article presents the author's approach to the organization of research activities of high school students, based on the idea of integrating the process of educational cognition and modern digital technologies. As a key aspect of personal development in teaching mathematics, we consider the activity of mastering complex knowledge in the course of performing ordered, forming a single motivational and applied integrity of the complex tasks of a research nature, in the process of which there is an active mastery of knowledge, deeper penetration into the essence of mathematical concepts,methods, ideas. At the same time, intelligent management of research activities of high school students is carried out within the framework of using the functionality of a hybrid learning environment that integrates the functions of artificial neural networks, genetic algorithms, fuzzy logic, expert systems, providing individualization of learning, synthesis of mathematical and computer modeling of the content of knowledge and procedures. The paper describes the content and functionality of the basic modules of the intellectual learning system, describes the theoretical, methodological, and technological aspects of the organization of research activities of high school students to master complex mathematical knowledge.