Documente Academic
Documente Profesional
Documente Cultură
Abstract
As developing parallel software is rather a difficult task at present, the questions of theoretical training of specialists and investigation of methodology of projecting such systems become very urgent. Within the framework of this article we provide historical and technical information preparing a programmer for gaining knowledge in the sphere of developing parallel computer systems.
For reader
This document is part of a series of articles devoted to issues of creating quality and effective program solutions for modern 64-bit multi-core systems. You can read other articles on the site http://www.viva64.com.
Introduction
It is very difficult for programmers who only begin to use multi-processor computers to master all the peculiarities of their usage while developing programs for applied tasks. As practice shows difficulties begin when effectiveness and mobility are required of parallel software being developed. It is explained by that universal means simplifying a programmer's labor and providing full access to debugging information are only being developed. The problem is that there are no standards in the sphe of re creating and debugging programs for parallel systems because the field of computer science is very young. Correspondingly, there are no logically complete training courses on parallel programming for beginners at present. Development of multi-processor computers is inseparably linked with development of parallel programming technologies, both universal and for concrete computer architectures. By a programming technology, that is by organization of work with memory, we mean usage of means of controlling a concrete computer. It should be noted that while developing software (both controlling means and means for solving applied tasks) for super-computers special attention should be paid to programming technique, i.e. building of a logical program architecture. By this we mean development and addition of paralleling algorithms increasing effectiveness of their execution on multi-processor computers.
imagine effective work without using computers in such spheres as production scheduling and control, projecting and developing complex devices, publishing, education, in other words, in all the fields where processing of large sizes of information is needed. Such tasks appeared in the middle of the previous century due to development of atomic energetics, aircraft building, rocket-cosmic technologies and some other science and technique fields [1]. Nowadays, the field of tasks demanding powerful computing resources for their solution has extended even more. This relates to fundamental changes in the very organization of scientific investigations. Because of wide introduction of computers, computational modeling and numerical experiment have developed greatly [2]. Filling the gap between physical experiments and analytical approaches, computational modeling allowed us to investigate phenomena which are either too complicated to be investigated through analytical approaches or too expensive or dangerous to be inves tigated experimentally. Meanwhile, numerical experiment allowed us to make the process of scientific and technical search much cheaper. It became possible to model in real time the processes of intensive physico-chemical and nuclear reactions, global atmospheric processes, processes of economical and industrial development of regions etc. It is obvious that solution of such great tasks requires great computational resources. Usage of computers for computational purposes has always remained the main force of the progress in computer technologies. That's why it is no wonder that as a main characteristic of computers we use such an index as performance, i.e. the value showing what quantity of arithmetic operations it can perform in a time unit. It is this index which shows the scale of progress achieved in computer technologies. Thus, for example, performance of one of the first computers EDSAC was only about 100 operations per second, while peak performance of Earth Simulator, one of the most powerful super computers nowadays, is 40 trillion operations per second. Thus, performance has increased a 400 billion times! There is no other sphere of human activity where progress is so evident and so great. Of course, anyone would immediately ask: why did it become possible? Strangely enough, the answer is rather simple: because of 1000-time increase of electronic circuits' performance and maximum extension of paralleling of data processing. The idea of parallel data processing as a powerful source for increasing performance of computers was expressed by Charles Babbage about a hundred years before the first electronic computer appeared. But the level of technological development in the middle of the 19th century didn't allow him to fulfill this idea. With the appearance of the first electronic computers these ideas became more than once the starting point when developing the most advanced and high-performance computer systems [3]. Without exaggeration we can say that the whole history of development of high-performance computer systems is the history of fulfilling the ideas of parallel processing at a certain stage of development of computer technologies, naturally, combined with increase of speed and safety of electronic circuits. Brand new decisions in increasing performance of computer systems were introduction of pipeline organization of command execution, inclusion of vector operations into the command system allowing you to process whole data arrays by one command; distribution of calculations among many processors. Combination of these 3 mechanisms in the architecture of the super-computer Earth Simulator consisting of 5120 vector-pipeline processors allowed it to gain record performance, which excesses performance of modern personal computers by 20000 times. It is obviously that such systems are too expensive and are produced in single copies [4]. And what about commercial production nowadays? The wide variety of computers produced in the world today
can be roughly divided into four classes: Personal Computers (PC); Workstations (WS); Supercomputers (SC); cluster systems [5]. This division is very approximate because of rapid progress in the sphere of development of microelectronic technologies. Performance of computers of every class doubles nearly every 18 months at present (in accordance to the so called Moore's Law). Because of this the supercomputers of the beginning of the 90-s often yield to modern workstations in performance, and personal computers become successful rivals to workstations. However, let's try to classify them somehow. Personal computers. Strange enough, in this case we mean single-processor systems on Intel or AMD platforms controlled by single-user OS (Microsoft Windows and others). They are used mostly as a personal work place. Workstations. Most often these are computers with RISC-processors with multi-user OS relating to UNIX OS family. They contain from one to four processors, support remote control [6] and can maintain needs of a small group of users. Supercomputers. Their distinctive feature is that they are usually large and, consequently, very expensive multi-processor systems. In most cases supercomputers use the same commercial processors as workstations. That's why the difference between them is often rather in quantity than in quality. For example, we can speak of a 4-processor workstation by SUN company and a 64-processor supercomputer by the same company. Most likely, the both use the same microprocessors. Cluster systems. In recent years they have been used in the whole world as a cheap alternative to supercomputers. A system of the required performance is assembled from ready-made commercial computers united in their turn by some commercial DCE. Thus, multi-processor systems which have been early associated with supercomputers mostly, nowadays become popular in the whole range of produced computer systems, from personal computers to supercomputers on the basis of vector pipeline processors. On the one hand, this circumstance increases availability of supercomputer technologies and, on the other hand, makes mastering them urgent as you need to use special programming technologies for all the types of multi-processor systems in order to allow a program to fully use the resources of a high-performance computer system [7, 8]. Usually this is implemented by dividing a program with the help of some tool into parallel branches each of which is executed on a separate processor.
values allows us to significantly speed up solving the task. And finally we should mention that using supercomputers is always more effective for maintaining needs of a large group of users than using the corresponding number of single-processor workstations as it is easier in this case to provide balanced and more effective load of computational resources with the help of the task managing system. Unlike common multi-user systems, OS of supercomputers, as a rule, in order to get the maximum rate of program execution don't permit to share resources of one processor between different, simultaneously executed programs. That's why there can be the following modes of using an nprocessor system as two opposite variants:
y y
all the resources are allocated for execution of one program and in this case we expect an n-fold speed-up of program execution in comparison to a single-processor system; n common single-processor programs are executed simultaneously and the user expects that other programs won't influence the speed of execution of his program.
One control processor is defined while all the other processors are used as processing nodes, i.e. computing nodes. Each computing processor performs primary tasks - solution of ODE system for the next mesh point with the corresponding local parameters. The control processor distributes the primary tasks among the computing processors and collects the results. In the beginning of the next step each processor waits for a new data chunk, processes it, returns the result and starts waiting for the next task until instead of the next task it gets a message that all the mesh points are processed. As there is no need to synchronize primary tasks, different processors can get different number of computational nodes as the data processing is finished. Thereby the problem of balanced load of processors is solved even if the time for solving the equation system for different mesh points or processors' performance vary greatly. In case of heterogeneous computational load when computing different points of the spatial mesh, usage of the "group decision" method potentially allows you to significantly reduce downtime and increase effectiveness of paralleling in comparison to the geometrical parallelism method considered further. The advantages of this method can be fully implemented if the data for processing are from the beginning concentrated on one of the processors which in this case can serve as the control processor. When the source data are initially distributed among the processors at random, preliminary collection of the data corresponding to all the computational points on one of the processors is required to use this method. The necessity of the preliminary data copying from all the processors to one and the following return of the results from this processor to the processors-"holders" of the points significantly reduces effectiveness of this method and makes it of little use for solving most tasks of computational modeling.
It is much more difficult to write a parallel program than to write a sequential one. Creation of software for parallel computers is the central problem of supercomputer calculations [13]. Partially the problem of choosing the optimal number of parallel branches in correspondence with the criterion of minimum total time costs can be solved with the help of automations of parallel program generation. A particular case of solving this problem for the computer systems with MIMD architecture is considered in the article by V.A. Kostenko "To the question of evaluating the optimal parallelism level" [14]. Effectiveness of using multi-processor computer systems is to a large degree determined by the quality of the applied parallel programs. A program is considered effective when all the processors defined for processes are loaded during its execution. But practically it is impossible.
enlargement of paralleling units; decrease of complexity of the algorithms of generating parallel procedures (subprograms); preliminary preparation of the package of different source data variants; paralleling of the algorithms of generating parallel procedures (subprograms).
investigation of the common program structure; definition of the main computational core, input-output localization; definition of the potential parallelism of a fragment; definition of the sequential fragments of calculations and attempt to use alternative algorithms for such fragments; definition and minimization of data redistribution points; conversion of the traditional loop for parallel algorithms; minimization of the number and size of temporary arrays for optimizing cache-memory handling; passing on from the source program working with full arrays to the program processing only a local chunk distributed for a processor: change of arrays' sizes and the corresponding transformation of the program text.
We should note that solution of these tasks allows us to perform an effective port of a sequential program on a parallel architecture. The process of developing a parallel program is very long and laborious despite that, as a rule, there already exists an implementation of its "sequential" counterpart. A program is usually developed on a computer with a certain architecture and its practical application is performed on another computer, more powerful and with the typology different from that of the former machine. This approach allows you to economize computer time on more powerful supercomputers which are much fewer than cheaper ones. When porting a parallel program on computers with a different architecture a programmer faces the problem of invalidity of once developed parallel procedures. At present there are no universal means of adapting programs to a concrete architecture of supercomputers and that's why this problem has to be mostly solved manually what makes the process very labor-intensive [15]. To save labor of a programmer RAS mathematical institutions are developing libraries of effective procedures and algorithms for concrete architectures of supercomputers (RAS Ural Department, Research-and-development computer center of MSU named after M.V. Lomonosov). Using these libraries can partially save labor of an applied programmer not only at the stage of modifying a program for more powerful supercomputers, but at the stage of the primary development of a parallel program.
Process control is one of the most important tasks of OS. To perform this function on supercomputers semaphore technology [22] can be used which consists in locking and unlocking of processes. Semaphores have been traditionally used for synchronizing processes addressing shared data. Each process should exclude for all the other processes the possibility of simultaneous address to its data. When solving applied tasks the size of the received information in most cases is so large that the possibility of verification - the detailed analysis of the data received directly by a computational program - is impossible. As there are no universal graphical packages with visualization of different isometric projections and color gammas for such situations, applied software developers are advised to start developing such packages.
global optimization of the whole computational sequence of tasks [27]. That's why a package of programs as a product used for scientific-applied purposes not only by its creators but by end users as well, should be developed at an absolutely new programming level. When developing modern software it is necessary not only to take into consideration non-linear (with feedback) relations between all the links of a calculation chain but also implement the possibility of segmenting a program at high, middle and low paralleling levels of the computational process. Segmentation is necessary for more effective usage of multi-processor systems. Besides, when developing a numerical algorithm we should coordinate the issues of accuracy and safety of the end software and also the issues of its effectiveness and portability on a concrete supercomputer's architecture. Such parallel programming differs greatly from the traditional programming, i.e. sequential programming.
The aim of any work connected with parallel programming is review of interrelations between the mathematical algorithm's structure and a multi-processor computational system's architecture. Depending on the complexity of the stated task different types of interrelations can be implemented. These interrelations are called the levels of decomposition of the source task. They can be defined as follows [31]: The first level - division of a task into subtasks. The second level - division of each separate subtask into a subset of quasi-single-type procedures executed simultaneously at different source data. In mathematical physics this parallelism type is called geometrical parallelism or data parallelism as paralleling is performed in this case by distributing calculations in different points of the computational field into different processors. The third level - paralleling of separate procedures. The fourth and the deepest paralleling level - division of arithmetical processes according to the number of processors. It is recommended not to use the last level on supercomputers with distributed memory in which for each processor local memory is allocated. The researchers of most applied tasks are advised to stop the process of their decomposition at the second level.
We can propose a thesis that loops are one of the most important program constructions with accessible parallelism. The problem of extracting fine-grained parallelism (parallelism inside loops) from these constructions is of great importance in view of increasing popularity of superscalar computers [36]. The article [37] presents algorithms for computational procedures and also results received with their help and based on the high-accuracy parallel arithmetic methodology. It is suggested that this methodology be used for solving applied tasks of linear algebra and mathematical physi s. The c mentioned work is devoted to creating algorithmic and program means of supporting accurate array computations based on complex usage of parallelism of MIMD-systems [38] and multibit arithmetic with dynamic operand length. Special attention is paid to influence of roundings in basic array operations on the accuracy of matrix tasks' calculation. The work includes the library of programs and text examples demonstrating effectiveness of the developed approach. The given results show the possibility of performing accurate array computations with simultaneous message transfer on parallel computers. The developed package of applied programs can be adapted for execution on parallel computers of different types. It is obvious that using multibit arithmetic is not typical of supercomputers. Its usage will inevitably lead to slowdown of application execution. But time loss in this case will be compensated not only by calculations with maximum usage of standard data types but also by adaptation of highly effective parallel algorithms initially suited for execution on one-processor computers to means of high-accuracy processing. Multibit arithmetic should be used only in the most "heavy" algorithm sections. But even in this case the dynamic operand length helps process only a limited number of bits. It is this way which is supposed for reaching balance between speed and accuracy of computations. The article [39] analyzes in detail the vector-pipeline architecture of supercomputers of CRAY family. As the result of the performed research, programming factors reducing supercomputers' performance were discovered. To them relate:
y y y y y
sectioning of long vector operations (increases overhead costs); overload of commands' buffers (increases overhead costs); conflicts of memory access (in case of using shared resources); limited capacity of data transfer channels (depends on the supercomputer's architecture); other factors.
It also gives examples (in program codes) showing the way out. In some of the works mentioned above the development is singled out in which the algorithm structure doesn't adapt to the computer's structure but defines its structure itself [40]. The work is intended for creating new modern computer technologies and methods of parallel programming meant for increasing effectiveness of solving fundamental scientific and applied problems in the sphere of computational modeling of aerodynamics and gas-dynamics' tasks. Special attention is paid to theoretical issues of paralleling. The work considers different methods of decomposing a full task into simultaneously executed subtasks. Of high strategic importance is the theoretical stage of investigating the problem of paralleling a program complex, that is development of principles (and concrete methods on its basis) of optimal decomposition of the whole totality of algorithms, composing the processor system and its operational environment.
The article describes three main decomposition (segmentation) types for the program complex "Thread3" planned for development in the process of preparation for paralleling the algorithms which make it up:
y y y
One of the global types of high-level structuring for the task being solved is decomposition of the investigated physical process into subprocesses composing it and, consequently, segmentation of the common algorithm of solving the full task into several algorithms of solving the subtasks composing it. A segmental algorithm of parallel calculations of physical processes is suggested and meanwhile all the module-segments of the computational core of the program are launched simultaneously. Besides, inside each segment subsegments can simultaneously start working. When paralleling computational procedures of extreme importance is synchronization and routing of data improper organization of which either leads to incorrect calculation or to large overhead costs of computer and astronomical time due to various delays of calculations because of data waiting and, consequently, downtime of the processors in some segments. It is supposed that the latter leads to nonoptimality or even impossibility to use the configurable processor space. Geometrical decomposition (segmentation) of the full task and the following parallelization of calculations allows you to significantly reduce astronomical time needed for calculations. Geometrical decomposition consists in dividing the whole integration domain into a map of subdomains (subsegments) and also in a single-step calculation of the physical process' state in each subdomain followed by joining of solutions. The article lists requirements to mathematical definition of the task permitting geometrical decomposition. Technological decomposition implies segmentation of mathematically independent tasks. There can be several levels of technological decomposition. The most typical example of decomposition is paralleling a program into certain physico-mathemetical tasks each of which can similarly consist of algorithmically independent tasks. The process of technological decomposition depends greatly on the program's structure and numerical methods used in it. Using the latter decomposition type presupposes special attention to the parallel program's effectiveness.
Conclusion
Despite obvious success in using multi-processor systems there appear debates about their low effectiveness. Increase of multi-processor systems' performance is generally determined by balance between computational operation and data exchanges on its background. Non-fulfillment of this condition is one of the causes of performance loss during paralleling with increasing number of computational program modules. Evaluation of programs' effectiveness has been carried out since first multi-processor systems transputers. Even then the first attempts were made to successfully solve the problem of maximum usage of calculation time. When solving a concrete task, first of all it is necessary to search for parallelism variants by dividing a separate task into several subtasks. After that, data parallelism (or
geometrical parallelism) can be performed, that is division of computational field. This type of parallelism means that the computational field is divided into subfields each of which is correlated to a separate system's processor. When developing real parallel programs, as a rule, high effectiveness demands many changes of the program to find the best scheme of its paralleling. Success of this search is determined by simplicity of the program's modification.
References
1. V.N. Datsuk, A.A. Bukatov, A.I. Zhegulo. Electronic user's guide on the course "Multi-processor systems and parallel programming" Part I. Introduction into programming organization and methods of multi-processor computer systems. Rostov-on-Don, 2000. 2. E.V. Neupokoev, G.A. Tarnavskiy, V.A. Vshivkov. Paralleling marching algorithms: target computational experiments. // Autometriccs, N 4, volume 38, 2002, pp. 74-87. 3. V.V. Korneev. Parallel computer systems. Moscow: "Knowledge", 1999. - 320 pp. 4. G.I. Shpakovskiy. Parallel computers' architecture. - Minsk, 1989. - 136 pp. 5. A.O. Latsis. How to build and use a supercomputer. Moscow: Bestseller Publishing house, 2003. 274 pp. 6. D.U. Labutin. System of remote access to the computational cluster (access manager): highperformance parallel computations on cluster systems. Material of the second international scientific-practical seminar, Nizhny Novgorod: Nizhny Novgorod University Publishing house, 2002. pp.184-187. 7. K.E. Afanasyev. Multi-processor computer systems and parallel programming: tutorial/ K.E. Afanasyev, S.V. Stukolov, A.V. Demidov, V.V. Malishenko; Kemerovo State University. Kemerovo: Kuzbassvuzizdat, 2003. - 182 pp. 8. S.A. Nemnyugin, O.L. Stesik. Parallel programming for multi-processor computer systems. - St. Petersburg: BHV-Petersburg, 2002. - 400 pp. 9. A.V. Demidov, K.V. Sidelnikov. Emulation of parallel data processing on a personal computer // XLI International scientific student conference "Students and Scientific-and-Technological Advance". Collection of works. Novosibirsk, 2003. pp. 110-111. 10. A.A. Samarskiy, E.S. Nikolaev. Methods of solving mesh equations. Moscow: Science, 1978. 561 pp. 11. V.V. Samofalov, A.V. Konovalov, S.V. Sharf. Dynamics and statics: searching for compromise // Works of All-Russian scientific conference "High-performance computations and their applications". Moscow, 2000. pp. 165-167. 12. S.K. Godunov, V.S. Ryabenkiy. Difference schemes. - Moscow: Science, 1973. - 400 pp. 13. V.V. Voevodin. Supercomputers: yesterday, today, tomorrow. // Collection of popular science articles "Russian science at the dawn of the new century". Under the editorship of academician V.P. Skulachov. Moscow: Scientific world, 2001. pp. 475-483. 14. V.A. Kostenko. To the question of evaluating the optimal parallelism level. // Programming. 1995, 4, pp. 24-28. 15. A.S. Antonov, V.V. Voevodin. Effective adaptation of sequential programs to the modern vectorpipeline and array-parallel supercomputers. // Programming. 1996, 4, pp. 37-51. 16. E. Sallivan. Time is money. Creating a team of software developers/Translated from English. Moscow: Publishing house "Russkaya Redaktsiya", 2002. - 368 pp.: illustrations.
17. V.A. Galatenko, K.A. Kostuhin. Debugging and monitoring of distributed heterogeneous systems. // Programming, 2002, 1. pp. 27-37. 18. V.A. Krukov, R.V. Udovichenko. "Debugging of DVM programs". Programming. - 2001. N. 3.pp.19-29. 19. A.P. Sapozhnikov, T.F. Sapozhnikova. Reengineering technology of distributed computations in the local network. Works of international conference "Distributed computations and Gridtechnologies in science and education" (Dubna, June 29 - July 2 2004). 11-2004-205, Dubna, JINR, 2004. pp.183-190. 20. V.V. Samofalov, A.V. Konovalov. Technology of debugging programs for computers with mass parallelism // "Issues of atomic science and technique". Series Mathemetical modeling of physical processes. 1996. Issue 4. pp. 52-56. 21. M.K. Valiev. Applying temporal logic to program specification. // Programming. 1998, 2, pp. 3-9. 22. V.A. Krukov. OS of distributed computer systems. (tutorial). 23. N.L. Zaharyeva, V.B. Hoziev, P.D. Shirkov. Modeling and education.// Mathematical modeling, 1999, volume 11, N 5, pp. 101-116. 24. A.A. Shalito. Automatic program projecting. Algorithmization and programming of logical control tasks. "Izvestiya akademii nauk. Teoriya i sistemi upravleniya" magazine, issue 6. NovemberDecember 2000. pp.63-81. 25. N.G. Markov, E.A. Miroshnichenko, A.V. Saraykin. Modeling of parallel software using PSnetworks. // Programming. 1995, N 5, pp. 24-32. 26. N.M. Ershov. Building graphs of computational algorithms by autotracing method. // Programming. 2000, N 6, pp. 58-64. 27. V.P. Gergel, R.G. Strongin. Fundamentals of parallel computations for multi-processor computer systems. Tutorial. Nizhniy Novgorod: Publishing house of Nizhniy Novgorod State University named after N.I. Lobachevskiy, 2000. - 176 pp. 28. V.P. Ivannikov, N.R. Kovalevskiy, V.M. Metelskiy. About minimal time of implementing distributed competing processes in synchronous modes. // Programming. 2000, N 5, pp. 44-52. 29. M.K. Valiev. Applying temporal logic to program specification. // Programming. 1998, 2, pp. 3-9. 30. O.L. Bandman. Cellular-neural models of spatio-temporal dynamics. // Programming. 1999, N 1, pp. 4-17. 31. A.E. Duysekulov, T.G. Elizarova. Using multi-processor computer systems for implementing kinetically coordinated difference schemes of gas-dynamics. // Mathematical modeling, 1990, volume. 2, N 7, pp. 139-147. 32. O.A. Dmitrieva. Parallel algorithms of numerical solution of simple differential equations. //Mathematical modeling N 5, 2000, pp. 81-86. 33. N.R. Bahvalov. Numerical methods // Main editorship of physico-mathematical literature of "Science" publishing house, Moscow, 1975. - 632 pp. 34. D.B. Moskvin, V.A. Pavlov. Experience of using MPI technology for solving a system of secondorder Fredholm integral equations. //Mathematical modeling, 2000, N 8, pp. 3-8. 35. A.N. Zavorin. Parallel solution of linear systems when modeling electric circuits. // Mathematical modeling, 1991, volume 3, N 3, pp. 91-96. 36. Ki-Chang Kim. Fine-grained paralleling of incomplete loop nests. // Programming. 1997, N 2, pp. 52-66. 37. V.A. Morozov, A.P. Vazhenin. Matrix arithmetic of multiple accuracy for parallel systems with message transfer. // Programming. 1999, N1, pp. 66-77. 38. V.V. Voevodin. Mathematical models and methods in parallel processes. - Moscow: Science, 1986. - 296 pp.
39. V.V. Voevodin. Is it easy to get the promised gigaflop? // Programming. 1995, N 4, pp. 13-23. 40. G.A. Tarnavskiy, R.I. Shpak. Decomposition of methods and paralleling of algorithms of solving aerodynamics and physical gas-dynamics tasks: computer system "Thread-3". // Programming. 2000, N 6, pp. 45-57.
Additional references.
1. A.R. Antonov. Parallel programming using MPI technology: tutorial. - Moscow: MSU publishing house, 2004. - 71 pp. 2. R.B. Berozin, V.M. Paskonov. Component system of visualizing the results of computations on multi-processor computer systems // Materials of All-Russian scientific conference "Highperformance computations and their applications", 2000, pp. 202-203. 3. N.V. Bocharov. Parallel programming technologies and technique. Review.// "Programming", 2003, N 1. PP. 5-23. UDC 681.3.06 4. V.V. Voevodin, Vl.V. Voevodin. Parallel computations. - St.Petersburg, 2002. - 600 pp. 5. V.P. Gergel, A.N. Svistunov. Development of an integrated environment of high-performance computations on cluster systems. Materials of the second international scientific-practical seminar, Nyzhniy Novgorod: Nyzhniy Novgorod publishing house, 2002. PP.78 -82. 6. V.P. Ilyin. About paralleling strategies in mathematical modeling. // Programming. 1999, N 1, pp. 41-46. 7. A.N. Karpov. Data visualization on parallel computer complexes // 15th International conference GRAPHICON-2005. Novosibirsk, Russia, June 20-24 2005. 8. N.A. Konovalov, V.A. Krukov, A.A. Pogrebtsov, U.L. Sazanov. C-DVM - the language fordeveloping mobile parallel programs. // Programming. - 1999. N1. PP. 20-28. 9. N.A. Konovalov, V.A. Krukov, R.N. Mikhailov, L.A. Pogrebtsov. Fortran-DVM - the language for developing mobile parallel programs. // Programming. 1995, N 1. PP. 49-54. 10. R.V. Popova, R.V. Sharf. Organization of saving temporary data on MBC // Theses of the report from All-Russian conference "Urgent problems of applied mathematics and mechanics" (Yekaterinburg, February 3-7 2003), pp. 62. 11. I.V. Prangishvili, R.Ya. Vilenkin, I.L. Medvedev. Parallel computer systems with shared control. Moscow: Energoatomizdat, 1983. - 312 pp. 12. L.B. Sokolinskiy. Parallel machines of databases. // Collection of popular science articles "Russian science at the dawn of the new century". Under the editorship of V.P. Skulachov. -Moscow: scientific world, 2001. PP. 484-494. 13. E. Tanenbaum. Distributed systems. Principles and paradigms. - St.Petersburg: Piter, 2003. - 877 pp. 14. M.V. Yakobovskiy, R.A. Sukov. Dynamic load balancing // Materials of "High-performance computations and their applications" conference, Chernogolovka, 2000, PP. 34 -39. 15. A.V. Komolkin, R.A. Nemnugin. Electronic tutorial "Programming for high-performance computers".