Second International Workshop on Knowledge Discovery and Data Mining
A Model of Distributed Interprocess Communication System
Xiao Peng1
Information Science& Engineering College
Dalian Polytechnic University
Dalian, China
e-mail: forkp@tom.com
Abstract—With the increasingly widespread application of the
network, a distributed operating system based on Linux cluster
has been developing rapidly. This paper presents a kernel level
distributed interprocess communication system model with
support for distributed process synchronization and
communication. This system model uses the System V
interprocess communication programming interface and
enhances it to provide functionality in distributed
environments. A key feature of this system model is the use of
the semaphore interface to support distributed synchronization
and the implementation is done at the Linux kernel level to
reduce the overhead. Finally, we realized this model
introduced in this paper in redhat Linux cluster and test result
show it is a good solution of distributed processes
communication. Compared with other similar systems, the
advantage of this system is that the implementation is done at
the Linux kernel level to reduce the overhead and System V
IPC API is easy to extend. User is easy to call the application
interface of the distributed operating system, without
considering any details of the network.
distributed system;interprocess communication;Linux kernel
I. INTRODUCTION
How to maintain the communication channels between
the distributed processes is the key technology to realize the
distributed operating systems. For software programmers in
a distributed environment, a practical distributed
interprocess communication system should have the
following features[1]. Firstly it is transparent, it must provide
programming interface for the software programmers in
distributed environments. User is easy to call the application
interface of a distributed operating system, without
considering any details of the network. Secondly, it should
support the communication of processes in distributed
environment. Finally, it should provide calling interface
such as the C Language. In view of the above three
requirements, This paper presents a kernel level distributed
interprocess communication system based on Linux cluster
with support for distributed process synchronization and
communication.
Linux system provides several process communication
mechanisms; it can be classified into two kinds: process
communication in one node, and process communication in
network environment. Process communication in one node
includes: signal, pipe and FIFO, shared memory, message
queue etc. Process communication in network environment
978-0-7695-3543-2/09 $25.00 © 2009 IEEE
DOI 10.1109/WKDD.2009.37
Li Yuanyuan2, Deng Wu 3
Software College
Dalian Jiaotong University
Dalian, China
e-mail: lyy3232312@tom.com, dw7689@163.com
only includes socket. There are still other distributed process
communication systems, as RPC and RMI[2].
Socket is very complicated to use, not only because that
there are many reduplicate codes to be written, but also
brings trouble to the maintenance. On the other side, socket
method needs IP address and port number to build TCP link,
or send UDP message, so the transparency can’t be
matched.
If we want to use RPC or RMI system, we should install it
additionally. The system needs high performance hardwares
and we must pay for the communication system. The
programmers also have to learn new transport mechanism
and how to use the transport system. RPC and RMI system
are fit for large scale enterprise application system.
So we need a new communication mechanism, which
should have simple C interfaces just like the interfaces of
process communication in a single node, and guarantee the
transparency of distributed system.
System V IPC API is one of the most commonly used
application programming interfaces for interprocess
communication. Most Unix and Linux systems provide the
ipcs command. This command lists the status of various
POSIX interprocess communication mechanisms, including
shared-memory segments. Much of the information for the
command comes from the data structure struct shmid_ds,
which is available in the /usr/include/sys/shm.h file. While
System V IPC API only works for processes running on a
single machine, we extend System V IPC application
programming interface (API) to support distributed
semaphores and Linux cluster. In this implementation the
communication mechanisms and the used methods are
hidden from the programmer.
user user user
Application
Layer process1 process2 process3
Interlayer Distributed Processes Communication system
Linux Linux Linux
Platform
hardware hardware hardware
node1 node2 network
Figure 1. Distributed Interprocess Communication System Software
Architecture Model
276
 Figure 1 shows the Distributed Interprocess
Communication System Software Architecture in this paper:
1) The system bottom is composed of Linux
computers networking each other. It consists of a group of
daemon processes in different network nodes using services
provided by the Linux operating system, which cooperate
with each other, work together to complete signal
transmission.
2) For user, the model of distributed process
communication system shielding characteristics of the
distribution. User calls the application interface of the
distributed operating system, without considering any
details of the network.
This system is working between transport layer and
application layer. The system is comprised of Name Server
and a local module in user’s node. The Name Server
provides name service for the whole system, includes:
register, log out, and name query. Every user process should
register itself before it becomes the user of our system. The
system will allocate a unique name, we call it distributed
process number named pid_d. Distributed process number is
the unique identifier of user process in the Distributed
Interprocess Communication System. When the user process
terminates from our system, it should log out from the
system and free the pid_d.
II. SYSTEM DESIGN
C/S model is the most commonly used distributed
systems architecture[3]. The Distributed Interprocess
Communication System is also based on pure TCP/IP
protocol and C/S model to provide service to user processes,
as shown in Figure 2.
user signal1
process
Distributed
IPC System
user signal2
process
In distributed system everyone should have a unique
identifier. The operating system process ID is the Pid on
Unix systems and the thread number on Windows systems,
Pid is the unique identifier of process in a single computer.
IP and port number can identify process in network
communication. While in distributed communication system
process can be identified by distributed process id named
pid_d. Once a user process will access distributed system,
distributed process id(pid_d) can be converted into a
structure of IP and pid, which can be completed by a
machine named Name Server(NS), as shown in Figure 3, the
part of dotted line represents the Distributed Interprocess
Communication System. Name Server has functionality
very similar to that of network name server: naming
resolution. The Name Server waits for incoming client
requests by listening to a specified port. Once a request is
received, the Name Server accepts a connection from the
client socket to complete the connection and implements
specific services(such as registration, cancellation and name
resolution). When several nodes are connected to one
another by a communication network, the users at different
nodes have the opportunity to exchange data and
information. Whenever a node in the distributed
communication system sends signal, the node must interact
with the Name Server to find out if the distributed process
id of receiver has been previously existed and which node it
is running on. By Name Server the messages are passed to
destination between processes of the distributed system,
such as messages are passed between processes in the
single-computer message system.
Name Sever can maintain the distributed process table of
the distributed system and provide name service for the
entire system, which includes registration, cancellation and
name resolution. User node components can receive
requests of users, which consists of daemon and a set of
signal2 user child processes. When user requests such as registration,
process
cancellation and signal transmission arrive to daemon, the
daemon creates the corresponding child process to provide
user service.
signal1 process

Figure 2. C/S Model III. SYSTEM WORKFLOW

In this paper, firstly we introduce the registration and
NS cancellation of a process in this system, and then describe a
Name Server(NS) typical communication between two distributed processes.
Figure 4 shows a registration process:
pid_d {IP, pid} 1) User process 1 running in node 1 sends a request of
registration to local_Daemon and provides its own pid.
2) Local_Daemon of node 1 creates a child process
User User
named subp_reg.
Local Daemon

Local Daemon

process 1 process 2 3) Subp_reg sends the IP address of node 1 and the

pid of user process 1 to NS_Daemon in Name Server
signum machine by network.
{pid_d,signum
}
4) NS_Daemon creates a unique pid_d for the user
signum process 1 and add the struct {pid_d, IP, pid} into a link list
node 1 node 2
which is named NS_table. NS_Daemon maintains NS_table
Figure 3. Distributed Communication System Prototype including insert, delete, and search operation, every item in

277
the NS_table records user process’s information including 4) NS_Daemon will search the relevant records in
pid_d, IP and pid. NS_table by pid_d and delete it.
5) Then NS_daemon will return pid_d to Subp_reg in 5) Then NS_daemon will return the result to
node 1 by network. Subp_ureg in node 1 by network.
6) Finally, Subp_reg return pid_d to user process 1, 6) Finally, Subp_ureg return result to user process 1,
which is the unique identifier of user process 1 in the user process 1 is separated from the Distributed Interprocess
distributed system. Communication System.
Figure 5 shows a cancellation process: Figure 6 describes a typical communication between two
1) User process 1 running in node 1 sends a request of distributed processes: A process named user process 1 send
cancellation to local_Daemon and provides it own pid_d kill signal to another process running in another machine by
and pid. the Distributed Interprocess Communication System.
2) Local_Daemon of node 1 creates another child The whole communication procedure is: user process
process named subp_ureg. register itself first, then call the signal_d ( ) function to send
3) Subp_ureg sends the structure {pid_d, IP, pid} to signal. The system queries the IP address and port number
NS_Daemon in Name Server machine by network. to Name Server by the pid_d. After receiving the response
from Name Server, the local Daemon encodes the pid of
node 1 Name Server(NS) destination process and signal value, then sends it to the
destination node. And the local Daemon on the destination
local_Daemon
NS_table node calls kill( ) to send the signal to destination process. At
last, when exit from our system, the user process log out
②fork()
④{pid_d, IP ,pid} from the Distributed Interprocess Communication System.
⑤pid_d
①pid subp_reg

⑥pid_d ③{IP, pid} IV. MAIN USER INTERFACE

NS_Daemon
1) Registration Interface: function pid_d regist( )
user processs 1
The user of the system should register before it accesses
the system, the system will add the information of user
Figure 4. Registration Workflow
process into its NS_table, as shown in Figure 4.
2) Cancellation Interface: function int uregist(pid_d)
node 1 Name Server(NS)
When user no longer send signal by the system, it must
local_Daemon release system resources by cancellation, as shown in
NS_table Figure 5.
②fork() 3) Signal Interface: function int signal_d(pid_d,sign)
Distributed processes will communicate with each other
subp_ureg ③{pid_d, IP, pid} ④{pid_d, IP ,pid}
①{pid_d, pid} by this interface. User process send signal to the destination
⑥result process by calling the signal_d( ) function, which take pid_d
⑤result NS_Daemon as parameter. Function signal_d( ) has the similar
user process 1 parameters list like kill( ), only replace pid with pid_d,
Figure 5. Cancellation Workflow pid_d hides the information about the actual location of
process, and guarantees transparency of distributed system.
Name Server(NS) User local_Daemon are responsible for receiving users’
⑤{IP, pid} requests, and provide actually signal sending service.
NS_Daemon NS_table
④pid_d
TABLE 1. TEST RESULTS OF 100 SIGNALS
③pid_d network Start time End time Communication time
node 2
node 1 ⑥{IP, pid} second microsecond second microsecond second microsecond
1188018687 622354 1188018690 725608 3 103254
local_Daemon local_Daemon 1188018723 362368 1188018726 719351 3 356983
1188018772 976347 1188018776 186132 3 209785
⑦{IP,pid, sig_kill} 1188018813 752433 1188018817 185589 3 433156
②fork() ⑧fork() 1188018879 573231 1188018882 920663 3 347432
1188018936 537861 1188018940 369478 3 831617
subp_kill ⑩success subp_remote 1188018993 332756 1188018996 516178 3 183422
1188019063 539875 1188019066 914232 3 374357
①{pid_d, sig_kill} 1188019114 283654 1188019117 548963 3 265309
⑨kill() 1188019162 316751 1188019165 735274 3 418523
Total communication time 33 523838
user process 1 user process 2 Average time 3 352384

Figure 6. Communication Between Two Distributed Processes

278
 V. IMPLEMENTATION AND TEST REFERENCES
Finally, we realized the system by GCC in redhat9 linux [1] George Coulouris, Distributed Systems:Concepts and
cluster. To improve the performance of the system and to Design 3rdn ed, New York: Addison Wesley, 2000.
make the overhead acceptable for distributed systems, the [2] William Stallings, Operating Systems:Internals and
functionality is implemented as part of the kernel. How to Design Principles 4th ed, Prentice Hall, 2001.
[3] Andrew S.Tanenbaum, Distributed Operating Systems,
measure the real time of sending signal by the Distributed Prentice Hall, 2006.
Interprocess Communication System? We designed a sender [4] E.M. Ammann, “DIPC-a monitor for distributed inter-
process and a receiver process and let the sender process process communication,” in 3rd Euromicro Workshop on
send 100 signals to the receiver every time continuously. Parallel and Distributed Processing, pp.272.
One way of measuring time is to use the Linux library call [5] A.F. Diaz , J. Ortega , F.J. Fernandez , M. Anguita , A.
gettimeofday. This method has a theoretical maximum Canas , A. Prieto, “An Efficient OS Support for
resolution of one microsecond and the time is a total of Communication on Linux Clusters,” in 2001 International
seconds from 1900-01-01 00:00:00. Table 1 shows some Conference on Parallel Processing Workshops, pp.0397.
experiment data, 100 signals transmission takes 3 seconds [6] Jin-Soo Kim , Kangho Kim , Sung-In Jung, “Building a
high-performance communication layer over virtual
on average, thus send a signal takes 0.03 seconds. interface architecture on Linux clusters,” in Proceedings
According to the result of test, we conclude that the speed of the 15th international conference on Supercomputing,
of transport is close to socket mechanism, and the system pp.30092b.
has simple call interface, can guarantee the transparency of [7] Jiong Zhao, A heavily commented Linux kernel source
distributed system. code, Machine industry publishing house, 2004.
[8] DeCao Mao,XiMing Hu, Analysis Linux kernel source,
University of Zhejiang, 2001.
VI. CONCLUSION [9] W.Richard Stevens, UNIX Network Programming
With development of computer networks, especially the Volume 2: Interprocess Communications 2nd ed, Prentice
development of the Internet, distributed systems have been Hall, 2002.
[10] Avi Silberschatz ,Peter Baer Galvin,Greg Gagne,
applied to all kinds of fields. It has played an important role
Operating System Concepts Problem Solutions (Seventh
in people’s daily life. It is necessary for a distributed system Edition ), John Wiley & Sons, Inc,2005
to offer a powerful and flexible inter process
communication function, and to effectively release and get
information in a wide area of computing environment.
Hence, interprocess communication of the distributed
computing systems becomes an important question for
discussion. Compared with other distributed IPC systems,
the system in this paper has the following features:
1) A key feature of the Distributed Interprocess
Communication System is the use of the interlayer on Linux
kernel to support distributed IPC. The implementation is
done at the Linux kernel level to reduce the overhead. It is
also easy to transplant to other UNIX systems;
2) It resolves the communication problems of
distributed processes perfectly;
3) It is based on linux cluster, which is low cost and
easy to realize. Its implementation is simple, practical and
efficient;
4) It provides the simple interface for users and
programmers.
The Distributed Interprocess Communication System can
be used as part of large scale distributed system, to provide
signal sending service. And it can also be used singly to
content the request of sending signal between different
nodes. This system provides theory and practical foundation
of the implementation of distributed system. The advantage
of this system is that the implementation is done at the
Linux kernel level to reduce the overhead and System V
IPC API is easy to extend.

279