Enhancement of Industrial Ethernet Performance Using

Multicasting/VLAN techniques
Qutaiba I. Ali
Computer Eng. Dept.
Mosul University.
Mosul, Iraq
QQQ1 @maktoob.com
Abstract- This paper studies Industrial Ethernet performance
under different circumstances using the network simulation
package(OPNET).It was assumed that industrial Ethernet is
used to connect the different components of an electrical
substation automation system. In the first simulation case,
unicast transmission procedure is assumed. Unicast consumes
CPU power and resources of the industrial node. In addition, if
the packet production rate is high, queuing delay is created at
different network layers of the industrial node. All this adds
extra delay to the total value of the latency. Then, a
comprehensive study to the effect of using multicasting was
made. MulticastingNVLAN techniques are used to enhance the
real time performance of industrial Ethernet by modifying
packet generation process. Industrial nodes use multicasting
technique to generate a packet forwarded to all members of a
multicast group in the same time. Meanwhile, VLAN technique
is used to forward the multicast traffic to its intended
destinations only.

Keywords Industrial Ethernet, Latency, Multicasting,

Virtual LAN
I. INTRODUCTION

Many field bus vendors are moving forward to unify

their efforts to establish a common network for their
industrial solutions. Industrial Ethernet was chosen to be
that solution. Industrial Ethernet uses all types of the
protocols of traditional Ethernet including the Transport
Control Protocol (TCP), the Internet Protocol (IP) and the
media access and signaling technologies found in all
Ethernet networks [ 1 ].
Much attention has been focused on the use of Ethernet
technology directly at the device level. The decreasing cost
and increasing capabilities of network interfaces and
microprocessors have accelerated the movement of
communications network connections down to the
instrument and device level. The Ethernet-Based Intelligent
Electronic Device (IED) is a measurement or I/0 device
with an Ethernet connection directly on the device itself.
This approach provides a relatively inexpensive option for
networked data acquisition, and provides greater versatility
in terms of locating the measurement device in sizeconstrained areas or harsh environments [2].
11. TCP/IP AND UDP/IP USAGE IN INDUSTRIAL ETHERNET

A de facto protocol standard for network

communication is the IETF protocol suite, usually called
Transmission Control Protocol/Internet Protocol (TCP/IP).

Basil Sh. Mahmood

Electronic Eng. College
Mosul University.
Mosul, Iraq
Basil Mahmoodgyahoo.com
This suite contains two transport protocols: Transmission
Control Protocol (TCP) and User datagram protocol
(UDP). The main differences are that TCP is slow, reliable,
and connection-oriented whereas UDP is fast, unreliable,
and connectionless. In the case of high-speed measurement
data, TCP is less useful due to the protocol overhead
involved. Therefore, UDP was chosen as the measurement
data real-time protocol. This turns out to be more than
satisfactory, since measurement values is sampled at a very
high rate. Therefore, if a data set is lost, another set will be
coming along shortly [3, 4]
III. INTRODUCTION TO SUBSTATION AUTOMATION SYSTEM

A substation is a large number of switchgears

controlled, supervised and protected by a Substation
Automation System (SA). The Substation Automation
System (SA) is a network of functions realized inside
devices strongly interacting as a system. The Substation
Automation System (SA) comprises full station and bay
protection as well as control, monitoring and
communication functions and provides all functions
required for the safe and reliable operation of the
substations. A typical high voltage substation connects 3-10
transmission lines (feeder bay); two or more power
transformers (transformer bay) and has a physical dimension
of hundreds of meters [5].
For protection purposes, reaction times of the complete
system must be in the order of 4-10 msec. (1 msec. for extra
high performance systems) between fault occurrence and
circuit breaker deenergising. Other monitoring includes the
state of the electrical process, currents and voltages, status
information from circuit breakers, gas insulation units and
more[6].
Communication requirements for the substation were
implemented using different techniques, such as point-topoint and field bus systems (especially Modbus type[7]).
Industrial Ethernet was entered strongly into this field and
unified efforts of different organizations result the
development of the International Standard - IEC 61850 Communication Networks and Systems in Substations[6].
A typical layout for a substation consists of several sub
networks. Each sub network (bay) consists of the following
components[5,7,8]:
1.(8) IEDs used as sensors (Sn) to measure
different quantities at different sampling rates.

2.(3) IEDs work as actuators (ACn) for different

purposes (e.g., circuit breaker).
3.One local controller which has the following
functions:
* Connection to remote centers (SCADA system purposes)
* Automation and local control
Also, there is a global controller responsible for the
whole substation's automation and protection functions. In
addition, global controller can take the tripping actions of
any local controller in the case of its malfunction.
In the control room, there is a computer used for:
* Operator interface for monitoring and control of the entire
substation
* Maintenance interface
* System administration interface.
This is called Human Machine Interface (HMI).
IV. SIMULATING INDUSTRIAL ETHERNET
PERFORMANCE IN SUBSTATION AUTOMATION

The research tool used in this paper is OPNET

(OPtimized NETworks). It is an advanced package that
allows the user to design and study communication
networks, devices, protocols, and application [4].
An OPNET model (represents five sub networks
substation automation system) is built to test the
performance of Industrial Ethernet under hard real time
conditions (with lmSec. as the dead time). The different
parts of the model have the performance listed in table (1)
[7].
TABLE I. INDUSTRIAL ETHERNET PERFORMANCE

~~Fast Ethernet (Data Rate:1I00

Mbps)

Type
EthernetEthernet
Type

Sensors packet processing rate

5000 Packet/Sec.

Actuators packet processing rate

Local controller's packet processing

5000 Packet/Sec.
10000 Packet/Sec.

Global
controller's
processing rate

30000 Packet/Sec.

rate

packet

UTP for short distances-Fiber

optics for long distances

The traffic pattern (which represents the control

activities of the different parts of the substation automation
system) presented in references [7, 9, 10] is adopted and it
can be summarized as listed in table (2)(The detailed
description of the protection algorithms for the substation is
beyond the scope of this paper).

As listed in the table, UDP protocol is used to transfer

time critical data (measured samples) while TCP is used to
transfer administrator's configuration information to some
nodes. SCADA information is transferred periodically form
global controller and from each local controller (summery
of Bay data) to HMI in the control room for monitoring and
control purposes. The traffic in the table, represent the
traffic in one bay (sub network) and it is repeated in the
other bays.
The metrics used to evaluate the performance of
Industrial Ethernet are [7]:
*Local Latency: {the latency measured from the
application layer in the sensors to the application layer in
the local controller} + {the latency measured from the
application layer in the local controller to the application
layer in the circuit breaker}.
*Global Latency: {the latency measured from the
application layer in the sensors to the application layer in
the global controller} + {the latency measured from the
application layer in the global controller to the application
layer in the circuit breaker}.
V. COMMENTS ON THE SIMULATION RESULTS

From running the simulation model, different statistics

(which explain the behavior of the system) were collected as
follow:
A. Traffic map of the Substation Automation system
The complete traffic flow of the control data packets on
the network is shown in figure (1). The values shown in the
figure are calculated by the simulation program as follows:
By taking node (S1) as an example, the throughput (actual
data rate on a network channel) can be calculated by
multiplying the packet production rate (4030 Packet/Sec.)
by the packet size (32 Byte + headers at different network's
stack layers), which results in (3.8 Mbps).
The throughput on each segment represents low
utilization of the whole channel capacity (100 Mbps).That
means, congestion problem (nodes traffic exceed channel
capacity) is avoided and hence, Ethernet delay has a
minimal effect on the total latency. FTP traffic causes a
transit increase in the throughput values shown in the traffic
map.

ps

S2

TABLE II TRAFFIC PATTERN OF A TYPICAL SUBSTATION

AUTOMATION SYSTEM
Source
Destination
Packet / Packet
Sec.
Length
32 Byte
S 1 -S2-S3
Local Cont.- AC1-AC21000
Global Cont.
All Sensors
32 Byte
Local Cont.-AC1-AC2
10

Flp 7j..f.

0-0
S5

0~02
t:m

OSE

Local Cont.
Global Cont.

AC3
AC3's in all subnet's

HMI

S1-S2-S3-Local
Global Cont.

Local
Cont.Global Cont.

HMI

Cont.-

250
250

16 Byte
16 Byte

1 file each

1Mbyte

2 file/min.

1Mbyte

2N

ftp -*

0-----B.

L -1

SUBI

Figure 1. Traffic Map of the Reference Model

Local and Global Latency:

Figure (2) shows the change in local and global

latencies over simulation time. The following observations
could be extracted from the graphs:

*Most of the latency is concentrated inside the node's

network stack.
*Latency graphs have two regions: constant latency region
and unstable region (since the system fails to respond within
the dead line time of (lmSec.)).
Constant latency region represents the system behavior
when the traffic on the network is of the real time data only.
The value of this latency is less than (lmSec.) which
responds successfully to the real time bounds of the system.
However, this value is still relatively high, because for each
measured sample , the sensors IEDs generate four packets
addressed to four different destinations( refer to table
(2)).This gives a rise to the node's CPU utilization (see
Table(3)) and generates more queuing delay inside the
nodes, and all that increase latency.
The unstable behavior of the system shown in
Figure(2), is coming up from the highly loaded nodes that
are subjected to more traffic during the file transfer
operation which adds more delay to the packet generation
operation and hence, increased latency. File transfer
protocol (FTP) is used to handle the operation of file
transfer and it needs a maximum time of (3.5 Sec. ) to
complete its task.
*Global latency has a lower value than local latency. This is
caused by the higher processing capabilities of the global
controller compared to local controller. On the other hand,
the Ethernet delay to the global controller has an average
value of (67 [tSec.) while the average value of Ethernet
delay on the local controller is equal to (46 [tSec.) .This
assures that Ethernet delay (in this model) has a minor effect
on the total latency.
*The small peaks in the graphs were caused by the upload
operation (SCADA information transfer) from the
controllers to the control room.
*The Sensor nodes (especially S1-S3) suffer from a higher
utilization of their associated CPU more than other nodes,
See table (3)

(~~~~FTP Downioe

1.8
1 .6
1 .4

1.2

FTP Um

10.8

0.6
0.4
0.2
0

VI. MODIFYING INDUSTRIAL ETHERNET BEHAVIOR

USING MULTICASTING / VLAN TECHNIQUES

It is noted in figure (2) that latency values are still

relatively high. Most of the remaining latency comes from
the originating nodes (sensors). The packet generation
technique used by the sensors follows the unicast procedure
as follows [f1]:
1. Measured samples are converted to data bits in the
sensor's application layer and some processing tasks
are implemented on them (according to the specific
application).
2. At transport layer, multiple segments is created
depending on the number of the intended destinations.
For each segment, a transport layer header is added
with all necessary calculations.
3. Each segment is forwarded to the network layer. IP
Addresses are added in addition to all other network
layer's header parts. At this point, segments are called
datagrams.
4. Datagrams now reach the data link layer, which adds
more headers (including M\AC addresses) and then
send them out ( as packets) to the transmission
medium.
5. This sequence is true and repeated for all other copies,
but with different destination addresses.
Unicast procedure consumes CPU power and resources
of the industrial node (return to Table 3). In addition, if the
packet production rate is high, queuing delay is created at
different network layers of the industrial node. All this adds
extra delay to the total value of the latency. Unicast is
usually used when the node intends to send to a specific
destination If the node intends to send the same packet to
multiple receivers at the same time, multicasting technique
is used. Today, this technique is used by various internet
applications, such as: video conferencing, corporate
communications, distance learning, and distribution of
software, and news. The main contribution of multicasting
in these fields is to minimize channel utilization, and hence,
improves internet services characteristics [12].
Some researchers [7, 13] suggested (as a future work)
the use of multicasting to enhance the performance of the
real time systems. However, none of them studied the effect
of such a suggestion. In this paper, a comprehensive study
to the effect of using multicasting was made.
A.

50

100
Simulation Time(Sec.)

1~Global Latency

'Local

150

200

Latenco

Figure 2. Latancy Variation of the Reference Model

TABLE III. CPU UTILIZATION OF THE SUBSTATION NODES
Node
Associated CPU Utilization %l
Global Controller 60%0
Local Controller
34%0

S1-S3

83%l

AC1,AC2

31%
10%

S4-S8

AC3

1%t

Using Multicast in Industrial Ethernet

Multicast is based on the concept of grouping. A

multicast group is an arbitrary group of receivers that
expresses an interest in receiving a particular data stream.
This group has no physical or geographical boundaries the
hosts can be located anywhere on the Internet or any private
network [12].
IP multicast addresses specify a "set" of IP hosts that
have joined a group and are interested in receiving multicast
traffic designated for that particular group [12].
The Internet Assigned Numbers Authority (IANA)
controls the assignment of IP multicast addresses. IANA has
assigned the IPv4 Class D address space to be used for IP
multicast. Therefore, all IP multicast group addresses fall in
the range from 224.0.0.0 through 239.255.255.255[12].In
this paper, IP multicast was used to enhance the real time
performance of Industrial Ethernet by modifying packet
generation process. Nodes were arranged into several

multicast groups according to traffic pattern listed earlier in

table (2). Table (4) detailed this arrangement. Accordingly
when sensors generates a packet, its address is the value of
IP multicast group and therefore this packet is forwarded to
all members of the multicast group in the same time.
TABLE IV.
Multicast
Group 1
Multicast
Group 2
Multicast
Group 3
Multicast
Group 4
Multicast
Group 5
Multicast
Group 6
Multicast
Group 7
Multicast
Group 8
Multicast
Group 9
Multicast
Group 10
Multicast
Group It

MULTICAST GROUPS FOR DIFFERENT NODES

S 1 - Local Controller - AC I - AC2 - Global Controller
S2 - Local Controller - AC1 - AC2 - Global Controller

S3-Local Controller - AC1 - AC2 - Global Controller

S1 - Local Controller - AC I - AC2

DW

S4 - Local Controller - AC t

AC2

S5 - Local Controller - AC t - AC2

SUJBI

S6 - Local Controller -AC t - AC2

Figure 4. Traffic Map in Multicasting Case

S7 - Local Controller -AC t - AC2

B.

S8 - Local Controller - AC t - AC2

f*1

.M

_
=
.t~

%_

0 7

QJ 0.6

4)

05-

04

:%

S3- Local Controller - ACt - AC2

09
T_
Z-

S2 - Local Controller - AC t - AC2

The multicast groups were fed into OPNET simulation

environment. The results obtained from running the
simulation show that CPU utilization of the sensors falls
(from 83% to 20%) which decrease the delay inside those
nodes. The positive effect of multicasting technique on
latency is shown in shown in figure (3).
hUoIq

because more packets are competing on the same output

port. In the internet, the problem of flooding was solved by
using Internet Group Management Protocol Snooping
(IGMPS) which is installed on the network's routers [14]. In
the present system, routers were not used. The alternative
solution is to use virtual LAN (VLAN) technique.

03

Virtual LAN Overview

Virtual LAN is a new technique based on the

segmentation of any switched LAN into multiple logical
LANs. Nodes that share some features or belong to a certain
group can be assigned to the same VLAN. so that, they can
receive and transmit packets to the members of that VLAN
only. In other words, the single broadcast domain in the
traditional switched LANs is segmented into multiple
broadcast domains. VLAN technique provides enhanced
network features such as
simplification of network
management, controlled traffic activity and workgroup and
network security [ 1 5].
In general, there are three basic models for determining
how a packet gets assigned to a VLAN[ 1 5]:
* Port-based VLANs
t MnAC address-based VLANs
b Protocol-based VLANs (Layer 3 VLANs)
The VLAN membership of a packet is indicated by a
tag that is added to the packet. This method is defined in the
IEEE standard 802. 1Q, approved in 1998. When a packet
arrives at its local switch, the VLAN membership can be
determined as port based, M\AC address-based or protocolbased. When the packet is transferred to other switches the
VLAN membership can be defined through the tag that was
added by the first switch [15].

~~~~
0.201

50

100

150

Simulation Time (Sec.)

I-Global Latency

Local Latency

Figure 3. Latency Variation in Multicasting case

However, Multicasting causes an increase in network

traffic. The reason behind this increase belongs to the fact
that Ethernet switch do not have the ability to forward
multicast traffic to its intended destinations only. Ethernet
switch keeps a list of the physical addresses of the nodes
connected to it, if it receives any packet carried a different
destination address (including a multicast group address), it
will forward it to all other ports except the one which come
from. This is called "flooding" [13]. The effect of flooding
on the network traffic can be clearly shown in the figure (4).
In flooding, the traffic on the network is increased and
the nodes receive more traffic, that most of it does not
belong to those nodes .This causes additional load on the
node's NIC, and hence, more delay. In addition, Ethernet
delay increased (especially queuing delay in the switches)

C. Using Multicasting / VLAN in Industrial Ethernet

One of the benefits behind using VLAN technique is

the isolation of the traffic of different multicast groups. This
may solve the flooding problem mentioned earlier. In order
to investigate the effectiveness of this solution, multiple
VLANs were created according to the multicast groups
listed earlier, see table (5).

TABLE V. DIVIDING THE NODES INTO DIFFERENT VLANS

VLAN1

SI - Local Controller -AC -AC2-Global Controller-HMI

VLAN2

S2 - Local Controller - AC I - AC2 - Global Controller-HMI

VLAN3

S3-Local Controller - AC1 - AC2 - Global Controller-HMI

VLAN4

SI - Local Controller - AC I - AC2

VLAN5

S2 - Local Controller - AC t

AC2

VLAN6

S3- Local Controller - AC t

AC2

VLAN7

S4 - Local Controller - ACt

VLAN8

S5 - Local Controller - ACt - AC2-HMI

VLAN9

S6 - Local Controller - AC t - AC2-HMI

VLAN1O

S7 - Local Controller -ACt - AC2-HMI

VLAN1 1

S8 - Local Controller - AC t - AC2-HMI

VLAN12

Local Controller- Global Controller-AC3

AC2-HMI

The VLANs shown in the table above were submitted

to the OPNET environment. The available type of VLAN in
OPNET environment is port based VLAN. The change in
latency is shown in Figure (5) and the traffic map of the
network is shown in Figure (6).
08
07

OX
0.6

> 0.4x 03-

0.2 -

0
0

50

100
Simulation time (Sec.)

-Global latency

150

Local Latenr|

Figure 5. Latency Variation in Multicasting/VLAN Case

0(-(

01

VII. CONCLUSIONS
In this paper , Industrial Ethernet is studied (using
OPNET simulation package) under different circumstances
of the very hard real time constraints of a substation
automation system. The reference model of the system
assumes the use of the unicast procedures in the generation
process of the packets. Unicast consumes CPU power and
resources of the industrial node. In addition, if the packet
production rate is high, queuing delay is created at different
network layers of the industrial node. All this adds extra
delay to the total value of the latency. Multicasting
technique is used to decrease the latency by freeing the
nodes from running the unnecessary tasks. However,
flooding problem could affect on the network delay and
hence, latency. Partitioning the network to multiple VLANs
, guides the multicasting traffic to the intended destinations
only and enhance the nodes ability to receive and process
more traffic, without threatening their primary tasks

REFERENCES

Q 0.5
E

O1

As compared to Figure (4), the traffic on the network is

greatly reduced and the flooding problem was totally
removed which affects positively on the latency .The other
gain beyond using Multicasting / VLAN solution, is the
optimized use of industrial node resources, especially the
sensors CPU utilization, which was reduced to (20%).
Multicasting frees the nodes from running the unnecessary
tasks. At the same time, partitioning the network to multiple
VLANs , guides the multicasting traffic to the intended
destinations only. It is also noted that the FTP response
becomes (3 Sec. ), which reflects the enhanced abilities of
the nodes to receive and process more traffic, without
threatening their primary tasks.

--

0~~

Figure 6. Traffic Map in Multicasting/VLAN Case

200

[1] "Industrial Ethernet: A Control Engineer's Guide",

Cisco Systems, 2001.
[2] Potter , " Using Ethernet for Industrial I/0 and Data
Acquisition", National Instruments Corporation 1999.
[3] "Understanding and Evaluating Ethernet and TCP/IP
Technologies for Industrial Automation",OPTO
Company, 2001.
[4] R. Daoud ,H. Elsayed ,H. Amer and S. Eid
"Performance of Fast and Gigabit Ethernet in
Networked Control Systems," IEEE Proceedings of
International MWSCAS, 27-3 0 December 2003.
[5] Sample Specification: Substation Automation (SA),
High-Voltage Transmission Substation, 2003, available
at:www.abb.com
[6] "IEC 61850 - An Overview for Users", available at:
www. sisconet.com/downloads,2004.
[7] T. Skeie, S. Johannessen and C. Brunner, "Ethernet in
Substation Automation," IEEE Control Syst. Vol. 22,
no. 3, June 2002.
[8] D. Dolezilek , "Case Study Of A Large Transmission
And
Distribution
Substation
Automation
Project",Schweitzer
Engineering
Laboratories,
Inc.,Pullman, 1999.
[9] P. Paith ,"transmission Network Protection(Theory and
Practice)", MARCEH DEKKER publishing, 1998.
[10] A. Apostolov , " Distributed Intelligence in Integrated
Substation Protection and Control Systems", ALSTOM
Company, 1999.
[11] "Understanding TCP/IP" , available at:
www.techsupportalert.com/pdf/cO4 1 00.pdf,2000.

[12]"IP Multicast Technology Overview", Cisco systems,

2001.
[13] D. Zaniewski, "Managing Multicast Messages In
Ethernet/Ip Networks To Deliver Time-Critical Data"
,Rockwell Automation, Presented at the ODVA CIP
Networks Conference & 10th Annual Meeting,
November 16-18, 2004.
[14] A.S. Tanenbaum , "Computer Networks", PrenticeHall Publishing,2003.
[15]"The Virtual LAN", 3Com Inc, available at:
www.3com.com/nsc/200374.html,2002.