Industrial Ethernet - Issues and Requirements: Marzio Pozzuoli

RuggedCom
Industrial Strength Networks™
Industrial Ethernet –
Issues and Requirements
Marzio Pozzuoli
RuggedCom Inc. - Industrial Strength Networks
Concord, Ontario, Canada
Presented at University of Toronto

for the
IEEE Instrumentation & Measurement Society / IEEE Communications Society
RuggedCom
Key Topics Covered…
The Emergence of Industrial Ethernet – “Ethernet everywhere!”

• Industrial automation and process control applications
• Electric power utilities – substation automation applications
• Future growth and emerging dominance of Ethernet for industrial applications
The Industrial Environment – “No place for the faint of heart!”

• Key issues and requirements for harsh industrial / substation environments
• EPRI and Rockwell studies confirm what most engineers already knew.
• IEC 61000-6-2, IEC 61850-3, IEEE P1613 Standards to the rescue
Real-time, Deterministic Performance – “Look Ma, No Collisions!”

• Technological improvements in modern Ethernet technology - facts and myths.
• IEEE 802.3x, 802.1p, 802.1Q, 802.1w, IGMP Snooping / Filtering
• Fault tolerant network architectures.
A comparison of Industrial Ethernet protocols “ Different Strokes..”

• Offerings from major industrial automation OEM’s
• Key highlights.
RuggedCom
The Emergence of Industrial Ethernet –

“Ethernet everywhere!”
RuggedCom
“Industrial Ethernet”
Ethernet on the plant floor…
RuggedCom
Ethernet – A Brief History: From back office to factory floor
Xerox PARC invention back in the 70’s!

• Original version based on CSMA/CD technology for use with multiple devices
in a bus architecture over coax cable.
• Supported 10Base5 (5Mbps) and cheaper 10Base2 (2Mbps) speeds
Dominant LAN technology at the enterprise layer in the 80’s & 90’s!
• Higher speeds (10/100/1000Mbps) help drive the PC networking revolution.
• Switching Hub (Bridge) Technology eliminates collisions and improves
performance.
Emergence of Industrial Ethernet in the new millennium.

• Major automation OEM’s begin incorporating Ethernet ports into their devices.
• All popular industrial “Fieldbus” protocols ported to work over Ethernet
• Major automation OEM’s begin to promote Ethernet for process control and
factory automation.
5
RuggedCom
All Major Automation OEM’s Have Offering…
Schneider’s
Schneider’s
Transparent
Transparent Factory
Factory EtherNet/IP
EtherNet/IP
Siemens’
Siemens’ –– ITlution
ITlution ABB’s
ABB’s -- Industrial
IndustrialIT
IT
All
All have
have one
one thing
thing in
in common
common –– Ethernet
Ethernet technology
technology on
on the
the factory
factory floor!
floor!
6
RuggedCom
Schneider Electric’s – Transparent Factory
Schneider’s
Schneider’s -- Transparent
Transparent Factory,
Factory, based
based on
on Modbus/TCP
Modbus/TCP over
over
Ethernet.
Ethernet. 7
RuggedCom
Allen-Bradley’s – EtherNet/IP
A-B’s
A-B’s -- EtherNet/IP:
EtherNet/IP: The
The “IP”
“IP” Stands
Stands for
for Industrial
Industrial Protocol.
Protocol.
ControlNet
ControlNet & & DeviceNet
DeviceNet Application
Application Layers
Layers over
over Ethernet.
Ethernet.
8
RuggedCom
“Ethernet in the
Substation!”
Even electric utilities do it…
RuggedCom
Ethernet in the Substation…Why?
Back in the 90’s North American Utilities were seeking a common
communications architecture for substation and utility automation.
• Every IED vendor supported their own, often proprietary, protocols.
• Protocol converters were required in order to integrate multi-vendor IEDs.
• Costs were high, performance was low!
Utilities and vendors under the auspices of EPRI developed the Utility
Communications Architecture (UCA2.0 / IEC 61850)
• A collection of standards to allow for a Utility communications architecture
• Supporting: multi-vendor IED interoperability, real-time control over a
substation LAN and a seamless flow of information across the entire Utility
enterprise.
Ethernet chosen as the underlying technology for UCA / IEC 61850

• Ethernet is the most prevalent local area network technology in the world.
• Large installed base: over 95% of all back office or enterprise layers.
• Non-proprietary technology with multiple vendors.
• No single IED vendor would have an advantage over the other.
10
RuggedCom
Traditional Substation
Inter
Inter IED
IED control
control signaling
signaling done
done via
via wiring
wiring or
or low
low speed
speed serial
serial communications.
communications.
Limited
Limited to
to simple
simple schemes
schemes due
due to
to cost
cost and
and complexity
complexity of
of wiring
wiring
and
and limited
limited performance
performance capabilities
capabilities of
of communications.
communications. 11
RuggedCom
Connecting IEDs via the LAN
ALL
ALL IEDs
IEDs are
are connected
connected via
via aa single
single connection
connection (or
(or dual
dual redundancy)
redundancy) to
to the
the
LAN.
LAN.
Simple
Simple or
or complex
complex control
control schemes
schemes are
are possible
possible with
with no
no increased
increased wiring
wiring
costs
costs or
or complexity.
complexity.
Other
Other real-time
real-time data
data (e.g.
(e.g. analog,
analog, status)
status) can
can be
be shared
shared across
across ALL
ALL IEDs.
IEDs.
12
RuggedCom
The LAN Based Substation
IEC 61850 (UCA)

Substation LAN PC
10/100Mbps Ethernet
E th e rn e t S w itch
Digital Relay Metering IED RTU PLC LTC
IEC 61850-9-2
‘Process Bus’ LAN
100Mbps Ethernet
Digital Modern 13
CT/VTs Switchgear
RuggedCom
GE’s Substation Vision
Substation Automation Applications (GE Energy Services)

14
RuggedCom
SEL’s Substation Vision
Substation Automation Applications (SEL Architecture)
15
RuggedCom
ABB/ALSTOM/Siemen’s Vision – IEC 61850-9-2 Substation
Typical Utility Process Bus (IEC 61850) Network (Siemens PTD)

16
RuggedCom
The Proliferation of Ethernet for Industrial/Process Control
Worldwide Shipments of Industrial IED's by Network

Interfaces (% Shipped)
30
26.3
25 2000
20 2005 20.1
18
15
11.1
10 7.78.2
9.2 8.5
6
5 4 3.5
0.1
0 I/O
t
P
t
s
e
Ne
bu
rn
e
ce
s
od
he
ot
Bu
u
i
ib
m
Et
ev
M
FF
of
Re
D
Pr
-B
A
Source: VDC “Global Markets and User Needs for Industrial Distributed/Remote I/O, Second Edition”
Ethernet is becoming “the new RS232 for Process Control.”

17
RuggedCom
Industrial Ethernet – Growth Projections
Industrial Devices with Ethernet Ports
5,000,000 4600000
4,000,000
3,000,000 2900000
2,000,000
1200000
1,000,000 718,000
0
2002 2003 2004 2005
Source: Ethernet At The Device Level Worldwide Outlook by ARC Advisory Group, June 2001
CAGR = 110% (2002 – 2005)!

18
RuggedCom
The Industrial Environment –

“No place for the faint of heart!”
RuggedCom
Industrial Strength Networks™ What is “Industrial” Ethernet…
Ethernet designed for industry…

Standard IEEE 802.3 in an industrialized design
Higher temperature ranges
Rugged and metal housing
Fan-less products that withstand vibrations
Industrial connectors and cables – CAT5E
High speed redundancy
EMC Immunity
Source: Bill King – Siemens Energy & Automation (ISA 2002 Conference Presentations)
Higher Reliability and Availability
Good qualitative definition – but the devil is in the details…
Higher temperature ranges - how much higher?
EMC Immunity – which standards and what levels?

20
RuggedCom
The Substation / Industrial Environment
Industrial Substation
• Electric Fields Phenomena

Phenomena Encountered
Encountered
• Magnetic Fields
• Electrostatic Discharge
• Conducted High Frequency Electrical Transients
• High Energy Power Surges
• Ground Potential Rise during ground faults
• Climactic Variation: Temperature & Humidity
• Seismic / Vibration
• Pollution: Dust, Metallic Particles, Condensation, Solar Radiation
21
RuggedCom
IEC 61000-6-2…the bare minimum!
IEC 61000-6-2 –Generic Immunity Requirements for Industrial Environments

CE Requirements
IEC 61000-6-2 (Immunity) Generic Immunity Standard for Industrial Environments Test Levels Pass/Fail Criteria
Enclosure Contact +/- 4kV B
IEC 61000-4-2 ESD
Enclosure Air +/- 8kV B
IEC 61000-4-3 Radiated RFI Enclosure ports 10 V/m, 80 to 1000Mhz A
Signal ports +/- 1kV @ 5kHz B
IEC 61000-4-4 Burst (Fast Transient) D.C Power ports +/- 2kV @ 5kHz B
A.C. Power ports +/- 2kV @ 5kHz B
Signal ports +/- 1kV line-to-earth B
IEC 61000-4-5 Surge D.C Power ports +/- 0.5kV line-to-earth/line B
A.C. Power ports +/- 2kV line-to-earth, +/- 1kV line-to-line B
Signal ports 10V @ 0, 5-80 MHz A
D.C Power ports 10V @ 0, 5-80 MHz A
IEC 61000-4-6 Induced (Conducted) RFI
A.C. Power ports 10V @ 0, 5-80 MHz A
Earth ground ports 10V @ 0, 5-80 MHz A
IEC 61000-4-8 Magnetic Field Enclosure ports 30 A/m @ 50, 60 Hz A
IEC 61000-4-11 Voltage Dips & Interrupts A.C. Power ports >95% reduction for 250 periods C
* Performace criterion A refers to continuous operation of the DUT as intednded during, and after test. Performance criterion B refers to spontaneous
recovery after the test with no loss of function or operational performance. Performance criterion C refers to allowable temporary loss of function with
recovery through the device controls.
IEC
IEC 61000-6-2
61000-6-2 required
required for
for CE
CE mark
mark compliance
compliance for
for electronic
electronic equipment
equipment
(e.g.
(e.g. PLC’s)
PLC’s) in
in industrial
industrial environments.
environments.
IEC
IEC 61000-6-2
61000-6-2 should
should be
be the
the minimum
minimum requirement
requirement for
for networking
networking
equipment
equipment in
in industrial
industrial environments.
environments. 22
RuggedCom
What are industrial (factory floor) users specifying?
The following excerpt was taken from an RFP specification document

from a major manufacturer who is intent is to use Ethernet on the factory
floor:
Required and Desired Features of Industrial Ethernet Switches

1. The primary requirement for an industrial Ethernet switch is
that it be environmentally hardened to operate under the same
extremes in operating conditions (temperature, vibration,
humidity, etc.) as an industrial PLC (Programmable Logic
Controller).
Industrial
Industrial users
users are
are often
often taking
taking the
the common
common sense
sense approach
approach inin specifying
specifying
that
that the
the Ethernet
Ethernet networking
networking equipment
equipment be be us
us robust
robust as
as the
the IED’s
IED’s
connecting
connecting to
to it.
it.
23
RuggedCom
Industrial Strength Networks™ EMI Phenomena - Frequency of Occurrence
Continuous
Continuous Transient
Transient Phenomena
Phenomena Transient
Transient Phenomena
Phenomena
Phenomena
Phenomena (High
(High Occurrence)
Occurrence) (Low
(Low Occurrence)
Occurrence)
•• Radiated
Radiated RFI
RFI •• Electrostatic
Electrostatic Discharge
Discharge •• Power
Power Frequency
Frequency Variation
Variation
•• Induced
Induced RFI
RFI •• Voltage
Voltage Dips
Dips •• Power
Power System
System Faults
Faults
•• Power
Power freq.
freq. Magnetic
Magnetic Field
Field •• Lightning
Lightning •• Short
Short Duration
Duration Power
Power Freq.
Freq.
•• Slow
Slow Voltage
Voltage Variations
Variations •• HV
HV Switching
Switching by
by Isolators
Isolators Magnetic
Magnetic Fields
Fields
•• Harmonics,
Harmonics, Interharmonics
Interharmonics •• Reactive
Reactive Load
Load Switching
Switching
•• Ripple
Ripple on
on d.c.
d.c. power
power supply
supply
•• Power
Power Frequency
Frequency Voltage
Voltage
Devices in industrial / substation environments must deal with a

combination of EMI phenomena which are continuous and transient.
24
RuggedCom
Industrial Strength Networks™ Devices are ‘Hardened’ for the Substation
Substation LAN
Digital Relay Metering IED RTU PLC LTC
Type Test Standards

For Substation Equipment:
IEC 60255-x
IEEE C37.90.x
Devices connected to the substation LAN are specifically ‘Hardened’

for the substation environment.
What about the Ethernet LAN? 25
RuggedCom
Industrial Strength Networks™ IEC 61850-3 –Communications networks and systems in substations
New IEC Standard (Jan/2002)

“Communications networks and systems
in substations”
Reliability EMC Environmental

26
RuggedCom
Industrial Strength Networks™ IEC 61850-3 (61000-6-5): Apparatus Port Definitions …
Power
Power Ports
Ports Enclosure
Enclosure Port
Port Signal
Signal Ports
Ports
(P/S In) APPARATUS (Local)

(Field)
E th ern et S w itch
(To HV)
(P/S Out)
(Telecom)
Functional
Functional Earth
Earth
Port
Port
(Other than Safety)
A port is defined as a “particular interface of the specified apparatus

with the external electromagnetic environment”
Type tests are defined and assigned for each specific port type.
27
RuggedCom
Industrial Strength Networks™ IEC 61850-3 (61000-6-5): Substation Definitions
CONTROL BUILDING (Switch Yard)

High Voltage
Equipment
SHIELDED AREA
(if any) PROTECTION
TELECOM IED KIOSK
Power ROOM (L) (P) (L)
Line
Carrier
(F) (H)
(T) (F) (F)
Remote Earth Earth Network
Signal Port Connection Ethernet Sw itch Ethernet Sw itch
Types
• Local (L)
• Field (F) Typical Locations of
• HV Equipment (H) Substation Ethernet Equipment
• Telecom – Power Line Carrier (T)
• Protected (P)
28
RuggedCom
IEC 61850-3…EMI Immunity Requirements
UTILITY IEC 61850-3 (61000-6-5) Communications Networks and Systems In Substations (Jan 2002)
TEST Description Test Levels Severity Levels

Enclosure Contact +/- 6kV 3
IEC 61000-4-2 ESD
Enclosure Air +/- 8kV 3
IEC 61000-4-3 Radiated RFI Enclosure ports 10 V/m 3
Signal ports +/- 4kV @ 2.5kHz x
D.C. Power ports +/- 4kV 4
IEC 61000-4-4 Burst (Fast Transient)
A.C. Power ports +/- 4kV 4
Earth ground ports3 +/- 4kV 4
Signal ports +/- 4kV line-to-earth, +/- 2kV line-to-line 4
IEC 61000-4-5 Surge D.C. Power ports +/- 2kV line-to-earth, +/- 1kV line-to-line 3
A.C. Power ports +/- 4kV line-to-earth, +/- 2kV line-to-line 4
Signal ports 10V 3
D.C Power ports 10V 3
IEC 61000-4-6 Induced (Conducted) RFI
A.C. Power ports 10V 3
Earth ground ports3 10V 3
IEC 61000-4-8 Magnetic Field Enclosure ports 40 A/m continuous, 1000 A/m for 1 s N/A
D.C. Power ports 30% for 0.1s, 60% for 0.1s, 100% for 0.05s N/A
IEC 61000-4-29
Voltage Dips & Interrupts 30% for 1 period, 60% for 50 periods N/A
A.C. Power ports
IEC 61000-4-11 100% for 5 periods, 100% for 50 periods2 N/A
Signal ports 2.5kV common, 1kV differential mode @ 1MHz 3
IEC 61000-4-12 Damped Oscillatory D.C. Power ports 2.5kV common, 1kV differential mode @ 1MHz 3
A.C. Power ports 2.5kV common, 1kV differential mode @ 1MHz 3
Signal ports 30V Continous, 300V for 1s 4
IEC 61000-4-16 Mains Frequency Voltage
D.C. Power ports 30V Continous, 300V for 1s 4
IEC 61000-4-17 Ripple on D.C. Power Supply D.C. Power ports 10% 3
Issued
Issued January
January 2002
2002 –– in
in recognition
recognition of
of the
the proliferation
proliferation of
of Ethernet
Ethernet in
in the
the
Substation.
Substation.
More
More tests
tests and
and higher
higher test
test levels
levels than
than IEC
IEC 61000-6-2.
61000-6-2.
Reflects
Reflects the
the substation
substation environment.
environment. 29
RuggedCom
Industrial Strength Networks™ IEC 61850-3 (61000-6-5): Performance Criteria
IEC 61000-6-5 Table 7 - Performance Criteria for the most relevant functions
(in descending order of criticality)
Functional requirements versus electromagnetic phenomena

Functions Transient phenomena Transient phenomena
Continous phenomena
with high occurance with low occurance
Protection and teleprotection
On-line processing and
regulation
Metering
Command and Control Short delay
Supervision ** No Delays or Data Loss ** Temporary loss, self

recovered
Man-machine interface Stop and reset
Alarm Short delay, temporary wrong indication
Data transmission and No loss, possible bit
Temporary loss
telecommunication error rate degradation
Data acquisition and storage Temporary degradation
Measurement Temporary degradation, self recovered
Off-line processing Temporary degradation Temporary loss and reset
Passive monitoring Temporary degradation Temporary loss

Self-diagnosis Temporary loss, self recovered
30
RuggedCom
IEEE P1613 – Coming soon…
IEEE
IEEE working
working on
on North
North American
American equivalent
equivalent of
of IEC
IEC 61850-3.
61850-3.
31
RuggedCom
IEEE P1613…EMI Immunity Requirements
IEEE P1613 – Draft Standard Environmental Requirements for Communications Devices Installed in Electric
Power Substations
TEST Description Test Levels Severity Levels
Enclosure Contact +/- 8kV N/A
IEEE C37.90.3 ESD
Enclosure Air +/- 15kV N/A
IEEE C37.90.2 Radiated RFI Enclosure ports 35 V/m N/A
Signal ports +/- 4kV @ 2.5kHz N/A
D.C. Power ports +/- 4kV N/A
IEEE C37.90.1 Fast Transient
A.C. Power ports +/- 4kV N/A
Earth ground ports3 +/- 4kV N/A
Signal ports 2.5kV common mode @ 1MHz N/A
IEEE C37.90.1 Oscillatory D.C. Power ports 2.5kV common & differential mode @ 1MHz N/A
A.C. Power ports 2.5kV common & differential mode @ 1MHz N/A
Signal ports 2kVac N/A
IEEE C37.90 Dielectric Strength D.C. Power ports 2kVac N/A
A.C. Power ports 2kVac N/A
Borrows
Borrows heavily
heavily from
from existing
existing IEEE
IEEE C37.90.x
C37.90.x standards
standards for
for Protective
Protective
Relaying
Relaying devices.
devices.
Fewer
Fewer tests
tests than
than IEC
IEC 61850-3
61850-3 but
but with
with test
test levels
levels just
just as
as high
high and
and higher
higher in
in
the
the case
case of
of Radiated
Radiated RFI:
RFI: 35V/m
35V/m v.s.v.s. 10V/m
10V/m !!
Defines
Defines two
two classes
classes of
of devices:
devices:
Class
Class 11 devices
devices allow
allow communications
communications errors
errors or
or loss
loss during
during EMI
EMI type
type tests.
tests.
Class
Class 22 devices
devices allow zero
zero communications
communications errors
errors or
or loss
loss during
during EMI
EMI type
type tests.
tests.
32
RuggedCom
Industrial Strength Networks™ EPRI Tests: EMI Susceptibility of STP Copper Cables
EPRI (AEP) 1997 EMI Immunity Testing
Transient IEC 1000-4-4

Generator E.F.T.
Copper
CAT5 Cable
Coupling
Coupling
Data Frames
33
RuggedCom
Industrial Strength Networks™ The EPRI Tests (2002): A closer look…
Transient
Generator IEC 61000-4-4
Electrical Fast
Copper CAT5 Cable Transients
Coupling
Coupling Medium
Medium
Data Frames
Media Media
Converter Converter
Fiber Optical
Cable
Network Analyzer
Simulator
(SmartBits)
•Electrical Fast Transients (IEC 61000-4-4) Applied to CAT5 cable

•Resultant Frame Loss:
• 32% @ 1kV Unacceptable Performance For
• 66%@ 2kV Real-Time Control!
• 75% @ -2kV 34
RuggedCom Real-time Performance - An Illustrative Example During
a Power System Fault...
Power System Fault

Current Waveform
τ = 16.7ms τ = 16.7ms
• Fault period will be period of:

• High levels of transient EMI phenomena!
• Sub-cycle (i.e. ≤ 16.67ms) processing by IEDs and critical
communications performance by the LAN in ( < 4ms)!
• Can’t afford frame errors, delays or loss of
communications!
35
RuggedCom
Industry Studies…confirm what most of us already knew!
Rockwell Automation - Study CMR of CAT-5 Cable

IEC 61000-4-6 (Induced RFI) SmarBits 600
Communications Tester
AC Mains
EUT
AC Signal
Port
Fiber Optical
DC Connections
Metallic DC Power Source
Plane
(grounded)
CDN-AF2 Power
Schaffner NSG2070-1 EUT Grounding
150KHz – 80MHz Connection
Non-Metallic / Non-Conductive Table

Ground
Strap
80 cm
Metallic Ground Plane
•Induced RFI (IEC 61000-4-6) Applied to CAT5 cable to test CMR.

•Common mode noise coupling will occur via adjacent cabling
•Resultant Bit Error Rate:
• 22% @ 10Vrms (noise coupled)!
•Error rates of this magnitude render the network useless! 36
RuggedCom
Real-time, Deterministic Performance –

“Look Ma, No Collisions!”
RuggedCom
Key advances in Ethernet technology for real-time

control!
IEEE 802.3x Full Duplex Operation
IEEE 802.3p Priority Queuing
IEEE 802.3Q VLAN
IEEE 802.3w Rapid Spanning Tree
IGMP Snooping (Multicast Filtering)
••Modern
Modern Ethernet
Ethernet technology
technology is
is well
well suited
suited for
for real-time
real-time control!
control!
••The
The age
age old
old knock
knock against
against Ethernet
Ethernet ofof “lack
“lack of
of deterministic
deterministic performance”
performance”
is
is more
more aa red
red herring
herring than
than reality
reality when
when it
it comes
comes toto modern
modern Ethernet.
38
Ethernet.
RuggedCom
Look ma no collisions…
Q. How do you make Ethernet “deterministic” or how do you
prevent collisions from occurring on an Ethernet network?
Ans. You use an Ethernet switching hub (i.e. Switch) with full-
duplex ports – IEEE 802.3x!
S W I T C H
Tx Rx Tx Rx Tx Rx
Full-Duplex
Ports
IED
IED IED
IED IED
IED
#1
#1 #2
#2 #N
#N
Ethernet Switches with full-duplex ports don’t have collisions!

39
RuggedCom
Industrial Strength Networks™ Minding your “p’s” and “Q’s”
Standard Ethernet Frame – IEEE 802.1
SYNCH MAC MAC LENGTH

DESTINATION SOURCE OR DATA FCS
PREAMBLE ADRESSS ADRESSS TYPE
Ethernet Frame with new VLAN – IEEE 802.1Q & Priority IEEE 802.1p Tag.
SYNCH MAC MAC 802.1p 802.1Q LENGTH

DESTINATION SOURCE VLAN OR DATA FCS
PREAMBLE ADRESSS ADRESSS PRIORITY ID TYPE
IEEE 802.1Q VLAN Tag
•• Allows
Allows priority
priority tagging
tagging of
of mission
mission critical
critical frames
frames –– 802.1p
802.1p
•• Allows
Allows isolation
isolation &
& grouping
grouping of
of IEDs
IEDs into
into virtual
virtual LANs
LANs –– 802.1Q
802.1Q VLAN
VLAN
40
RuggedCom
Industrial Strength Networks™ Minding your “p’s” and “Q’s”
Q. How do I make sure during periods of heavy traffic that

critical messages make to it through without delay?
Ans. You tag them with a higher priority – IEEE 802.1p.
High-Priority Queue
In-coming Frames
Priority
Port Transmit Queue
Queue
Mapping
Low-Priority Queue Inside the Switch

(per port)
IEEE 802.1p allows prioritization for critical messages! 41

RuggedCom
An IEEE 802.1p Priority example…
High-Priority Queue
In-coming Frames
Priority
Port Transmit Queue
Queue
Mapping
Low-Priority Queue Inside the Switch

(per port)
•• “Real-time”
“Real-time” control
control packets
packets can
can be
be assigned
assigned to
to higher
higher priority.
priority.
•• Ensures
Ensures real-time
real-time control
control messages
messages will
will get
get through
through even
even during
during network
network congestion.
congestion.
•• Priority
Priority can
can be
be ‘tag
‘tag header’
header’ based
based or
or MAC
MAC address
address or
or port
port based
based for
for legacy
legacy IEDs.
IEDs.
42
RuggedCom
Minding your “p’s” and “Q’s”
Q. How do you segregate network traffic between devices used

for data collection and those utilized for real-time control?
Ans. You assign them to different VLAN’s – IEEE 802.1Q.
Substation
Substation
Computer
Computer
S W I T C H
VLAN
VLAN VLAN
VLAN
11 22
IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED
11 22 33 44 55 66 77 88
Protective
Protective Relaying
Relaying IEDs
IEDs Data
Data collection
collection IEDs
IEDs
IEEE 802.1Q allows segregation of devices into VLAN’s !43

RuggedCom
An IEEE 802.1Q VLAN example..
HMI
Computer
S W I T C H
VLAN
VLAN VLAN
VLAN
11 22
IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED
11 22 33 44 55 66 77 88
PLC
PLC IEDs
IEDs Data
Data collection
collection IEDs
IEDs
•• Traffic
Traffic from
from data
data IEDs
IEDs in
in VLAN
VLAN 22 isolated
isolated from
from IEDs
IEDs in
in VLAN
VLAN 11
•• HMI
HMI Computer
Computer can
can communicate
communicate to
to both
both
•• VLAN
VLAN can
can be
be ‘tag
‘tag header’
header’ based
based or
or MAC
MAC address
address or
or port
port based
based for
for legacy 44
legacy IEDs.
IEDs.
RuggedCom
IGMP – Management for Multicast Messaging..
Substation
Substation
Computer
Computer
S W I T C H
Multicast Traffic
IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED IED
IED
11 22 33 44 55 66 77 88
Producer
Producer IEDs
IEDs Consumer
Consumer IEDs
IEDs
•• Multicast
Multicast Traffic
Traffic from
from Sensor
Sensor IEDs
IEDs can
can be
be assigned
assigned by
by the
the Switch
Switch to
to specific
specific
Consumer
Consumer IEDs.
IEDs.
•• For
For example,
example, Multicast
Multicast Traffic
Traffic from
from the
the Producer
Producer IEDs
IEDs 1, 1, 2,
2, 33 &
& 44 can
can be
be assigned
assigned
only
only to
to the
the consumer
consumer IEDs
IEDs which
which require
require itit (e.g.
(e.g. IEDs
IEDs 66 & & 8).
8).
45
RuggedCom
How Real-Time can an Ethernet network be?
“Industrial Grade” Ethernet Switch
100m 10/100Mbps
10/100Mbps
RJ45
RJ45 Ports
Ports
Sensor IED’s that multicast their

respective sensor value every 1ms
(‘producer’ IED’s)
PLC
PLC is the
‘consumer’ IED
46
RuggedCom
Bandwidth Example – Calculations
Given X number of producer IED’s and one consumer IED:
1. Port Bandwidth of consumer device (On uplink port) = BWConsumer (Mbits/second)
2. Frame size generated by producer IED’s = FS (in bits)
3. IED Period of reporting by producer devices =TProducer (ms)
Given a 16 port full-duplex 100Mbps Ethernet Switch with:
BWConsumer = 100Mb/s
The maximum number of possible frame bits @ 100% utilization of this port is defined as:
BWMAX = 100,000,000 bits/second
(This number represents total traffic including CRC and preamble)
For this analysis assume the following:
The frame size generated by producer devices , FS=64 bytes in length.
The period of IED reporting, Treporting = 1 ms (i.e. 1000 times / second)
Therefore, it can be calculated that each producer device will consume:
64 bytes x 8 bits/byte x 1000/s = 512kbits/s of bandwidth.
Therefore one can conclude the following:
Each IED consumes (512,000)/(100,000,000) = 0.512 % of bandwidth each second
(100,000,000) / (512,000) = 195. This implies 195 producer devices can be used before the
maximum channel BW is reached. At this point channel flow control would kick in to relieve the
pressure.
47
RuggedCom Bandwidth Calculations for Different Reporting
Periods and Frame Sizes
Effect of multi-cast messaging-based data collection on Uplink Bandwidth

Frame Consumer IED Uplink Percent Max # Possible
Size Reporting Bandwidth Utilization of Producer IEDs
(Bytes) Period Available Consumer (See Note)
(ms) (Mb/s) Uplink
(%)
64 1 100 0.512 195
64 0.5 100 1.024 97
64 0.25 100 2.048 48
64 0.1 100 5.12 19
128 1 100 1.024 97
256 1 100 2.048 48
512 1 100 4.096 24
128 0.5 100 2.048 48
256 0.25 100 8.192 12
512 0.1 100 40.96 2
Bandwidth
Bandwidth scales
scales linearly
linearly allowing
allowing for
for aa wide
wide variety
variety of
of possible
possible
configurations.
configurations.
Support
Support of
of IGMP
IGMP (Internet
(Internet Group
Group Management
Management Protocol)
Protocol) allows
allows for
for
multicast
multicast message
message filtering
filtering and
and producer-subscriber
producer-subscriber groupings.
groupings. 48
Equivalent
Equivalent Layer
Layer 22 protocol
protocol is
is –– GMRP.
GMRP.
RuggedCom
Network Architectures for

Connecting IEDs on the
Factory Floor
SWITCH
IED
IED
(Intelligent Electronic Device)
IED
IED
IED
Meter
Sensors
Remote
Relay
RTU
HMI
PLC
IEDIED
Meter
Sensors
Remote I/O
Relay
RTU
HMI
PLC
IED
IED
IEDIED
IEDIED
IED
I/O
RuggedCom
CASCADING ARCHITECTURE
•• Cost-effective
Cost-effective bus
bus architecture
architecture –– messages
messages cascade
cascade from
from switch
switch to
to switch.
switch.
•• Maximum
Maximum number
number of
of “hops”
“hops” (N)
(N) is
is determined
determined by
by worst
worst case
case latency
latency
requirements.
requirements.
1 2 3 4 N
SWITCH SWITCH SWITCH SWITCH SWITCH
IED IED IED IED IED IED IED IED IED IED
IED IED IED IED IED
IED IED IED IED IED
•• Typical
Typical Switch
Switch Latency
Latency =5us
=5us ++ Frame
Frame Time
Time (100Mbps
(100Mbps Port)
Port)
•• Therefore,
Therefore, for
for aa 64
64 byte
byte frame
frame (512
(512 bits):
bits): the
the frame
frame time
time == 5.12us
5.12us (100Mbps)
(100Mbps)
•Total
•Total Latency
Latency per per switch
switch “hop”
“hop” == 5us
5us ++ 5.12us
5.12us == 10.12us
10.12us
•For
•For NN == 10
10 the
the worst
worst case
case latency
latency would
would be:
be: 10
10 xx 10.12us
10.12us == 101.2us.
101.2us.
50
RuggedCom
RING ARCHITECTURE
N
SWITCH
PATH PATH
1 2
IED IED
IED
IED
Fault
Fault
1 2 3 4 N-1
IED IED IED IED ••Fault

Fault Tolerant
IED IED Ring Architecture!
Tolerant Ring Architecture!
IED IED IED IED
IED IED IED IED IED
IED IED IED IED IED
••Rapid
Rapid (i.e.
(i.e. ms)
ms) Reconfiguration
Reconfiguration via
via IEEE
IEEE 802.1w
802.1w Rapid
Rapid Spanning
Spanning Tree!
Tree!
••Typical
Typical reconfiguration
reconfiguration times
times << 50ms
50ms for
for ring
ring with
with 10
10 switches.
switches. 51
RuggedCom
STAR ARCHITECTURE
N
SWITCH
1 2 3 4 N-1
IED IED IED IED IED
IED IED IED IED IED
•• Low-Latency
Low-Latency Architecture
Architecture –– Any
Any IED
IED to
to IED
IED communications
communications requires
requires
only
only two
two ‘hops’.
‘hops’.
52
RuggedCom
Industrial Strength Networks™ Fault Tolerant Hybrid (Star/Ring) ARCHITECTURE
Fault
Fault
Fault
Fault
N N-1
SWITCH SWITCH
Fault
Fault
1 2 3 4 N-2
IED IED IED IED IED
IED IED IED IED IED
••Low-Latency,
Low-Latency, Fault
Fault Tolerant
Tolerant Architecture.
Architecture.
••Able
Able to
to tolerate
tolerate failure
failure of
of uplink
uplink or
or backbone
backbone switches.
switches. 53
RuggedCom
High Redundancy Architecture via IED’s with Dual Ethernet Ports
Fault
Fault
Fault
Fault
SWITCH SWITCH SWITCH SWITCH
Fault
Fault
IED IED
IED
SWITCH
SWITCH
IED
SWITCH
SWITCH
IED
Fault
Fault
IED
IED
IED IED
IED
SWITCH
IED
SWITCH
SWITCH
SWITCH
IED
IED
IED
Fault
Fault
IED IED
IED
SWITCH
IED
SWITCH
SWITCH
IED
IED
IED
••Able
Able to
to tolerate
tolerate multiple
multiple fault
fault types!
types!
••Rapid
Rapid (i.e.
(i.e. ms)
ms) Reconfiguration
Reconfiguration via
via IEEE
IEEE 802.1w
802.1w Rapid
Rapid Spanning
Spanning Tree!
Tree!
IED IED
IED
SWITCH
IED
SWITCH
SWITCH
SWITCH
IED
54
IED
IED
RuggedCom
A comparison of Industrial Ethernet protocols -

“ Different strokes for different folks!”
RuggedCom
7 Layer OSI Model
7 Layer OSI Model
Application
Presentation
Session Software
Transport
Network
Data Link
Hardware
Physical
•Used to describe any network or protocol.

•Helps us compare “apples to apples”.
56
RuggedCom
Different strokes for different folks…
The Substation Automation Sector The Process Automation Sector
IEC61850(UCA2) ProfiNet Modbus/TCP

DNP3/TCP Modbus/TCP Application Fieldbus HSE EtherNet/IP
Presentation
Session
TCP/UDP TCP/IP Transport TCP/UDP TCP/IP

IP Network IP
IEEE 802.1 Data Link IEEE 802.1

Ethernet Ethernet
IEEE 802.3 Physical IEEE 802.3
• Bottom half is pretty much consistent. • Bottom half is pretty much consistent.
• Application Layer is dominated by a • Application Layer is still being hotly

few players. Trend is towards 61850. contested by major OEM’s.
57
RuggedCom
Industrial Strength Networks™ Fieldbus Protocols Over Ethernet
Rockwell’s – EtherNet/IP Schneider’s – Modbus/TCP

• Released by ControlNet International and ODVA. The “IP” • Released in March 1999 “Open Modbus/TCP Specification”
stands for “Industrial Protocol” • Uses the Modbus Application Layer (MBAP) bundled with
• Uses the CIP (Control & Information Protocol) Application TCP/IP
Layer from ControlNet and DeviceNet. • PROS:
• PROS: • TCP and Modbus are widely supported
• Supports Object Oriented communications model • Easy to implement
• Supports publisher-subscriber model • CONS:
• CONS: • Does not support object oriented communications
• Difficulty in supporting legacy A-B protocols. model (i.e. you still need to know register addresses
of every device!)
• Does not support publisher-subscriber model
Siemen’s (PTO) – ProfiNet Foundation Fieldbus’ – HSE

• Released by Profibus Trade Organization • HSE = High Speed Ethernet - using 100Mbps Ethernet
• Based on Microsoft’s DCOM protocol • Uses the Application Layer from their H1 standard with
• PROS: TCP/IP (UDP)
• DCOM supports Object Oriented communications • PROS:
• Major vendor backing (Siemens) • Supports Object Oriented Communications Model
• CONS: • Supports Publisher Subscriber model
• Does not support publisher-subscriber model • Supports Network Management
• Poor real-time control? • Major Vendor Support
• CONS:
• Limited support outside process control sector
Source: ISA Training Institute – “Is Ethernet Ready for the Plant Floor” online seminar.
No convergence (i.e. One protocol) on the horizon for industrial/process

58
automation sectors…yet!
RuggedCom
Can’t we all just get along…
Q. How do you get multiple devices which communicate using
different protocols to coexist on a common network?
Ans. You use Ethernet!
S W I T C H
RTU
RTU Relay
Relay Meter
Meter PLC
PLC
(DNP3)
(DNP3) (UCA2/IEC61850)
(UCA2/IEC61850) (Modbus)
(Modbus) (Fieldbus)
(Fieldbus)
Ethernet allows multiple protocols to coexist on the same network!

59
RuggedCom
Key Takeaways…
Industrial Ethernet requires network devices (e.g. Switches) to operate
reliably in industrial environments.
• IEC 61850-3, IEEE P1613 specifically define EMI and Environmental
requirements for substation environments.
• IEC 61000-6-2 should be minimum requirement for EMI immunity in industrial
environments.
• Other requirements, depending on industry sector may be Class 1 Division 2
(Petrochem), IP67/NEMA 4x (Mining).
For real-time control applications Industrial Ethernet devices and

network equipment should support:
• IEEE 802.3x Full-Duplex operation for collision free operation
• IEEE 802.1p Prioritization to allow real-time critical messages to get through.
• IEEE 802.1Q VLAN to allow isolation of critical IEDs from non-critical IEDs .
• IEEE 802.1w Rapid Spanning Tree to allow fault-tolerant ring architectures
with rapid (i.e. <50ms) reconfiguration times.
• IGMP Snooping / Multicast filtering to prevent multicast intensive protocols
(e.g. EtherNet/IP) from inundating non participating IEDs on the LAN.
60
RuggedCom
Thank you.

Industrial Ethernet - Issues and Requirements: Marzio Pozzuoli

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Industrial Ethernet - Issues and Requirements: Marzio Pozzuoli

Încărcat de

Drepturi de autor:

Formate disponibile

RuggedCom

Industrial Strength Networks™

Presented at University of Toronto

The Emergence of Industrial Ethernet – “Ethernet everywhere!”

The Industrial Environment – “No place for the faint of heart!”

Real-time, Deterministic Performance – “Look Ma, No Collisions!”

A comparison of Industrial Ethernet protocols “ Different Strokes..”

The Emergence of Industrial Ethernet –

Xerox PARC invention back in the 70’s!

Emergence of Industrial Ethernet in the new millennium.

Ethernet chosen as the underlying technology for UCA / IEC 61850

IEC 61850 (UCA)

Digital Relay Metering IED RTU PLC LTC

Substation Automation Applications (GE Energy Services)

Substation Automation Applications (SEL Architecture)

Typical Utility Process Bus (IEC 61850) Network (Siemens PTD)

Worldwide Shipments of Industrial IED's by Network

Ethernet is becoming “the new RS232 for Process Control.”

Industrial Devices with Ethernet Ports

CAGR = 110% (2002 – 2005)!

The Industrial Environment –

Ethernet designed for industry…

Higher Reliability and Availability

Good qualitative definition – but the devil is in the details…

Higher temperature ranges - how much higher?

EMC Immunity – which standards and what levels?

• Electric Fields Phenomena

IEC 61000-6-2 –Generic Immunity Requirements for Industrial Environments

The following excerpt was taken from an RFP specification document

Required and Desired Features of Industrial Ethernet Switches

Devices in industrial / substation environments must deal with a

Digital Relay Metering IED RTU PLC LTC

Type Test Standards

Devices connected to the substation LAN are specifically ‘Hardened’

New IEC Standard (Jan/2002)

Reliability EMC Environmental

(P/S In) APPARATUS (Local)

(Other than Safety)

A port is defined as a “particular interface of the specified apparatus

CONTROL BUILDING (Switch Yard)

Signal Port Connection Ethernet Sw itch Ethernet Sw itch

TEST Description Test Levels Severity Levels

(in descending order of criticality)

Functional requirements versus electromagnetic phenomena

Supervision ** No Delays or Data Loss ** Temporary loss, self

Off-line processing Temporary degradation Temporary loss and reset

Passive monitoring Temporary degradation Temporary loss

EPRI (AEP) 1997 EMI Immunity Testing

Transient IEC 1000-4-4

•Electrical Fast Transients (IEC 61000-4-4) Applied to CAT5 cable

Power System Fault

• Fault period will be period of:

Rockwell Automation - Study CMR of CAT-5 Cable

Non-Metallic / Non-Conductive Table

Metallic Ground Plane

•Induced RFI (IEC 61000-4-6) Applied to CAT5 cable to test CMR.

Real-time, Deterministic Performance –

Key advances in Ethernet technology for real-time

Ethernet Switches with full-duplex ports don’t have collisions!

Standard Ethernet Frame – IEEE 802.1

SYNCH MAC MAC LENGTH

SYNCH MAC MAC 802.1p 802.1Q LENGTH

IEEE 802.1Q VLAN Tag

Xerox PARC invention back in the 70’s!

Emergence of Industrial Ethernet in the new millennium.

Ethernet chosen as the underlying technology for UCA / IEC 61850

Higher Reliability and Availability

Supervision No Delays or Data Loss Temporary loss, self

For real-time control applications Industrial Ethernet devices and