1

The Communication Backbone for

Substation Automation and IEC61850
Ren Midence
RuggedCom Inc.
Vice President, Marketing
Phone: +1 905 266 11 39
Mobile: +1 647 504 58 15
E-mail: ReneMidence@ruggedcom.com
2
Agenda
Background: Substations Past and Present
Ethernet Refresh
Switching Refresh
Sources of Latency in a Switched Ethernet
IEC 61850 Core Components, Architecture &
Communication Model
IEEE 802.1Q VLANs
User-Priority / Classes of Service / IEEE 802.1P
Example
3
Substation Network - Past
4
Substation Network - Present
5
Typical Substation Diagram
Sub
Transmission
Line No. 1
Sub
Transmission
Line No. 2
Power
Transformers
Tie Breaker
Feeder Breakers
IED 4
TX 1
IED 1
LN 1 IED 3
TB 1
IED 2
LN 2
IED 5
TX 1
IED 6
FDR 1
IED 7
FDR 2
IED 8
FDR 3
IED 9
FDR 4
IED 10
FDR 5
IED 11
FDR 6
IED 12
TB 2
IED 14
BUS 2
IED 13
BUS 1
Wiring requirements:
CTs Total 22 x 4c cables
2 x Transmission Lines Distance
Protection
2 x HV Busbar Protection
2 x Bus Tie Busbar Protection
2 x 2 Step Down Transformers
6 x Feeder Protection
6 x LV Busbar Protection
2 x LV Tie Breaker
VTs Total 8 x 2c cables; 4 x 4c Cables
2 Single Phase Transmission Lines
2 x Three Phase HV Bus
2 x Three Phase LV Bus
6 x Single Phase LV Feeders
Circuit Breakers Control Total 14 x
12c cables
2 x Transmission Lines
1 x HV Bus Tie
2 x HV Step Down Transformers
2 x LV Step Down Transformers
6 x LV Feeders
1 x LV Tie Breaker
Power Transformer Control 2 x 12c
Cables
Communication cables between relays
and RTU or PLC Total 14 cables
Total: At least 50 multi-core cables from
the HV equipment to the Control Building
6
Substation Network Future
7
Configuring the LAN for IEC 61850
8
Ethernet Refresh
Ethernet
Initially: shared media (collisions)
MAC (Media Access Control) protocol based-on CSMA/CD
(Carrier Sense Multiple Access with Collision Detection)
Contention-based
Set of devices that contend for access to a segment are
in the same collision domain
Bus network
Hub Network
9
Ethernet Refresh:
MAC -Addressing
Ethernet also defines framing of messages
Multiple Access network: addresses required
Destination / Source
3 ways of delivering messages:
Unicast (one): 00-1B-5B-D1-4A-F5
Multicast (more than 1): 01-80-C2-00-00-00
Broadcast (more than 1All): FF-FF-FF-FF-FF-FF
10
Where Are We?
Communication between IEDs is necessary
Ethernet has many advantages over serial
communications
Shared medium means less predictability: collisions
11
Switches Help
Switches helped by breaking-up collision domains
Node connected directly to a switch, doesnt contend for access
to media
No collisions, (more predictable)
12
Switching Refresher
Switches
Work at L2 (Data-Link Layer)
Store and then forward received frames to the appropriate
interface
Maintain a forwarding database (switching table) containing
MAC Addresses (00-1B-5B-D1-4A-F5)
Port numbers (Port 1)
Additional information (VLAN ID)
Can buffer traffic should the egress port be busy
13
Switching Refresher
Optimize traffic flows by Learning: the location of nodes
dynamically.
Specifically, they learn the Switch Address and associate it with the
ingress port
If the address is unknown it gets flooded out all ports, except the
one it was received on
If the address is known (in switch Data Base), it gets forwarded
to the appropriate port
Also considered filtering, since only the port to which the destination
host is connected will receive this frame
14
Switching Refresher
1 Broadcast domain per switch (~ per VLAN)
Broadcast frames are flooded out all ports (in the same VLAN)
Multicast frames are flooded out all ports (in the same VLAN)
Unicast frames:
flooded (if the destination MAC address is not in the MAC Data
Base)
forwarded to only one port (if the destination MAC address is in
the MAC Data Base)
15
Where Are We?
Communication between IEDs is necessary
Ethernet has many advantages over serial
communications
Shared medium means less predictability: collisions
Switches improve Ethernet performance by eliminating
collisions
16
Switched Ethernet
For some applications, the network must guarantee a
certain maximum latency or the application may not
behave (or fail outright)
Switched Ethernet Networks have many sources of
latency:
Store & Forward, Switch Fabric, Wireline, Queuing
All of these latencies (except for queuing) are
deterministic
The effects of frame queuing can also be calculated providing
one knows the nature of all sources of traffic on the network
17
Who Cares?
The benefits and drawbacks of Ethernet technology are
important
IEC Chose Ethernet as the underlying technology for
61850
18
IEC 61850 Primer
Culmination of:
Comprehensive EPRI-Project UCA 2.0 &
European Experience (IEC 60870-5-101, -103, -104)
Framework for SS communication, high-level description
of SS automation
Standards required:
Interoperability & Integration: data representation, how devices
should look / behave to network applications
Intuitive device/data modeling/naming: Hierarchical and
structured
Fast and convenient communication
Lower cost: install, configure, maintain, & also connect legacy
devices
19
Core Components
Object Model (info available from primary equipment &
SS automation functions)
Inter-IED Message/communication format
Environmental robustness
Configuration language
20
Architecture
Process Bus: 61850-enabled IEDs get digitized power
grid condition data via MUs
Station Bus: 61850-enabled IEDs communicate with e/o
using station busses
21
IEC61850 Substation Architecture
I
E
C
6
1
8
5
0
P
r
o
c
e
s
s

B
u
s
I
E
C
6
1
8
5
0
S
t
a
t
i
o
n

B
u
s
22
IEC61850 Substation Architecture
I
E
C
6
1
8
5
0
S
t
a
t
i
o
n

B
u
s
23
Ethernet Packet
Station Bus
Status, Trips, Close,
GOOSE / GSSE
A, V, Hz, W, Wh,
Var, Varh, SOE
IED Block Diagram
29
IEC61850 Substation Architecture
I
E
C
6
1
8
5
0
P
r
o
c
e
s
s

B
u
s
30
Ethernet Packet
Process Bus
Merging Unit Block Diagram
I/Os
Process Bus data is used by
other IEDs in the network
to perform real time
protection & control mission
critical actions
31
Remarks
Ethernet has been chosen as the common
communications architecture in Substation automation
The main goal is having interoperability between a
variety of substation Intelligent Electronic Devices (IEDs)
IEC 61850 is the International standard has as its
underlying network technology - Ethernet.
Requirements for Ethernet in the substation environment
requires real-time and reliable performance
Considerations when designing substation automation
and networking:
Equipment designed to meet IEC and IEEE standards for
substation hardened equipment
Type of cables to be used
Network topology
Network recovery i.e. rapid spanning tree
Network security
32
Communication Model
GOOSE messages
Used for fast transmission of substation events, such as
commands, alarms, indications
A GOOSE message sent by an IED can be received several
receivers
Examples:
Tripping of switchgear
Providing position status of interlocking
33
Connecting the Two
(Ethernet and 61850)
Details of 61850 irrelevant to switches
Switches are Interested in the Ethernet part only
61850 Architects knew importance of predictable latency
inside the sub
Also knew that Switched Ethernet unpredictable due to
queuing
34
Answer: Use VLANs
Question:
If queuing / buffering is important to operation of
Ethernet > How to minimize any adverse effect?
Answer:
Identify the relative priority of frames
Increase available bandwidth, reduce cost, improve
security
Using standard mechanism: IEEE (802.1P/Q > 802.1D-2004)
35
VLANs - Basic Facts
VLAN: a logical group of ports
on a single device or across multiple switches
Used construct separate virtual networks (independent
domains that share cabling and switch infrastructure)
IEEE 802.1Q defines how to carry multiple VLANs over
the same physical link (between switches)
Broadcasts are contained to a single VLAN
Since devices in the same VLAN are in the same
broadcast domain, they can communicate without a
router (L3 device)
A Layer 3 device is required to communicate between
VLANs
36
Cost Effective Solution
37
Relay
Meter
Red, Green, and
Blue VLANs are
separate networks
that share the
same
infrastructure.
SCADA
Surveillance
Billing
Trunk ports
carry traffic
from all
VLANs on
same fiber
MPEG
IEEE 802.1Q VLANs in the Substations
38
Frame format
Tagged v. Untagged
Standard
Frame
Dest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
Dest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
TPID TCI
Priority CFI VID
2 bytes
3 bits 1 bit 12 bits
2 bytes
Tagged
Frame
39
Edge port vs. Trunk port
R
e
d
B
l
u
e
G
r
e
e
n
G
r
e
e
n
R
e
d
B
l
u
e
G
r
e
e
n
R
e
d
B
l
u
e
Trunk Link
(Carries all 3 VLANs)
Trunk:
A VLAN Trunk carries traffic for multiple VLANs over
the same physical link
A VLAN tag identifies the VLAN membership of the
frames on the trunk
Trunk ports send tagged frames
Edge:
A VLAN Edge port carries traffic for a single VLAN
Edge ports send untagged frames
40
IEEE 802.1P Prioritization
Class of Service (CoS)
High priority traffic gets to
bypass queue
Multiple egress traffic
queues: higher priority
traffic sent first
Reduces jitter and latency
for time-sensitive traffic
(voice / GOOSE)
Shares tag header with
802.1Q VLAN
41
GOOSE, Priority, and VLANs
Different traffic flows in the
61850 substation network
merit separate VLANs:
Substation LAN management
(e.g. switches, routers,
modems, etc.)
SCADA/Engineering Access
GOOSE Messages
Process bus (IEC 61850-9-2
sampled values)
Synchrophasors
Protection A vs. Protection B
Video surveillance and access
control
GOOSE should have the highest Class of Service (CoS) as per IEEE 802.1P
GOOSE is faster than conventional I/O
42
802.1P/Q Summary
VLANs
Logical group of ports (Broadcast domain)
Convey group membership via IEEE 802.1Q Tag
Tag carries VLAN ID and User-priority (they work in tandem)
Trunk, Tagged
Edge, Untagged
Standard
Frame
Dest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
Dest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
TPID TCI
Priority CFI VID
2 bytes
3 bits 1 bit 12 bits
2 bytes
Tagged
Frame
43
Where Are We?
Communication between IEDs is necessary
Ethernet has many advantages over serial
communications
Shared medium means less predictability: collisions
Switches improve Ethernet performance by eliminating
collisions
Queuing latency minimized with Classes of Service
(CoS/802.1P)
CoS (priority) information carried in VLAN tag
44
IEC61850 Standard
Multicast Addressing
The multicast (destination) addresses use following structure:
First 3 Bytes assigned by IEEE: 01-0C-CD.
4
th
Byte shall be: 01 for GOOSE, 02 for GSSE, and 04 for multicast
sampled values.
Last 2 Bytes to be used as individual addresses assigned by the range
defined in Table B.1.
Source: IEC61850 Standard 61850-8-1
First edition 2004-05
Communication networks and systems in substations Part 8-1: Specific Communication Service
Mapping (SCSM) Mappings to MMS (ISO 9506-1 and ISO 9506-2) and to ISO/IEC 8802-3
45
Remember:
61850-enabled IEDs use messages to communicate
GOOSE messages:
Multicast delivery (flooded by switches)
So, all nodes see message (regardless of whether its required)
Want to convey priority to switch
Not necessarily set VID
Priority Frame:
46
GOOSE and Network Performance
Generic Object Oriented Substation Event (GOOSE):
Very powerful IED to IED messaging mechanism
Allows efficient real time exchange of any state or analog
parameter.
IEC 61850-5 Type 1A Trip total transfer time at 3ms
Transfer time = Application to Application and includes
IED times + Network delays.
Worst case of total network delay is
100 s at 100MBps links speeds
10 s at 1Gbps
Small networks or for less demanding applications the
delay introduced by switches is tolerable
47
GOOSE and Network Performance
For larger networks of say twenty switches with 100
Mbps links, a delay of 2 ms could impact the protection
scheme
GOOSE messages are priority tagged
Highest Class of Service (CoS IEEE 802.1P) in the network
GOOSE frames are placed in the front of the store and forward
queue
Frames already being sent are not interrupted
Recommendation:
Consider Gigabit links for networks running highly demanding
IEC 61850 GOOSE and Sampled Values applications
48
VLAN Advanced - Cost Effective Solution
Sub Transmission
Line No. 1
Sub Transmission
Line No. 2
Power
Transformers
Tie Breaker
Feeder Breakers
IED 4
TX 1
IED 1
LN 1 IED 3
TB 1
IED 2
LN 2
IED 5
TX 1
IED 6
FDR 1
IED 7
FDR 2
IED 8
FDR 3
IED 9
FDR 4
IED 10
FDR 5
IED 11
FDR 6
IED 12
TB 2
IED 14
BUS 2
IED 13
BUS 1
VLAN 10 Transmission +
HV Bus Protection IEDs
VLAN 20 Distribution + LV
Bust Protection IEDs
VLAN 30 Bus Bar
Protection IEDs
VLAN 40 Transformer
Protection IEDs
VLAN 50 Video
Surveillance
VLAN 1 Engineering /
Operations / SCADA
49
VLAN Advanced - Cost Effective Solution
Sub Transmission
Line No. 1
Sub Transmission
Line No. 2
Power
Transformers
Tie Breaker
Feeder Breakers
IED 4
TX 1
IED 1
LN 1 IED 3
TB 1
IED 2
LN 2
IED 5
TX 1
IED 6
FDR 1
IED 7
FDR 2
IED 8
FDR 3
IED 9
FDR 4
IED 10
FDR 5
IED 11
FDR 6
IED 12
TB 2
IED 14
BUS 2
IED 13
BUS 1
VLAN 10 Transmission +
HV Bus Protection IEDs
VLAN 20 Distribution + LV
Bust Protection IEDs
VLAN 30 Bus Bar
Protection IEDs
VLAN 40 Transformer
Protection IEDs
VLAN 50 Video
Surveillance
VLAN 1 Engineering /
Operations / SCADA
50
Example
IED1 LN1
IED2 LN2
IED3 TB1
IED4 TX1
IED5 TX2
IED6 FDR1
IED7 FDR2
IED8 FDR3
IED9 FDR4
IED10 FDR5
IED11 FDR6
IED12 TB2
IED13 BUS1
IED14 BUS2
SCADA
Engineering /
Operations
Video
Surveillance
VLAN 10 Transmission +
HV Bus Protection IEDs
VLAN 20 Distribution + LV
Bust Protection IEDs
VLAN 30 Bus Bar
Protection IEDs
VLAN 40 Transformer
Protection IEDs
VLAN 50 Video
Surveillance
VLAN 1 Engineering /
Operations / SCADA
51
IED1 LN1
IED2 LN2
IED3 TB1
IED4 TX1
IED5 TX2
IED6 FDR1
IED7 FDR2
IED8 FDR3
IED9 FDR4
IED10 FDR5
IED11 FDR6
IED12 TB2
IED13 BUS1
IED14 BUS2
SCADA
Engineering /
Operations
Video
Surveillance
Network Behaviour
VLAN 10 Transmission +
HV Bus Protection IEDs
VLAN 20 Distribution + LV
Bust Protection IEDs
VLAN 30 Bus Bar
Protection IEDs
VLAN 40 Transformer
Protection IEDs
VLAN 50 Video
Surveillance
VLAN 1 Engineering /
Operations / SCADA
52
IED1 LN1
IED2 LN2
IED3 TB1
IED4 TX1
IED5 TX2
IED6 FDR1
IED7 FDR2
IED8 FDR3
IED9 FDR4
IED10 FDR5
IED11 FDR6
IED12 TB2
IED13 BUS1
IED14 BUS2
SCADA
Engineering /
Operations
Video
Surveillance
Network Behaviour
VLAN 10 Transmission +
HV Bus Protection IEDs
VLAN 20 Distribution + LV
Bust Protection IEDs
VLAN 30 Bus Bar
Protection IEDs
VLAN 40 Transformer
Protection IEDs
VLAN 50 Video
Surveillance
VLAN 1 Engineering /
Operations / SCADA
53
Where Are We?
Communication between IEDs is necessary
Ethernet has many advantages over serial communications
Shared medium means less predictability: collisions
Switches improve Ethernet performance by eliminating collisions
Queuing latency minimized with Classes of Service (CoS/802.1P)
CoS (priority) information carried in VLAN tag
VLAN configuration on a link must match (IED <> Switch)
Trunk ports send traffic for multiple VLANs, each* identified by a
tag
Edge ports send traffic for a single VLAN, and typically send
untagged frames
54
Thank You!
Questions?
Ren Midence
RuggedCom Inc.
Vice President, Marketing
Phone: +1 905 266 11 39
Mobile: +1 647 504 58 15
E-mail: ReneMidence@ruggedcom.com
55
References
IEC 61850 -Communication Networks and Systems in
Substations:An Overview of Computer Science, Jianqing
Zhangand Carl A. Gunter, Illinois Security Lab
RuggedCom: The Communications Backbone for IEC
61850, IEC 61850 Seminar, derived from UCA Users
Group Meeting CIGRE 2006
DistribuTECH 2009 - Setting your network for IEC61850
Traffic, Rene Midence