Sunteți pe pagina 1din 41

IEC 61850 Network Architectures

July, 2010 Maciej Goraj maciejgoraj@ruggedcom.com

Agenda
1. 2. 3. 4. Requirements for substation communications network Types of protocols and traffic patterns in IEC 61850 standard Typical network architectures Problem of Multicast and Physical vs. Logical separation of Process Bus and Station Bus

Copyright RuggedCom Inc

Requirements for Substation Hardened Networking Equipment

Substation Environment
Generation Plant HV/MV Substation Wind Farm

EMI & Environmental Phenomena Typical of Substation Environments Electric and Magnetic Fields Electrostatic Discharge Conducted High Frequency Electrical Transients High Energy Power Surges Ground Potential Rise during ground faults Climactic Variation: Temperature & Humidity Seismic / Vibration Pollution: Dust, Metallic Particles, Corrosive Chemical Particles, Condensation, Solar Radiation, Salt, Bird Guano, etc.
4

Copyright RuggedCom Inc

EMI Phenomenon
Continuous
Phenomena Radiated RFI Induced RFI Power freq. Magnetic Field Slow Voltage Variations Harmonics, Interharmonics Ripple on d.c. power supply Power Frequency Voltage

Transient Phenomena (High Occurrence)


Electrostatic Discharge Voltage Dips Lightning Ground Potential Rise (GPR) HV Switching by Isolators Reactive Load Switching

Transient Phenomena (Low Occurrence)


Power Frequency Variation Power System Faults Short Duration Power Freq. Magnetic Fields

DevicesinsubstationsmustdealwithacombinationofEMI phenomenawhicharebothcontinuousandtransient.
Copyright RuggedCom Inc

Requirements for IEDs According to IEC 61850-3


Must operate properly under the influence of a variety of EMI phenomena commonly found in the substation IEC 61850-3 specifies a variety of type withstands tests designed to simulate EMI phenomena such as:
Inductive load switching Lightening strikes Electrostatic discharges from human contact Radio frequency interference due to personnel using portable radio handsets Ground potential rise resulting from high current fault conditions within the substation

Ethernetswitches,routers,deviceservers,mediaconverters shallmeetEMIrequirementstothesameextentasIEDs
Copyright RuggedCom Inc

Requirements for IEDs According to IEEE 1613


Standard for Environmental and Testing Requirements for Communications Networking Devices in Electric Power Substations It goes one step further by defining Class 2 operation which requires that, during the application of the type tests, the switch must experiment:
No communications errors No communications delays No communication interruptions

RuggedizedEthernetswitchshallbeseenasyetanotherIED
Copyright RuggedCom Inc

Fiber Optics Overview


Future proof Theoretically infinite bandwidth Up to 100 km distance possible Immune to EMI Supported by all current IEDs Lightweight Costs continue to drop Multi-mode for short distances Single-mode for long distances

Copyright RuggedCom Inc

Common Fiber Optic Connectors


ST Stick and Twist and SC Stick and Click historically popular LC becoming prevalent especially for Gigabit because small form factor (SFF) allows greater port density GBIC are pluggable SC transceivers using SC connectors SFP are Small Form Factor Pluggable ST SC

LC

MTRJ

Copyright RuggedCom Inc

Types of Protocols and Traffic Patterns in IEC 61850 Standard

10

Number of Devices in Electrical Substations


Typically 2040IEDs per substation Large substations mayhave 80120IEDs Power Plants,Oil&Gasinstallation 150 500IEDs Large installations with LVIEDs 10001500IEDs Large Wind Farms canhave +200IEDs LargeSolargenerationsitescanhave600 1500IEDs

Copyright RuggedCom Inc

11

IEC 61850 Ed. I Profiles and Protocols Stack

Will be moved to an Annex in Edition II of IEC 61850


Copyright RuggedCom Inc

12

IEC 61850 Ed. II Profiles and Protocols Stack


Sampled Values (Multicast) Generic Object Oriented Substation Event Core ACSI Services

Time Sync

SV
(Type 4)

GOOSE
(Type 1, 1A)

TimeSync (SNTP)
(Type 6)

MMS Protocol Suite


(Type 2, 3, 5)

UDP/IP

TCP/IP T-Profile

SMV

GOOSE HSR (O)

IP (O)

802.1Q

802.1Q

802.1Q (O)

802.1Q (O)

ISO/IEC 8802-3 ISO/IEC 8802-3 Ethertype

Copyright RuggedCom Inc

13

Types of traffic
Client-server MMS services:
Polling Reporting (Unsolicited and/or periodic) Asynchronous and unsolicited Less often synchronous (for heartbeat and for analogue values) Synchronous unsolicited transmission

GOOSE

Sampled Values (Process Bus)

IEC61850networkisacombinationofRawEthernet, MMS/TCP,SNTP,IEEE1588,TFTP,FTP,RSTP,SNMP,and otherEthernetbasedprotocols


Copyright RuggedCom Inc

14

Non-IP traffic in IEC 61850 - GOOSE


Encapsulated directly in Ethernet layer High priority, critical, asynchronous and unsolicited Less often synchronous (for heartbeat and for analogue values) MAC Multicast, uses VLAN for priority and traffic segregation Frame size approx. 92 250 bytes Periodic heartbeat messages of 1-60 seconds interval if no events occur 99% of time just the heartbeat message In case of event an avalanche can occur as many IEDs detect state changes Typically used for fast transmission of digital events Less often for transmission of analogue data, e.g. sent every 250ms

Copyright RuggedCom Inc

15

Retransmission Scheme in GOOSE


GOOSE is connectionless No confirmation from receivers Retransmission to increase the probablity of sucessful reception A burst of 5-6 messages sent in case of event (critical information) Example of implementation:
1st message: on event 2nd message: 4ms after event 3rd message: 16ms after event 4th message: 80ms after event 5th message: 500ms after event

Event occurs, GOOSE with incremented stNum sent immediately Heartbeat GOOSEs

Time

Burst of GOOSEs sent on event occurence


Copyright RuggedCom Inc

16

GOOSE and Network Performance


GOOSE messages shall be priority tagged
Configuration needed in IEDs and in Ethernet switches GOOSE frames with the priority tag in VLAN field configured are placed in the front of the store and forward queue Frames already being sent are not interrupted

Delay of frames introduced by network is almost zero Worst case of total network delay is
100 s at 100MBps links speeds 10 s at 1Gbps

Copyright RuggedCom Inc

17

GOOSE and Network Performance


IEC 61850-5 Type 1A Trip total transfer time defined at 4ms Transfer time = Application to Application and includes:
GOOSE encoding at sender + network delays + GOOSE decoding at receiver

It is difficult to measure as defined in IEC 61850-5 Because the timestamp is added in IED after the internal function execution time (one scan period) Typical measured GOOSE total transfer time including function execution time in IED is in the range of 6-12ms

Copyright RuggedCom Inc

18

Non-IP traffic in IEC 61850 Sampled Values


Encapsulated directly in Ethernet layer High priority, critical, synchronous and unsolicited MAC Multicast, uses VLAN for priority and traffic segregation Currently dedicated wiring (IRIG-B or 1PPS) used for time synch of devices, future will be IEEE 1588 A Merging Unit (sensor) sends 80 or 256 samples/power cycle. At 50Hz it is 4000 and 12800 samples per second respectively. A sample is a set of 8 analog values, 4 voltages + 4 currents @80 samples 4000 packets/sec A single Merging Unit uses approx. 4.4 5.2Mbit/s of bandwidth at 80 Smp The bandwidth used depends of sampling rate and if Data Set is according to IEC61850-9-2LE implementation or other Data Set 1Gbit Ethernet highly recommended for Process Bus in switched Ethernet
Copyright RuggedCom Inc

19

IP based traffic in IEC 61850


Client-Server services
MMS protocol over TCP, port 102 measurements, events, status indications 100-500ms delay accepted Traffic generated by a single IED rarely exceeds 10kbps Reports save bandwidth. Digitals via Buffered, Analogs via Unbuffred.

Time synchronization
SNTP or IEEE 1588 For redundancy mutiple time masters used

File transfer
MMS over TCP, FTP, TFTP, other protocols e.g. Modbus/TCP Typically Oscillography, sequence of events, data logs. Ocassionally configuration, settings, firmware upgrades, etc. File size typically 4 200 kbytes,
20

Copyright RuggedCom Inc

Typical Network Architectures

21

Star Topology
Not protected against single point of failure Simplicity

HMI

Gateway

Protection and Control IEDs


Copyright RuggedCom Inc

22

Redundant Star Topology


Blue LAN A Red LAN B

The entire network is duplicated Configuration and application complexity, cost issues Each device has 2 IP addresses, 2 application instances PRP will be the alternative HMI Gateway

Protection and Control IEDs


Copyright RuggedCom Inc

23

Single Ring Topology


Dashed Redundant Connections Blue Electrical 100Mpbs Red Fiber Optic 100Mbps

IEDs can be dual homed and connected via redundant links Redundancy with RSTP PRP or HSR will be the alternative HMI Gateway

Fiber Optic Ethernet Ring 100/1000 Mbps

Protection and Control IEDs


Copyright RuggedCom Inc

24

Multiple Rings Topology


Limited number of switches in each ring Minimize recovery time Division criteria by voltage levels or by several bays Redundancy with RSTP HMI Gateway

Primary Ring

Secondary Ring 1

Secondary Ring n

Protection and Control IEDs


Copyright RuggedCom Inc

25

Ring of IEDs
Dashed Lines Redundant LAN Connections

IEDs with Embedded Switch functionality Multiple rings may be needed Redundancy with RSTP HSR will be the alternative Gateway

HMI

Fiber Optic Ethernet Ring 100 Mbps

Protection and Control IEDs


Copyright RuggedCom Inc

26

Problem of Multicast and Physical vs. Logical separation of Process Bus and Station Bus

27

Problem of Multicast
Multicast is one-to-many communication scheme Multicast MAC traffic is by default propagated through the whole LAN Consumes link bandwidth and increases latency at switches Introduces significant overhead at receiving IEDs if multicast addresses not allocated properly

Copyright RuggedCom Inc

28

Impact of Multicast
Red MU (Merging Unit) multicasts Sampled Values to small group of IEDs It is dictated by the protection application In a large substation there can be dozens of IEDs sending multicast GOOSE and dozens of Merging Units sending multicast Sampled Values

NTP P IED IED

Primary Ring

Secondary Rings

P C MU

P C

P C MU

P C MU

P C

MU

Copyright RuggedCom Inc

29

Impact of Multicast
All nodes get the traffic red area Repeat for every IED/MU in network Critical messages delayed or maybe dropped Steady state traffic load can exceed 100Mbps for many MUs Excessive MU traffic can cause IEDs and PCs can mis-operate or crash
NTP P IED IED

Primary Ring

Secondary Rings

P C MU

P C

P C MU

P C MU

P C

MU

Multicastmustbefiltered
Copyright RuggedCom Inc

30

Multicast Addresses and Traffic Management


Efficient layer 2 multicast application
Proper allocation of multicast addresses Filtering of multicast traffic

Allocation of multicast addresses


improves processing times at receiving devices by discarding unwanted multicast traffic at hardware level required for multicast filtering

Multicast filtering
saves bandwidth and decreases latency at network switches by limiting the traffic only to restricted areas of the network Multicast filtering solves the primary problem of filtering unwanted GOOSE and SV traffic Use VLAN or MAC address filtering ? Static or dynamic filtering methods ?

Copyright RuggedCom Inc

31

Where we are today ?


In todays substations often no multicast management used at all Lack of knowledge at integrators and utilities Many users just tend to minimize configuration efforts and rely on default settings Until now the dominant method for restricting multicast traffic was the use of VLANs Static configuration: manual process for all IEDs and all network devices

Copyright RuggedCom Inc

32

Example of Misconfiguration
Case Study
50 IEDs in the same network all sending GOOSE No multicast filtering used Wrong! All IEDs send multicast with the same destination multicast MAC address Wrong! In case of event there is an avalanche of GOOSEs in the network and approx 20ms additional processing delay observed at the receiver Improper functioning!

Implementation internals of an IED


Network controller at IEDs has hash table that maps all possible multicast MACs to a small group of addresses Hash table permits discard unwanted multicast MACs at hardware level If all IEDs send with the same multicast destination MAC then at receiving IED these are mapped to the same hash and need to be discarded by software In some IED implementations decoding of GOOSE message takes up to 1.5ms Software decoding of 20 unwanted GOOSE messages can take up to 30ms!

Copyright RuggedCom Inc

33

VLAN (IEEE 802.1Q)


Virtual LAN: an independent Ethernet network that shares cabling infrastructure with other networks Each VLAN has a separate broadcast domain VLANs permit:
Priority tagging Logical separation of the network into various domains
6 bytes 6 bytes 2 bytes Variable

Dest.

Src.

Length / Type

Data

Standard Frame
Variable

6 bytes

6 bytes

2 bytes 2 bytes

2 bytes

Dest.

Src.

TPID

TCI

Length / Type

Data

3 bits

1 bit

12 bits

Priority CFI

VID

Tagged Frame

Copyright RuggedCom Inc

34

Use of VLANs
VLAN is suitable mechanism for isolation of unrelated traffic, eg. surveillance video from SCADA traffic VLANs configuration can be:
Static Dynamic (GVRP)

Today static configuration is a manual process Static configuration can be semi-automatic with future enhanced configuration tools Can use MAC address filtering instead of VLAN VLANs for priority tagging in order to increasing performance

Copyright RuggedCom Inc

35

Traffic Segregation with VLANs

Traffic separated with VLANs:


Substation LAN management SCADA/Engineering Access GOOSE Messages Process Bus (Sampled Values) Synchrophasors Protection A vs. Protection B

Differenttrafficflowsinasubstationnetworkmerit segregatingintoseparateVLANs
Copyright RuggedCom Inc

36

GMRP/MMRP for Dynamic Multicast Filtering


Generic Multicast Registration Protocol Publisher / subscriber model like IGMP Multicast filtered by default must subscribe to get it Adapts dynamically to any network topology and accommodates any application of 9-2 or GOOSE edge only pruning results in no traffic delay after topology change Allows process and station bus to co-exist on same physical network
P

Primary Ring

Secondary Rings

Switches prune the traffic automatically. Either optimally or edge ony

P C MU

P C
SV consumer sends a subscribe message to network periodically

MU
SV producer simply multicasts no change

FirstIEC61850110kVsubstationwithIEEE1588v2and dynamicGMRPmulticastfilteringcommissionedin2010
Copyright RuggedCom Inc

37

Problem of Time Synchronization and Data Sharing


Process Bus requires that Sampled Values coming from different sensors and received by an IED have to be synchronized Synchronization islands are possible, each island spans a protection zone Problem of Line Differential protection with one line end using Process Bus and the other line end using conventional wiring

Copyright RuggedCom Inc

38

Process Bus and Station Bus Separation


Process Bus and Station Bus are logically different Multicast traffic from Merging Units flooding the network A single Merging Unit consumes approx 5Mbps of bandwidth The problem of busbar protection based on Process Bus
In a topology with 60 feeders a process bus based busbar protection would have multicast traffic of > 400Mbps!

Copyright RuggedCom Inc

39

Physically or Logically Separate Networks?


Physically separate LANs are more costly network switches are duplicated Physically separate LANs are perceived as more secure Logically separate LANs are more flexible as Merging Units can be accessed from SCADA (remote maintenance, management, etc.) Logically separate LANs require network engineering or more sophisticated dynamic methods (GMRP, GVRP, etc.) Station Bus could also be connected to Process Bus via router

Copyright RuggedCom Inc

40

Questions?

41