Network System Management - Implementation and Applications of The IEC 62351

Network System Management: Implementations and
Applications of the IEC 62351-7 Standard
3002003738
10235028
10235028
Network System Management: Implementations and
Applications of the IEC 62351-7 Standard
3002003738
Technical Update, December 2014
EPRI Project Manager

R. King
ELECTRIC POWER RESEARCH INSTITUTE

3420 Hillview Avenue, Palo Alto, California 94304-1338  PO Box 10412, Palo Alto, California 94303-0813  USA
10235028800.313.3774  650.855.2121  askepri@epri.com  www.epri.com
DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES
THIS DOCUMENT WAS PREPARED BY THE ORGANIZATIONS NAMED BELOW AS AN ACCOUNT OF
WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI).
NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY
PERSON ACTING ON BEHALF OF ANY OF THEM:
(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH
RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM
DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED
RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS
SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR
(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING
ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED
OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS
DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN
THIS DOCUMENT.
REFERENCE HEREIN TO ANY SPECIFIC COMMERCIAL PRODUCT, PROCESS, OR SERVICE BY ITS
TRADE NAME, TRADEMARK, MANUFACTURER, OR OTHERWISE, DOES NOT NECESSARILY
CONSTITUTE OR IMPLY ITS ENDORSEMENT, RECOMMENDATION, OR FAVORING BY EPRI.
THE FOLLOWING ORGANIZATIONS PREPARED THIS REPORT:
Electric Power Research Institute (EPRI)
Systems Integration Specialists Company, Inc.
Doubletree Systems, Inc.
This is an EPRI Technical Update report. A Technical Update report is intended as an informal report of
continuing research, a meeting, or a topical study. It is not a final EPRI technical report.
NOTE
For further information about EPRI, call the EPRI Customer Assistance Center at 800.313.3774 or
e-mail askepri@epri.com.
Electric Power Research Institute, EPRI, and TOGETHER…SHAPING THE FUTURE OF
ELECTRICITY are registered service marks of the Electric Power Research Institute, Inc.
Copyright © 2014 Electric Power Research Institute, Inc. All rights reserved.
10235028
ACKNOWLEDGMENTS
The following organizations prepared this report:
Electric Power Research Institute (EPRI)
3420 Hillview Avenue
Palo Alto, CA 94304-1338
Principal Investigators
G. Chason
R. King
Systems Integration Specialists Company, Inc.
6605 19-1/2 Mile Road
Sterling Heights, MI 48314
Principal Investigator
H. Falk
Doubletree Systems Inc.
4030 Moorpark Ave., Suite 222
San Jose, CA 95117
Principal Investigator
J. Zha
This report describes research sponsored by EPRI.
This publication is a corporate document that should be cited in the literature in the following
manner:
Network System Management: Implementations and Applications of the IEC 62351-7 Standard.
EPRI, Palo Alto, CA: 2014. 3002003738.
10235028 iii
10235028
ABSTRACT
The reliable and efficient delivery of electric power is increasingly dependent on information
technology (IT) and communication infrastructures. This information infrastructure is
increasingly being used to support the control of operational assets and the monitoring of electric
grid and equipment health. Management of this infrastructure requires both resilient connectivity
and effective analytics for the reliable delivery of electric service. For the management of this
infrastructure to be reliable, it must also be secured.
Effective management of security for the broad array of operational technologies (OT) and IT
infrastructures found in modern electric grids can be a complex process. Capabilities for
managing security for the broad range of equipment found in disparate deployments makes
effective administration difficult. To address this difficulty, part 7 of the International
Electrotechnical Commission (IEC) 62351 standards series titled Security through network and
system management was developed.
To assist utilities in using this standard, the Electric Power Research Institute (EPRI) report
Network System Management: End-System-Related International Electromechanical
Commission (IEC) 62351-7 Object Definitions (3002000373), analyzed the potential for
implementing IEC 62351-7 in a standardized and interoperable approach. As part of the research
identified in the report, an initial Simple Network Management Protocol (SNMP) management
information base (MIB) and information models were developed. These were used to validate the
semantics of the standard.
To support the use of the IEC 62351-7 SNMP MIBs, further research has been conducted. This
research provides a review of manufacturer’s implementations and includes a technical guide for
exposing relevant IEC 62351-7 network and system management (NSM) objects via SNMP.
Equipment manufacturers can use this guidance to incorporate the SNMP MIBS into their
products. Utilities using products supporting the IEC 62351-7 MIBs can develop a more
comprehensive view of security and situational awareness for the monitored systems.
Keywords
Cyber security
IEC 62351-7
Network management system
Security object model
Simple Network Management Protocol (SNMP)
10235028 v
10235028
EXECUTIVE SUMMARY
Power system reliability has typically been ensured through the design of a communications
infrastructure, selection of applications and intelligent electronic devices (IEDs), and the
programming and configuration of both the communication and power system resources. The
management and oversight of the communications infrastructure has typically been separate
from that of the power system network. The power system has become more dependent on the
communications infrastructure with the installation of synchrophasors and distributed
automation. Therefore, common management and oversight becomes an issue that should be
addressed.
The power system communications infrastructure has begun to adopt corporate information
technologies such as TCP/IP and Ethernet infrastructures. With this adoption, information
technology (IT) centric network system management (NSM) tools may be used to manage the
intermediate communication systems and common server hardware. However, the technology to
monitor the communication statistics for the IEDs and serial communication (such as RS-232 or
radio) infrastructures does not exist. IEC 62351-7 represents a technical specification that
attempts to address the need for a single set of NSM definitions for the power system
infrastructure to address telecommunication management as well as cyber security monitoring
and management. The previous Electric Power Research Institute (EPRI) reports Network
Security Management for Transmission Systems (1024421) and Network System Management:
End-System-Related International Electromechanical Commission (IEC) 62351-7 Object
Definitions (3002000373) provided a transformation of IEC 62351-7 so that it can be
implemented.
As part of the current EPRI project, the usability of IEC 62351-7 was demonstrated through the
development of use cases, staging a laboratory in which implementations of the use cases could
be demonstrated, performing demonstrations to utilities, and receiving feedback.
10235028 vii
10235028
CONTENTS
1 BACKGROUND AND INTRODUCTION ...............................................................................1-1
1.1 Purpose and Scope ................................................................................................1-1
2 DEFINITIONS AND ACRONYMS .........................................................................................2-1
2.1 Definitions ...............................................................................................................2-1
2.2 Abbreviations and Acronyms ..................................................................................2-1
2.3 Notations ................................................................................................................2-3
3 USE OF NSM BY ELECTRIC SECTOR ...............................................................................3-1
3.1 NSM Comparison ...................................................................................................3-1
3.2 Legacy Power Device Monitoring and Management Strategies ..............................3-4
3.2.1 IT and OT interaction ........................................................................................3-4
3.2.2 Maintenance .....................................................................................................3-7
3.2.3 Security.............................................................................................................3-8
3.3 Benefits of NSM....................................................................................................3-12
3.3.1 Benefits...........................................................................................................3-12
3.3.2 Transition and Migration Toward IEC 62351-7 ................................................3-14
4 APPROACH .........................................................................................................................4-1
4.1 Use Case Development and Documentation ..........................................................4-1
4.2 Implementation Architectures .................................................................................4-1
4.2.1 Target Device Implementation ..........................................................................4-1
5 USE CASE DEFINITIONS ....................................................................................................5-1
5.1 Visualization ...........................................................................................................5-3
5.2 Traffic Analysis .......................................................................................................5-3
5.2.1 Substation Network Storm Detection and Prevention ........................................5-3
5.2.2 Protocol Monitoring and Denial of Service Detection and Prevention ................5-5
5.2.3 Traffic Pattern Analysis .....................................................................................5-6
5.3 Resource Exhaustion..............................................................................................5-7
5.4 Component Failure and Degradation Alarms ..........................................................5-8
6 DEMONSTRATION IMPLEMENTATION STRATEGY .........................................................6-1
6.1 EPRI Smart Substation ...........................................................................................6-2
6.2 SNE Software Architecture .....................................................................................6-5
6.3 SNE Deployment Requirements .............................................................................6-8
6.4 Network Topology Visualization ..............................................................................6-9
6.5 Resource Monitoring Use Case ............................................................................6-10
6.6 Power Supply Failure Use Case ...........................................................................6-15
6.7 Other Visualizations ..............................................................................................6-15
6.7.1 Lost Packet Detection .....................................................................................6-15
6.7.2 IED Clock Error Detection ...............................................................................6-16
6.7.3 Intrusion Detection ..........................................................................................6-17
10235028 ix
6.7.4 Traffic Statistics ..............................................................................................6-17
6.7.5 Summary Display ............................................................................................6-18
7 CURRENT STATE OF IEC 62351-7 .....................................................................................7-1
8 SUMMARY ...........................................................................................................................8-1
9 REFERENCES .....................................................................................................................9-1
A MIB UTILIZATION............................................................................................................... A-1
A.1 CISCO ................................................................................................................... A-1
A.1.1 CISCO Proprietary MIBs used directly ............................................................. A-1
A.1.2 Standardized MIBs .......................................................................................... A-2
A.1.3 IEC-62351 MIBs .............................................................................................. A-3
10235028 x
LIST OF FIGURES
Figure 1-1 EPRI’s integrated grid .............................................................................................1-1
Figure 1-2 NSM integration envisioned by IEC 62351-7 ...........................................................1-2
Figure 3-1 High-level ISO ITIL analysis process.......................................................................3-2
Figure 3-2 Typical IT/OT interaction .........................................................................................3-5
Figure 3-3 Interdependence of IT of network status as part of problem resolution....................3-6
Figure 3-4 Maintenance and work management ......................................................................3-7
Figure 3-5 Maintenance using NSM information.......................................................................3-8
Figure 3-6 Electronic security perimeter (ESP) enforcing SCADA access control ....................3-9
Figure 3-7 Maintenance bypasses RTU for direct access ......................................................3-10
Figure 3-8 Maintenance with direct access to devices............................................................3-10
Figure 3-9 Example of an X.509 digital certificate ..................................................................3-11
Figure 3-10 Identified IEC 62351-7 monitoring agents ...........................................................3-12
Figure 3-11 MIB hierarchy ......................................................................................................3-13
Figure 3-12 Possible placements of IEC 62351-7 proxy function ...........................................3-15
Figure 5-1 General information flow and actors ........................................................................5-2
Figure 5-2 General information flow and actors for non-SNMP IEDs ........................................5-2
Figure 5-3 General sequence for visualization .........................................................................5-3
Figure 5-4 Sequence for network storm detection and prevention ............................................5-4
Figure 5-5 Sequence for GOOSE denial of service detection and prevention ..........................5-5
Figure 5-6 General sequence for traffic pattern analysis ..........................................................5-7
Figure 5-7 General sequence for resource exhaustion .............................................................5-8
Figure 5-8 General sequence for component failure and degradation ......................................5-8
Figure 6-1 EPRI Smart Substation topology .............................................................................6-2
Figure 6-2 Substation Network Explorer architecture ...............................................................6-6
Figure 6-3 IEC 62351-7 gateway/proxy ....................................................................................6-8
Figure 6-4 Network topology view ............................................................................................6-9
Figure 6-5 Topology XML description .....................................................................................6-10
Figure 6-6 SNMP manager configuration ...............................................................................6-14
Figure 6-7 Summary view using SNMP ..................................................................................6-14
Figure 6-8 Power supply failure display using SNMP .............................................................6-15
Figure 6-9 Example of GOOSE protocol packet .....................................................................6-16
Figure 6-10 Example of detecting GOOSE packet loss ..........................................................6-16
Figure 6-11 Time accuracy detection .....................................................................................6-16
Figure 6-12 GOOSE intrusion detection proof of concept.......................................................6-17
Figure 6-13 GOOSE traffic statistics (per IED) .......................................................................6-18
Figure 6-14 GOOSE network traffic aggregate view ...............................................................6-18
Figure 6-15 Protocol summary for an IED ..............................................................................6-19
Figure 7-1 Overview of structure of IEC 62351-7 edition 2 .......................................................7-1
Figure 7-2 IEC 62351-7 edition 2 MIB structure for the device agent .......................................7-2
10235028 xi
10235028
LIST OF TABLES
Table 2-1 MIB notations ...........................................................................................................2-3
Table 3-1 ITU FCAPs categories and IEC 62351-7 ..................................................................3-3
Table 3-2 IEC 62351-7 applicability for FCAPS security functions............................................3-3
Table 6-1 SNE server requirements .........................................................................................6-8
Table 6-2 SNE client requirements...........................................................................................6-9
Table 6-3 MIBs utilized for SNE .............................................................................................6-11
10235028 xiii
10235028
1
BACKGROUND AND INTRODUCTION
1.1 Purpose and Scope
The use of Network System Monitoring (NSM), or Management, has long been a staple of IT in
regards to communication network infrastructure and major application platforms (e.g., servers).
The application of NSM within a utility has typically followed the normal IT practices and
protocol usage to achieve NSM. However, there is a major portion of the utility operational
infrastructure that has not been able to be properly integrated into an NSM infrastructure: the
Intelligent Electronic Devices (IEDs) that provide the ability of an operational utility to perform
its primary function of managing the electrical distribution or transmission network.
Figure 1-1
EPRI’s integrated grid
In order to achieve the integrated/smart grid, there is an expansion of communication capability

and IED deployment required to achieve the end objective. As communication requirements
expand, the need for network monitoring and management and cyber security become important.
Additionally, the separation of the information of IT, operational technology (OT), and
maintenance creates barriers to efficiencies that could be achieved. As an example,
communication network maintenance needs to be coordinated with OT so that communication
outages do not occur while the grid is under duress.
Although IT NSM has visibility to most intermediate systems (e.g. routers and switches), there is
typically no visibility into communication issues of the end devices. OT monitors information
regarding operations, but typically IEDs do not expose detailed information regarding
10235028 1-1
communications, protocol usages, or cyber security aberrances via Supervisor Control and Data
Acquisition (SCADA) protocols. Thus, providing a mechanism to expose end systems (e.g.,
IED) communication related information becomes important.
Some IED vendors have begun the exposure of NSM related information via SNMP which
allows integration into the NSM environment. However, the information exposed have
proprietary definitions and identifications. The proprietary nature of this information makes
integration into an enterprise wide NSM infrastructure difficult and costly.
The International Electrotechnical Commission (IEC) began to address the requirements and
standardization of the information through IEC 62351-7. IEC 62351-7 is intended to allow IED
communication related information/statistics to be integrated into an NSM environment.
Figure 1-2
NSM integration envisioned by IEC 62351-7
IEC 62351-7 provides the object definitions and mappings of these definitions to standardized
Management Information Base (MIB) object identifiers allowing different vendors to expose
information in a standardized manner.
To date, IEC 62351-7 has been a standards development with little or no implementation to
provide feedback to the standardization process. The project discussed in this document, EPRI
3002003738, develops use cases that are used to evaluate the applicability of IEC 62351-7 and
provide feedback in order to improve the standard.
Additionally, the project developed a proof-of-concept demonstration to further prove the
viability of the concepts in IEC 62351-7.
10235028 1-2
2
DEFINITIONS AND ACRONYMS
This section provides definitions and acronyms for key terms as they are used in this report.
When the definition is referenced from another document, the source is noted in brackets.
2.1 Definitions
Agent This term is used to refer to an SNMP Agent that allows the NSM to obtain
SNMP information from an IED or intermediate system.
Subagent This term is used to refer to the internal monitors within an IED or
intermediate system that provide the information to the Agent.
Intermediate Intermediate systems are communication components that allow
System communication between end systems. Intermediate systems typically are used
to refer to communication switches, routers, multiplexors, etc.
End System This term typically refers to the hardware platforms that actually communicate
with each other in order to accomplish a particular function. Within the scope
of this document, end systems should be thought of as IEDs and utility
applications that communicate in order to protect and maintain stability of the
electrical network. This term excludes applications and devices that are used to
maintain the communication infrastructure.
2.2 Abbreviations and Acronyms

DNP Distributed Network Protocol
EMS Energy Management System
EPRI Electric Power Research Institute
FTP File Transfer Protocol
HTTP Hypertext Transfer (or Transport) Protocol
IEC International Electrotechnical Commission
IED Intelligent Electronic Devices
IDS Intrusion Detection Systems
IEC International Electrotechnical Commission
IP Internetwork Protocol
ISO International Organization for Standards
IT Information Technology
10235028 2-1
ITU International Telecommunication Union
LAN Local Area Network
MIB Management Information Base
NMS Network Management System
NSM Network and Systems Management
OID Object Identifier
OT Operational Technology
SCADA Supervisory Control and Data Acquisition
SNMP Simple Network Management Protocol
TC Technical Committee
TCP Transmission Control Protocol
UDP User Datagram Protocol
WAN Wide Area Network
CRL Certificate Revocation List
RTU Remote Terminal Unit
PSP Physical Security Perimeter
ESP Electronic Security Perimeter
WMS Work Management System
RMON Remote Monitoring
FCAPS Fault, Configuration, Accounting, Performance, and Security
ITIL Information Technology Infrastructure Library
IDS Intrusion Detection System
IPS Intrusion Prevention Systems
10235028 2-2
2.3 Notations
Within this document, the notation < > will be used to indicate the MIB root that is being utilized
by the MIB definition.
Table 2-1
MIB notations
Notation MIB Root OID

<CISCO> 1.3.6.1.4.1.9
<Private> 1.3.6.1.4
<MIB-2 Interfaces> 1.3.6.1.2.1.2

< MIB-2 System> 1.3.6.1.2.1.1
<Private Enterprise> 1.3.6.1.4.1.35406

<Radiflow> 1.3.6.1.4.1.15004
10235028 2-3
10235028
3
USE OF NSM BY ELECTRIC SECTOR
The penetration of NSM within the electric sector has increased as the electric sector
communication infrastructure becomes more IT centric. However, the adoption of NSM has not
been as widely embraced within the electric sector environment, due to organizational silos and
the incompatibility between network management protocols and SCADA protocols. As the
ability to monitor the resources of the electric power control systems becomes increasingly
important, so does NSM.
However, NSM upgrades have not always been compatible with all components in existing
systems. This potential incompatibility may require asset owners and operators to field additional
systems to support the incorporation of comprehensive NSM into their operations while
maintaining legacy systems.
3.1 NSM Comparison

A direct comparison of typical legacy operations and the deployment of a 62351-7 capable NSM
provides a view of the potential gains a utility operator could realize. This comparison also
provides a framework that the operator could use in performing an assessment of the prospective
benefits for their respective systems.
The remainder of this section provides a mapping between the 62351-7 capable NSM and a
broad cross range of representative operations, policies, and procedures.
In order to do additional comparisons to those found in Section 3.2.1, the NSM functions need to
be included as part of a larger management architecture. The ITU defined an architecture known
as FCAPS (fault, configuration, accounting, performance, and security) to provide a functional
description of overall telecommunication network management tasks. ISO standardized a slightly
different specification known as ITIL (ISO 20000: Information Technology Infrastructure
Library). The ITIL specification builds on the capabilities of NSM to define best practices for
business processes regarding IT operations.
A wide assortment of ITIL white papers are available at http://www.itil-
officialsite.com/aboutitil/whatisitil.aspx. Within the scope of this document, ITIL provides high
level business processes as illustrated in Figure 3-1 below.
10235028 3-1
Figure 3-1
High-level ISO ITIL analysis process
In regards to IEC 62351-7, the high level ITIL process can be applied:
• Understand issues: There is limited visibility in the IEDs regarding NSM functionality.
When there is visibility, it is done in a proprietary mechanism/definition that makes it costly
to integrate the information. IEC 62351-7 provides a solution to these issues.
• Scope and capabilities of assets: As IEDs are used more for control and automation
purposes, more compute resources are being applied and local-area network (LAN)/wide-
area network (WAN) communication infrastructures are becoming more prevalent. This
allows for the potential of IT-oriented NSM and processes to be applied to improve overall
reliability. IEC 62351-7 provides an aspect of this potential improvement.
• Assess opportunity for improvement: IEC 62351-7 is intended to provide the capability to
monitor and diagnose issues in IEDs through the use of a standardized protocol and
definitions.
• Technological change: As cyber security has become more of a focus, IED vendors have
begun providing security related functions. Some have provided Syslog and SNMP
interfaces, but predominantly in a proprietary manner. In many cases, protocol specific (e.g.,
DNP, IEC 60870-5, and IEC 61850) related monitoring is not provided. With the advent of
IEC 62351-7, many of the SNMP proprietary definitions could be converted to standardized
definitions. Additionally, IEC 62351-7 provides protocol specific monitoring capability.
Thus IEC 62351-7 has the potential to lower integration and testing time as well as providing
visibility to a critical endpoint.
The high level process can be further refined when applied to digital informational assets that
are used within the electric utility domain. FCAPS assists with the classifications shown in
Table 3-1.
10235028 3-2
Table 3-1
ITU FCAPs categories and IEC 62351-7
FCAPS
Description IEC 62351-7 Applicability
Category
Provides monitoring of power supplies and
Provide the ability to monitor, diagnose,
(F)ault internal resources that can be used to
and respond to faults.
predict or notify specific faults.
IEC 62351-7 does not explicitly address
Provide a mechanism to determine and configuration. Some network configuration
(C)onfiguration
manage changes to digital assets. information (e.g. maximum allowed
connections) is indirectly exposed.
IEC 62351-7 provides information
Provide a mechanism for tracking
(A)ccounting regarding the peer of the remote protocol
network utilization.
exchange.
Provide a mechanism to measure the IEC 62351-7 provides additional
(P)erformance service level provided by the network and performance monitoring regarding specific
allowing planning for future network use. protocols.
Provides the ability to ensure that a
IEC 62351-7 provides standardization and
network environment is secure, and the
(S)ecurity monitoring to allow the detection, and
monitoring of security-related information
correlation, of security related events.
so that it can be analyzed regularly.
FCAPS further defines the security functions. FCAPS divides the security functions into security
services and security event detection and reporting. IEC 62351-7 provides security event
detection and reporting on several FCAPS specified security services. Table 3-2 below shows the
correlation of the event detection capability of IEC 62351-7 against the FCAPS security
functions.
Table 3-2
IEC 62351-7 applicability for FCAPS security functions
FCAPS FCAPS IEC 62351-7 Event Detection and

Security Function Sub-function Reporting Applicability (yes/no)
Authentication Yes.
Access Control Yes
Data Confidentiality Yes, through tamper detection
Data Integrity Yes, through tamper detection.
Security Services
Nonrepudiation No
Event Detection Yes
Audit Trail Management No
Security Recovery No
Additional Security Unauthorized User Detection Yes
event detection and
reporting Physical Tampering Yes, detects physical access.
10235028 3-3
Several existing ITIL tools could make use of the capabilities of IEC 62351-7 and integration
with these systems should be planned. These tools/systems are:
• Network Management Systems (e.g. HP OpenView, Industrial Defender, CISCO ROSA, and
others): These products should be evaluated in terms of their users to determine if integration
with IEC 62351-7 sources would be of benefit to the enterprise. There may be situations
where one NSM should be integrated with IEC 62351-7 and others should not.
• Intrusion Detection/Prevention Systems: These systems are typically deployed within an ESP
within the operational control center. However, these systems may be deployed within the
ESP of a substation. In these deployment scenarios, systems that implement IEC 62351-7
may have more accurate information upon which to base reporting or actions. In particular,
an IPS needs accurate information prior to taking preventive action that could disrupt
communications.
3.2 Legacy Power Device Monitoring and Management Strategies

For asset owners and operators to gauge the value of adding NSM to their respective
deployments, it is necessary to review current operations. This review can then be used to assess
potential gains and pitfalls associated with integrating NSM into deployments.
The remainder of this section includes a review of operations, processes, and procedures that
may potentially be impacted by the deployment of a 62351-7 capable NSM. The review includes
the identification of operations and system segments that could be impacted by the deployment
of a 62351-7 capable NSM.
3.2.1 IT and OT interaction
The typical utility has an OT organization responsible for Operations and an IT/Telecom
organization that is responsible for the communications network. Operations may be dependent
upon the communication infrastructure to communicate with the electrical network. Each group
has its own operators that oversee and react to information provided by the supervisory
monitoring systems.
The OT organization’s operators typically interact with information supplied by a SCADA/EMS
system. The SCADA/EMS system acquires information from IEDs in the field over the
IT/Telecom infrastructure. The IEDs, including RTUs, are attached to sensors/actuators
associated with electrical network equipment. The protocols used for information acquisition
include, but are not limited to, Modbus, DNP, IEC 60870-5, ICCP, and IEC 61850. The
operator’s responsibility is the operation/stability of the electrical network.
10235028 3-4
Operations Problem IT/Telecom
SCADA/
EMS
NSM
IEDs
Primary and Intermediate

Secondary Systems
Equipment
Figure 3-2
Typical IT/OT interaction
The IT/Telecom domain’s equivalent to the SCADA/EMS is typically a set of applications that
facilitate the NSM function. The IT operators interact with the information provided by the NSM
function(s). These functions acquire information through interacting with intermediate systems
such as: Ethernet Switches; Routers, microwave transmitters; SONET controllers; and TDM
multiplexors. The protocols used for information acquisition include, but are not limited to,
SNMP, Syslog, and RMON. The operator’s responsibility is to maintain the communication
integrity of the IT/Telecom infrastructure.
When the IT/Telecom infrastructure fails, the OT organization would typically see a loss of
information or quality changes. However, the silos of OT/IT prevent the OT operators from
having a view of the state of the IT network. Therefore, when the “loss of acquisition” event
occurs, the OT operator typically needs to communicate with the IT operator to determine if the
IT/Telecom network is the source of the problem or if there could be an IED failure.
The IT operator typically does not have a view of the state of the electrical network and may not
understand the impact of the failures that are being observed on the OT side of the organization.
It is clear, that only the IT operator would know if the IT network is degrading. This degradation
could impact OT’s ability to respond especially if the electrical network is under stress. When
the IT organization has knowledge of the communication network versus electrical network and
degradation or failures occur, the operator communicates with the OT operator. There is a blind
spot in the IT information, and that is the communication resources within the IEDs.
10235028 3-5
Operations Problem IT/Telecom
SCADA/
NSM
EMS
IEC 62351-7
IEDs
Primary and Intermediate

Secondary Systems
Equipment
Figure 3-3
Interdependence of IT of network status as part of problem resolution
There has been an ongoing evolution that allows the OT operators to have greater visibility of the
impact of IT communication network connectivity. In some instances, the SCADA/EMS vendors
have delivered interfaces to specific NSMs or sub-functions. Other IT information sharing can be
provided through an IT screen (typically a subset of the NSM screens). Either mechanism allows
the OT operator to gauge the health of the IT network. At this time, the OT operator must still
contact the IT operator in order for communication network issue mitigation.
IEC 62351-7 allows the NSM to receive network and security related information from the IED
itself. This new information source typically allows the IT organization to monitor end-to-end
(i.e., End systems and intermediate systems) and to have a more complete view of the
communication network activity.
Currently, IEC 62351-7 allows the NSM to monitor IED information regarding:
• Physical Access: Cabinet and IED access information is exposed.
• Communication Security: Authentication and signature problems as well as traffic issues are
exposed.
• SCADA protocol information: Traffic patterns and invalid communication information is
exposed.
• Clock information: Tampering and accuracy is exposed.
• Environment information: Temperature and power supply information is exposed.
IEC 62351-7 defers to other MIB standards in regards to Ethernet, IP, and TCP information. This
makes it consistent with MIB information typically exposed by the intermediate systems, thus
making IEDs easier to integrate into the NSM environment.
10235028 3-6
3.2.2 Maintenance
IT has been responsible for monitoring and problem mitigation regarding the communication
network, while OT has been able, in the past to request maintenance for failed devices. Both
organizations typically use a Work Management System (WMS) to initiate the maintenance
process. However, to date, the Maintenance organization has had no visibility of Operations or
IT, but Maintenance becomes responsible for mitigation of the problem in the instances of
equipment failure or degradation.
Work Management
System
SCADA/ NSM Maintenance

EMS
Problem Mitigation
IEDs Intermediate Involvement
Systems
Primary and
IT/Telecom
Secondary
Equipment
Operations
Figure 3-4
Maintenance and work management
As previously mentioned, the visibility of degradations in the IEDs has typically not been visible.
The advent of IEC 62351-7 allows the NSM to potentially provide Maintenance the additional
information that could lower the time of the maintenance activity.
10235028 3-7
Work Management
System
SCADA/ NSM Maintenance

EMS
Problem Mitigation
IEDs Intermediate Involvement
Systems
Primary and
IT/Telecom
Secondary
Equipment
Operations
Figure 3-5
Maintenance using NSM information
There are several IED related subsystems that are exposed via IEC 62351-7:
• IED Internal Temperature
• IED CPU and Memory Utilization
• IED Power cycle counts
• Power Supply failure
The additional information could be conveyed through the WMS, or directly to Maintenance.
In some situations, the Maintenance restoration period is dependent upon IED and
communication network problem mitigation. Thus restoration estimates of customer service
could become more accurate with the additional information provided.
3.2.3 Security
IEC 62351-7 specifically focused on providing more standardized security related information
than was previously available. Information provided includes, but is not limited to:
• Physical Security Perimeter (PSP) intrusion.
• Electronic Security Perimeter related information.
• Digital credential management.
The ability to monitor/notify on this additional information could potentially lead to quicker
security reaction times and audit/repudiation capability.
10235028 3-8
3.2.3.1 Physical Security Perimeter
As physical access to substations and other utility assets has become more of a focus, utilities
have installed a set of technologies. These technologies include video surveillance and
monitoring of key physical access points to the access. Monitoring of the physical access points
is currently done through limit switches (e.g. open/closed) through the SCADA system. It is
worthwhile to note that the use of SCADA, for this purpose, does not allow for information
standardization or information sharing to the group responsible for security.
IEC 62351-7 does not address video surveillance, but does standardized information regarding
physical:
• Perimeter Access
• Panel Access
• IED access
Through the use of the NSM infrastructure, this information can now be shared with the
appropriate security organization.
3.2.3.2 Electronic Security Perimeter
Before the advent of IEC 62351-7, the observation points for cyber intrusion were related to the
information provided by firewalls, intrusion detection systems (IDSs), and other intermediate
systems. Additionally, it is typical that access control is enforced at the edge of an electronic
security perimeter.
Electronic
Security
Perimeter
Substation LAN
or Communication
Infrastructure
Access Control
Substation Gateway
or RTU
Figure 3-6
Electronic security perimeter (ESP) enforcing SCADA access control
In the past, access control was enforced by the substation router, firewall, or substation gateway.
This “edge” enforcement was due, in large part, to the SCADA protocols not providing a
sufficient level of authentication and confidentiality. The “edge” access control does not aid in
cyber security regarding entities within the substation, or access for entities that bypass the
normal access control infrastructure. In many situations, Maintenance and substation engineering
directly access the IEDs bypassing some of the access control mechanisms.
10235028 3-9
Electronic
Security
Perimeter
Substation LAN
or Communication
Infrastructure
Access Control
Bypass
Maintenance or
Substation Engineering
Figure 3-7
Maintenance bypasses RTU for direct access
However, DNP, IEC 60870-5, and IEC 61850 now have authentication and confidentiality
capability. Thus, there is the ability to augment access control in the IEDs themselves.
Electronic
Security
Perimeter
Substation LAN Access Control
or Communication
Infrastructure
Bypass
Maintenance or
Substation Engineering
Figure 3-8
Maintenance with direct access to devices
IEC 62351-7 provides the capability to provide information regarding, but not limited to:
• Authentication failures for the secure SCADA protocols.
• Attack counts and attack vector source information
• Traffic analysis
Thus providing more detailed information and enforcement capability within the utility
infrastructure.
3.2.3.3 Digital Credential Management
The secure protocols rely on digital credentials. Most are based upon Digital Certificate
information that provides the capability of authentication and distribution of encryption/signing
keys.
10235028 3-10
Figure 3-9
Example of an X.509 digital certificate
When certificates are created, they contain an expiration date. Beyond that date the certificate is
considered invalid. Additionally, if a certificate is compromised (e.g. the private keys are
exposed), the certificate authority(s) are informed and issue a Certificate Revocation List (CRL).
In 2010, EPRI performed a survey of 36 members to determine if the members had processes in
place to manage the expiration of certificates and application of CRLs. Although the response to
the survey was minimal, several key observations were made: None of the responding members
had a process to manage expiring certificates.
In most cases, the utility did not track which certificates were installed on what equipment, nor
the criticality of the certificate form OT. None of the responding members had a formal process
to determine where/when to apply the CRLs.
The end result of the lack of processes is that communication could be disrupted due to expired
certificates or revoked certificates. In order to address this situation, many utility policies
continue to use expired and revoked certificates as their responsibility is to maintain the
operation of the electrical network even if there is decreased security.
Although IEC 62351-7 cannot solve the lack of process issue, it can provide information that
could be used to trigger digital credential management processes. The standard allows exposure
of generic information regarding:
• Warning of certificates about to expire.
• Warning of expired certificates being used.
• Warning of revoked certificates being used.
The use of this information and the creation of the appropriate digital credential management
processes could lead to utilities being able to minimize the degradation of security in order to
maintain the operational integrity of the electrical network.
10235028 3-11
3.3 Benefits of NSM
Having the descriptions for legacy operations and subsequent comparison to NSM, it is possible
to develop a view of potential benefits afforded by an NSM.
The remainder of this section will provide details and specific benefits, relative to legacy
operations, that could be realized with an enhanced NSM deployment.
An overview of considerations that should be included in planning the transition to and inclusion
of a 62351-7 capable NSM will be provided. This will assist asset owners and operators in
assessing the impact of the enhanced NSM to their respective operations.
3.3.1 Benefits
IEC 62351-7 provides a set of standardized definitions and SNMP MIB object identifiers that
should allow easier integration of information into a NSM/FCAPS environment. The
standardized definitions allow utilities to plan transition/migration strategies (see page 3-1).
The definitions extend the ability to react to issues that were probably un-monitored before, or
monitored in a proprietary manner. The categorization of the information follows the structure of
the IEC 62351-7 MIB/sub-agent architecture.
Monitoring Agent
Application Entity Device/IED

(IED Application) Environmental
DNP, IEC 60870-5,

Application IEC 61850
Presentation
Agent/Server
Session
TCP
Transport UDP
Network IP
Multiples of:
Data Link Keyboard, USB,
RS-232, RS-485,
Physical Ethernet
Monitoring Agent
Figure 3-10
Identified IEC 62351-7 monitoring agents
The MIB hierarchy is shown in Figure 3-11.
10235028 3-12
Figure 3-11
MIB hierarchy
Figure 3-11 shows NSM objects that can be supplied by the sub-agents. The information
described is limited to information that was either unavailable in the past or was only available
via proprietary definition/means.
• Environmental and Device sub-agents: This MIB branch standardizes objects that allow
detection of physical access, resource utilization (e.g. CPU, memory, storage, etc.), and
power supply failure. Specifically, physical access detection, in the past, has required utility
specific solutions.
• Protocol sub-agents: This MIB branch standardizes objects that allow monitoring that is
specific to particular protocols. The protocols are IEC 61850, DNP (IEEE 1815), and IEC
870-5. The intended use of the information is to determine communication resource
utilization, enhance security monitoring, and enhance intrusion detection.
• Interface sub-agents:
- The interfaces that need to be monitored are keyboards, USB, Clock, and Serial.
- These MIB branches allow detection if the physical devices (e.g. keyboards and USB) are
attached or in use. This is important information that can be used to augment the physical
access information and to further clarify the type of access being done.
Most IEDs have keyboard front panels that can allow settings/programs to be changed,
thus monitoring of its use could be important. Today’s IEDs are beginning to migrate to
allowing USB storage devices to be used for firmware upgrades or programming
changes. Thus monitoring of the use/attachment of such a device to an IED may be
important. Additionally, USB storage devices offer a new threat vector of information
disclosure since information could be directly obtained from the IED. However, this
threat vector is determined to be a minor issue since more dire threats could be
perpetrated due to having direct physical access.
10235028 3-13
The clock information (not shown in the diagram) was added due to the need discovered
as part of this project. Although there may be standardized MIBs evolving around IEEE
1588, there are no such MIBs available for IRIG, GPS, and NTP. The ability to detect
tampering of clocks is becoming a more prevalent concern in the industry and IEC
62351-7 provides a generic and well defined mechanism for initial detection and
diagnosis.
Protocols such as DNP and IEC 60870-5 can be operated over serial links and derivatives
(e.g. RS-422 or radio). When such protocols are in use, the monitoring of the serial links
is as important as monitoring Ethernet, IP, and Transport MIBs for TCP/IP based
protocols.
The MIB branches for Transport (e.g. UDP and TCP), Network (e.g. IP), and Ethernet are
placeholders should future information, not covered in the other referenced MIB standards, be
required. Thus, IEC 62351-7 current defers to other well established IT MIB definitions in these
areas.
3.3.2 Transition and Migration Toward IEC 62351-7
The need for a transition/migration strategy towards IEC 62351-7 is due to the industry adoption
of the standard. The level of adoption is in large part, not currently sufficient to expect IEDs to
implement IEC 62351-7. The lack of adoption is in large part, due to the history of IEC 62351-7
itself:
• IEC 62351-7 was published in July of 2010. This edition of the standard defined high level
objectives for NSM information, but did not provide concrete information or MIB definitions
(e.g. OIDs) that would have allowed interoperable implementations.
• EPRI’s project in 2012-2013 was targeted to address the lack of definitions and to provide
input to the IEC as a foundation for the Edition 2 of IEC 62351-7. The project resulted in a
report and a set of concrete definitions that were forwarded to IEC.
• Development of Edition 2 of IEC 62351-7. Based upon the EPRI input, the work on
defining Edition 2 of IEC 62351-7 has started and is anticipated to be standardized mid-2015.
The initial publication of the committee draft (CD) is expected prior to the end of 2014. It is
expected that the technical maturity of the definitions will occur 1-2 months after the
comment period of the CD. It is when this technical maturity is achieved that IED vendors
can consider actual adoption.
As the demonstration aspect of this project showed, migration needs to be planned for through
the use of mapping since the benefits of IEC 62351-7 Edition 2 will hopefully lead IED vendors
to adoption.
10235028 3-14
There are three transition/migration strategies currently identified that are possible for utilities:
1. Wait for IEC 62351-7 implementation to arrive within the utility enterprise and integrate
its definitions separately from the NSM definitions in use.
2. Wait for IEC 62351-7 implementation to arrive within the utility enterprise and integrate
the existing information into the IEC 62351-7 definitions through the use of mappings.
3. Begin mapping existing NSM information into IEC 62351-7 definitions and wait for IEC
62351-7 implementation to arrive in the enterprise.
Option 3 allows for a utility to gain experience with IEC 62351-7 in advance of actual
implementations of IEC 62351-7. This advanced experience would allow utilities to develop a
set of applications and processes that it may not have currently. Thus decreasing the time to
adoption and potentially increasing the ROI on adopting the standard.
Options 2 is a wait and see option. There is little doubt that standardized definitions will benefit
NSMs of the future. However, by waiting, the same processes and application development,
expedited by Option 3, would be delayed in this option.
Option 1 is the do nothing option. By treating IEC 62351-7 as a proprietary set of information,
enterprise level application/process development will probably not be achieved by a utility. This
option relegates IEC 62351-7 to an informational source and not the standardized information
framework that it is intended to be.
Option 2 and Option 3 requires equivalent proprietary MIBs to be mapped into the IEC 62351-7
definitions. The mappings may also require transformation/scaling to be provided (e.g. unit
conversion may be required). There are three architectural places where such a
mapping/transformation (a.k.a. a 62351-7 proxy) function can be placed. These options are
shown in Figure 3-12.
IED NSM
IEC 6235-7 IEC 6235-7
IEC 6235-7
Proxy
Proxy
Proxy
Proprietary
External
Operations IT/Telecom
Figure 3-12
Possible placements of IEC 62351-7 proxy function
10235028 3-15
The figure shows a proxy can:
• Be placed in an IED that is already exposing information in a proprietary mechanism. This
allows the IED to provide its proprietary information and that information mapped into the
IEC 62351-7 simultaneously thereby providing backward compatibility. This type of proxy
placement is important to be addressed within a utility’s migration strategy as if proprietary
information is already being used by the NSM, a utility needs to evaluate if it will continue to
use the proprietary information in parallel to the adoption of IEC 62351-7.
• Be placed within the NSM itself. Many nsms allow for a strategy to develop interfaces that
map proprietary mibs into more common/standardized mibs. This strategy is a strategy that
can probably be performed by IT/Telecom without requiring involvement by the Operations
department.
• Be placed in an external appliance. This option is similar to placing the proxy in the NSM but
allows the proxy to be placed anywhere within the enterprise. However, the placement of the
proxy may require involvement of the operations department if the proxy is deployed within
the operational ESP.
The demonstration, associated with this report, utilizes the external proxy concept.
10235028 3-16
4
APPROACH
This section describes the general process used to develop the demonstration and evaluation of
the capabilities of IEC 62351-7.
4.1 Use Case Development and Documentation

There were primary use cases evaluated for the demonstration and use of IEC 62351-7. The
specific use cases are document in the Use Case Definitions (see page 5-1) section. The
identified use cases involve:
• Visualization (see page 5-3)
• Traffic analysis (see page 5-3)
• Resource exhaustion (see page 5-7)
• Component failure and degradation (see page 5-8)
4.2 Implementation Architectures

Although all IEC 62351-7 deployment architectures could have been part of the demonstration,
due to the current adoption penetration of IEC 62351-7, only the gateway/proxy deployment
strategy was part of the demonstration. The actual mappings from the demonstration equipment
are provided in individual annexes.
4.2.1 Target Device Implementation
The following information describes the devices that were integrated as part of the
demonstration.
4.2.1.1 CISCO CGR 2000/CGS 2500 Implementation
The CISCO Connected Grid Router (CGR) 2000 and Connected Grid Switch (CGS) 2500 are
devices that meet the harsh environmental requirements of transmission and distribution
substation environments.
For further information see http://www.cisco.com/c/en/us/products/collateral/routers/
2000-series-connected-grid-routers/data_sheet_c78_593509.html and http://www.cisco.com/
c/en/us/products/collateral/switches/2500-series-connected-grid-switches/
data_sheet_c78_593672.html.
4.2.1.2 Radiflow 3180 Implementation
The RADiFlow 3180 is a ruggedized, compact, secure, protocol-aware switch/router designed
for the industrial control (e.g., substation) environment.
For further information see http://www.radiflow.com/products/3180-compact-switches.
10235028 4-1
4.2.1.3 RuggedCom RSG2288 and RSG 2100
The RUGGEDCOM RSG2288 and RSG2100 are utility grade, fully managed, modular Gigabit
Ethernet switches, designed to operate in electrically harsh and climatically demanding utility
substation and industrial environments.
For further information see http://w3.siemens.com/mcms/industrial-communication/en/
rugged-communication/products/switches-routers-layer-2/Pages/ethernet-layer-2-switches.aspx.
4.2.1.4 Schweitzer
Schweitzer 3620, 2622, and 2730M devices were used as part of the demonstration.
The Schweitzer 3620 and 3622 are devices that provide routing, VPN, and secure IED proxy
capabilities that has been substation hardened. For further information see
https://www.selinc.com/SEL-3620/ and https://www.selinc.com/SEL-3622/.
The Schweitzer 2730M is a substation hardened switch. For further information see
https://www.selinc.com/SEL-2730M/.
4.2.1.5 General Electric (GE) D60
The GE D60 is a distance line protection device. For further information see
http://www.gedigitalenergy.com/multilin/catalog/d60.htm.
10235028 4-2
5
USE CASE DEFINITIONS
This section will contain a description of the use cases that drive the demonstrations and
evaluations of the manufacturers’ 62351-7 MIB implementations. There are three areas of
concentration:
• Traffic Analysis: This category encompasses a wide range of different use cases. At its core
is the ability to use 62351-7 MIBs to determine if there is one or more different types of
abnormal traffic occurring. Once the monitoring and diagnostics occur, action can be taken to
mitigate the situation in many cases.
• Resource Exhaustion: This category encompasses evaluation of server resources and the
detection of when an abnormal amount of resources are being utilized.
The primary actors for all of these scenarios are:
• SNMP Agents: There could be multiple SNMP agent instances involved within any
particular use case depending upon the amount of coordination/distribution required to fulfill
a particular use case. SNMP Agents can provide:
- Monitoring: Exposes information utilized as part of the use case.
- Action: Provides the capability of taking a commanded action (e.g., disconnect a port).
Both types of agents need to provide the appropriate subagents needed to fulfil the use case.
• SNMP subagents, as defined in IEC 62351-7 or other standards, which provide the required
monitoring or action capability.
• Network System Monitoring Station: This actor provides the “centralized” capability to
monitor SNMP Agent/subagent information and to issue commands.
• Analysis tool: This actor represents the functionality of analyzing the monitored information
and creating actionable metrics. In some cases, this actor could be a program, in others it
could be a person.
• Operator: This actor is typically a human that uses the metric to take action.
Figure 5-1 depicts the general flow for all of the following use cases. There are SNMP Agents
that reside in the physical IEDs or Intermediate Systems (e.g. Routers or Switches). There are
subAgents in the physical device that provide information to the SNMP Agent. The SNMP agent
allows a Network System Manger (NSM) tool to read the information or receive notifications
(typically called Traps) that convey information that is being monitored by the subAgents. The
NSM then exposes this information to one or more analysis tools or calculation engines (shown
as the Analysis Actor). This analysis function is responsible for receiving the monitored
information and turning that information into a metric. In some cases, this metric is used to view
an aggregate of information. In other use cases, the metric is used by an operator to trigger an
action to mitigate a condition in the field. The action is dispatched through the operator
interacting with the NSM. The NSM then issues the appropriate command (SNMP Write) to the
appropriate SNMP Agent. The SNMP Agent then distributes the command to the appropriate
10235028 5-1
subAgent based upon the MIB Object Identifier (OID). In many use cases, the commanded
action is not dispatched to the same SNMP Agent that provided the monitoring information that
triggered an action to be taken.
IED or
One or more Intermediate
points of Systems
observation SNMP
SNMP SNMP
or action NSM Analysis Operator
subAgent(s)
subAgent
Agent(s)
Monitored
Information
Monitored
Informaiton
Metric(s)
Action
Command
Commanded
Action
Figure 5-1
General information flow and actors
IED or Intermediate Systems or

Proxy
IED with no SNMP SNMP

SNMP Proxy SNMP
NSM Analysis Operator
SNMP subAgent(s)
subAgent Agent(s)
Non-SNMP
Protocol
Translation to
SNMP MIBs Monitored
Information
Monitored
Informaiton
Metric(s)
Action
Command
Commanded
Action
Figure 5-2
General information flow and actors for non-SNMP IEDs
10235028 5-2
Today’s IEDs do not typically support SNMP. Therefore, in order to provide NSM monitoring
capability for such IEDs, a SNMP Proxy needs to be utilized. This proxy communicates with the
IED using a non-SNMP protocol (e.g. Modbus, DNP-3, IEC 60870-5, or IEC 61850). The proxy
is responsible for mapping the non-SNMP information into the appropriate SNMP MIBs and
providing the MIB values to the appropriate logical subAgent for exposure to the local SNMP
agent. The sequence of the information flow is the same as that shown in Figure 5-1 except for
the additional proxy function.
5.1 Visualization
Visualization of information is critical to the more advanced use cases. At its core, the use case
allows information to be monitored and provided to different actors. Figure 5-3 depicts
information being exposed to the operator. In other use cases, information can be exposed to
other entities (e.g. IT or Maintenance).
IED or
Intermediate
Systems
SNMP
SNMP SNMP
NSM Operator
subAgent(s)
subAgent
Agent(s)
Monitored
Information
Monitored
Informaiton
Figure 5-3
General sequence for visualization
5.2 Traffic Analysis

The three use cases being evaluated are:
• Substation Network Storm Detection and Prevention
• Protocol Monitoring and Denial of Service Detection
• Traffic Pattern Analysis for intrusion detection
5.2.1 Substation Network Storm Detection and Prevention
This use case utilizes Storm Threshold Configuration and Traffic Analysis use cases. The use
case details the requirements for the configuration of data storm thresholds. The configurations
described are used to detect and mitigate conditions associated with data storms. Mitigations are
achieved by supporting a user’s ability to turn off the port(s) that need to be managed in order to
prevent/correct the data storm.
10235028 5-3
The network storm detection and prevention use case adheres to the general use case sequences.
However, there are additional use case objects that have been added into the sequence. These
are:
• Knowledge of the Communication Network Topology: This knowledge is typically conveyed
through network diagrams so that source addresses can be associated with a given network
connection.
• Electrical System State Knowledge: This knowledge is a real-time perspective as to the
stress/operational status of the electrical network.
Figure 5-4 depicts the general flow for all of the network storm detection and prevention use
cases. The SNMP Agents and SubAgents generate monitored information regarding
Packets/Second and bandwidth utilization. Through the NSM, the Analysis determines if there is
a substation communication storm. In order to determine the source port of the storm, the
knowledge of the communication network topology is required. The Analysis would typically
produce Substation aggregate information so that the operator could determine how severe the
network storm is. Additionally, the Analysis would typically provide enough information to
determine which of the source ports were causing the storm (e.g. being recommended to be
disabled). In many situations, the operator may need to determine the port information based
upon knowledge of the communication network. However, for the purposes of this use case, that
analysis is deemed as part of the Analysis, although it is partially performed by the operator.
IED or
Intermediate
Systems
Intermediate Comunication Electrical
subAgent
SNMP SNMP System SNMP Network System State
NSM Analysis Operator
Ethernet
subAgent Agent Agent Topology Knowledge
From Packets/sec
multiple Bandwidth Utilization
ports ports(packets/sec,
From multiple bandwidth)
SNMP Agents ports(packets/sec,
bandwidth)
ports(packets/sec,
bandwidth)
Substation Aggregate
Source port(s) of storm
Close port(s)
Disable port(s)
Disable port(s)
Figure 5-4
Sequence for network storm detection and prevention
Unlike typical IT infrastructures, the operator would not necessarily take action solely based on
communication network information provided by the NSM. Additional information regarding the
Electrical System State is required in order for the operator to determine if the operational
integrity of the electrical network would be at risk if the ports were disabled. If the integrity of
the electrical network is jeopardized by disabling a port, the operator may not mitigate the
network storm so that control/stability of the electrical network is not impacted.
10235028 5-4
If the operator decides to mitigate the network storm, the operator interacts with the NSM to
close the appropriate ports.
5.2.2 Protocol Monitoring and Denial of Service Detection and Prevention
This use case provides details for monitoring and traffic analysis of per port protocols. The
monitoring and analysis are used as initial steps in the identification of intrusions or DOS
attacks.
The particular use case for the demo is concerned with GOOSE monitoring.
subAgent Electrical
SNMP
IECsubAgent
62351-7 SNMP Network System State
GOOSE Agent NSM Analysis Topology Operator
Knowledge
From RxPDU, TxPDU

multiple PDUCnt
ports Invalid PDU Count Information
From multiple
SNMP Agents Information
Abnormal
Traffic Detected
Determine Source of DOS
Find source
Read Detail MIBs
Details
Close port(s)
Disable port(s)
Disable port(s)
Figure 5-5
Sequence for GOOSE denial of service detection and prevention
The detection, and prevention, of a GOOSE Denial of Service (DOS) attack is similar to the
network storm detection and prevention use case. However, instead of generalized network
packet information being used (e.g. from Ethernet MIBs), the IEC 62351-7 GOOSE subAgent
provides counts and rates for the total number of GOOSE packets sent and received as well as
individual transmit , receive counts, and number of invalid Protocol Data Units (PDUs). The
NSM provides this information for Analysis.
The Analysis determines if there is abnormal traffic detected and conveys this information to the
operator. The Analysis may be configured with defined limits or some other type of adaptive
algorithm (e.g. percentage based upon total traffic).
Based upon the information that abnormal traffic has been detected by a specific set of SNMP
Agent(s), the operator needs to determine the source of the abnormal traffic. The use case shows
the use of IEC 62351-7 detailed information to provide the required information. This detailed
10235028 5-5
information would not typically be constantly monitored by the NSM and therefore the operator
would need to request the read/display of that information. The read of the detailed information
provided by IEC 62351-7 is one mechanism to determine the source of the DOS, however other
mechanism could also be utilized (e.g. RMON or network traffic sniffing). These alternatives are
not shown in the use case.
Once the source of the DOS is determined, the impact of mitigating the DOS needs to be
analyzed in regards to the mitigation impacting the Electrical System. If the operator decides to
mitigate the DOS, then the appropriate commands are issued through the NSM.
5.2.3 Traffic Pattern Analysis
This use case details the requirements for network traffic analysis as required for detecting
anomalies or intrusions. Typically, intrusion detection needs to detect specific abnormal patterns
of particular protocols. Many intermediate systems can provide information related to Ethernet,
TCP/IP, or UDP/IP. However, they do not typically provide information related to the traffic
patterns of DNP, IEC 60870-5, IEC 61850, or other telecontrol/SCADA like protocols. In order
to accomplish this, the end devices, and not the intermediate systems, need to provide the
monitored information to allow abnormal traffic patterns to be detected.
In this use case, there is a differentiation between the operator and IT as actors. Whereas an
operator may take quick action, more involved IT policies may be involved. Therefore:
• IT: Represents a set of resources and policies that need to be performed (e.g. report cyber
intrusion, analyze source routes as to where the attack is originating). Thus, IT represents a
scope and view that is larger than the operator and may cause interactions with entities that
are outside the operator’s utility.
Traffic pattern analysis is similar to the GOOSE Denial of Service (Figure 5-5) sequence.
However there are differences in the sequence resulting in IT following some predefined policy
to report/mitigate the traffic issue.
Figure 5-6 begins with the IEC 62351-7 subAgents, for the various protocols of interest,
providing counts for received and transmitted Protocol Data Units (e.g. DNP, Modbus, IEC
61850 application messages). The NSM receives this information from the SNMP Agent.
However, these agents would typically be in the IED itself since intermediate systems typically
don’t have application protocol packet inspection and interpretation capability. However, it is
feasible that an Intrusion Detection System (IDS) may have the required capability.
10235028 5-6
subAgent SNMP
SNMP
IECsubAgent
62351-7 Expected
IEDs NSM Analysis traffic pattern Operator IT
DNP, 870-5, 61850
From RxPDU, TxPDU

multiple Source Address
ports RxPDU, PDUCnt
From multiple
intermediate
systems RxPDU, TxPDU
RxPDU, TxPDU
Notification of invalid
traffic pattern
Problem
disclosure
IT Policy Defined Sequence
Figure 5-6
General sequence for traffic pattern analysis
The Analysis is provided expected limits for given traffic, and generates a notification of invalid
traffic patterns to the operator. The operator notifies IT and then IT performs the further
investigation, reporting, and mitigation based upon its policies.
5.3 Resource Exhaustion

This use case details the requirements for monitoring resource exhaustion conditions.
Identification of exhaustion conditions that is necessary to maintain communication
infrastructure and electrical grid resource availability.
Figure 5-7 shows the general sequence for the Resource Exhaustion use case. In many situations,
MIBs/subAgents other than IEC 62351-7 can be monitored by the NSM to provide the
information regarding specific resources. Typical resources, that may be monitored, are: memory
usage; CPU utilization; number of active TCP connections; number of IEC 61850 associations,
and others. The monitored resource information is provided to the NSM by the SNMP Agents.
The Analysis is configured with expected resource patterns/limits and generates a notification of
the resource issue and an identification of the system having problems to the operator. If the
operator is capable of mitigating the condition, the mitigation solution is evaluated against the
stability of the Electrical Network. In many instances, the mitigation would need to be performed
by IT or Maintenance. The mitigations all always performed with a knowledge of the impact to
the Electrical System.
10235028 5-7
IED or
Intermediate
Systems
Expected Electrical
subAgent
SNMP SNMP System State IT/Maintenance
NSM Analysis resource pattern Operator
Ethernet
subAgent Agent Knowledge
From Resource Information

multiple Resource Information
ports Resource Information
From multiple
SNMP Agents Resource
Information,
Source
Notification of Resource problem,
System
Operator Action
Substation Maintenance/IT
Figure 5-7
General sequence for resource exhaustion
5.4 Component Failure and Degradation Alarms

This use case provides details on requirements for monitoring a component failure or impending
failure.
IED or
Intermediate
Systems
Expected
subAgent
SNMP SNMP pattern IT/Maintenance Work
NSM Operator Analysis
Ethernet
subAgent Agent Order
From Component Infromation

multiple Component
Infromation
ports Component
Component Degedation
From multiple Information
SNMP Agents Resource Information
Source
Notification of Degradation or Limit
Issue
Component Failure
Component Information (Failure)

Issue
Figure 5-8
General sequence for component failure and degradation
10235028 5-8
Unlike the previous use cases, the component failure use case may not involve the NSM
operator. Instead the information provided by the NSM would be provided to IT/Maintenance
Analysis in order to detect degradation so that notifications can be generated so that the
appropriate Maintenance Work Order/action is taken. As an example, if the internal temperature
of an IED or cabinet exceeds the configured limits/expected pattern, the Analysis would generate
a notification of a temperature issue for a specific asset.
In some instances, there is no advanced warning/degradation detection possible. A power supply
failure is such an example. In this case, the NSM provides the component failure information to
IT/Maintenance and an appropriate Work Order/action is taken.
As with any NSM monitored information, the operator can view the information and could be
involved in derivative use cases.
10235028 5-9
10235028
6
DEMONSTRATION IMPLEMENTATION STRATEGY
There were four different implementation strategies for the demonstration of the use cases:
• Standardized MIBs directly sourced from IEDs or intermediate systems.
• Use a SNMP Proxy, shown in Figure 15, to convert information from systems that do not
support standardized MIBs into the standard MIB definition.
• A mixed implementation where standardized and SNMP Proxy strategies are utilized.
The mixed implementation strategy was chosen for the demonstration. In many instances, the
intermediate systems supported standardized MIBs (e.g. Ethernet) that could be utilized directly.
However, when it came to IEDs, the IEDs in the demonstration did not support the required
standardized MIBs. No implementations supported IEC 62351-7 MIBs. Therefore, the SNMP
Proxy was used to expose/proxy such MIBs.
The following parts of this chapter detail the actual implementation used for the demonstration.
The information provided includes which standardized MIBs were utilized, the mappings used to
translate non-standard information into standardized information. This information is provided
for each participant in the demonstration of a particular use case.
The implementations used for the use cases were:
• Substation Network Explorer (SNE): The SNE is an NSM tool and human-machine interface
developed in the EPRI Cyber Security Research Lab by DoubleTree, and used for the use
cases.
• CISCO Connected Grid Router (CGR) 2000
• CISCO Connected Grid Switch (CGS) 2500
• RuggedCom 2100 and 2288
• Radiflow 3180
• GE D60
• SEL 3620
• SEL 3622
• SEL 2730M:
Results from testing of manufacturer implementations will be detailed in the following sections.
As part of the project, the new Substation Network Explorer (SNE) is used to demonstrate the
specific use case. SNE product from Doubletree System Inc. has basic network monitoring,
network topology visualization and network traffic analysis capability of IEC 61850 substation.
An IEC 62351-7 Substation SNMP Gateway is implemented as part of SNE. The SNE is
customized to specific Use Case to support the demo.
10235028 6-1
6.1 EPRI Smart Substation
EPRI Smart Grid Substation Lab has a simulated Substation network to facilitate the
IEC 62351-7 use case demo. The topology is show below:
Figure 6-1
EPRI Smart Substation topology
The following IEDs are connected to test environment:

IED1:
Name KNXD60HF
Source MAC 00:a0:f4:00:3a:ee
Destination MAC 00:a0:f4:00:3a:ee
ICD/CID Name ged60hf121113.ICD
Switch Port KNX-SW1, Port 9
IED2:
Name SEL_487E_ACFG
Source MAC 00:30:a7:04:c4:54
Destination MAC 01:0c:cd:01:00:10
ICD/CID Name KNX 487E 121113.CID
Switch Port KNX-SW1, Port 17
10235028 6-2
IED3:
Name SEL_421_1CFG
Source MAC 00:30:a7:01:66:5d
Destination MAC 01:a0:f4:01:4c:30
ICD/CID Name KNX 421 121113.CID
Switch Port RuggedCom , Port 4
IED4:
Name LNX7SJ64CTRL
Source MAC 00:09:80:ff:14:41
Destination MAC 00:09:80:ff:14:41
ICD/CID Name 7SJ641 V4.7.icd
IED5:
Name AREVA P145

Source MAC 00:02:84:91:82:94
IP Address 10.1.0.68
ICD/CID Name AREVA_P14535D_EPRI.ICD
IED6:
Name GE D60 1 CTRL

Source MAC 00:a0:f4:01:c3:2c
ICD/CID Name ged60 61850 demo 21113.ICD
Switch Port RadiFlow, Port 1
10235028 6-3
The following network devices are used to form the test network:
Switch 1:
Name KNX-SW1
Model CGS 2520 24TP
IP Address
IOS Version IOS ipservicesk9-15.0(2SE5)
SNMP Enabled
SNMP Community public
IOS WSMA Configured
Switch 2:
Name KNX-SW2
Model CGS 2520 24TP
IOS Version IOS ipservicesk9-15.0(2SE5)
SNMP Enabled
IOS WSMA Configured
Switch 3:
Name RuggedCom
Model RSG2288
IOS Version v3.10.0
SNMP Enabled
Switch 4:
Name Radiflow
Model 1031
IOS Version Switch Software Version 3.5.03.19
SNMP Enabled
10235028 6-4
Router 5:
Name KNX-CGR
Model CGR2010
IOS Version IOS UNIVERSALK9_NPE-M-15.3
SNMP Enabled
IOS WSMA Configured
Switch 6:
Name EPRISEL-3620
Model 3620
IOS Version R200
SNMP Enabled
Switch 7:
Name EPRISEL-3622
Model 3622
IOS Version R200
SNMP Enabled
Switch 8:
Name EPRISEL-2730M
Model 2730M
IOS Version SEL-2730M-R102-V0-Z002001-D20131204
SNMP Enabled
6.2 SNE Software Architecture

The Substation Network Explorer (SNE) is implemented to support the demonstration of the
Network Monitoring, Traffic Analysis and overall Visualization of the substation network, as
well as report that information through IEC 62351-7 SNMP MIBS. The block diagram of all the
components of SNE is shown below:
10235028 6-5
Figure 6-2
Substation Network Explorer architecture
In order to make SNE very easy to use and manage, the SNE is developed as Web Application,
means that there is no need to install any client side software. User can use any web browser to
access it.
The SNE components include:
Web Server:
It can be any application server, like Tomcat or WebLogic. The user needs install one of those
application servers in the window environment.
Substation Model Module:
SNE will extract IEC 61850 SCD Model for IED Network Topology requirement analysis and
GOOSE traffic value analysis. This module will import SCD file and XML based network
topology file to support above mentioned function. It will update the network topology change
and alert the user with Alarm about Condition Based monitoring function.
Traffic Analysis Module:
This module will perform Protocol related calculation and analysis; the functions include:
• Network Latency Analysis
• Packets Loss Analysis
• IED Time Error Analysis
• Packets Throughput Analysis
• GOOSE Value Analysis
10235028 6-6
SNMP Manager:
This module will collect network topology data as well as device operation condition data. It is
capable of collecting any available SNMP MIBs data from device. Currently, it is collecting the
following SNMP MIBs information from network device
SNMP Get:
• Device Up Time (Since last reboot)
• Power Supply Status (Include redundant Power Supply)
• Device CPU Utilization
• Device MEM Utilization
• Device DISK Space Utilization
• Device Chassis Temperature
SNMP Trap:
• Power Supply Failure
• Storm Detection
• Login Failure
GOOSE Capturer:
The Capturer provides Wireshark like network packet receiver and filter.
Visualization GUI:
The web application which provides content rich visualization about substation network
topology, network traffic analysis and condition based device monitoring capability.
IEC 62351 SNMP Gateway:
The SNMP Gateway will collect Use Case related MIBs information using vendor specific MIBs
and convert it into IEC 62351-7 MIBs. The Gateway serves as an SNMP Agent to other NMS
system, the gateway’s logical diagram is shown below:
10235028 6-7
Figure 6-3
IEC 62351-7 gateway/proxy
6.3 SNE Deployment Requirements

Table 6-1
SNE server requirements
Attribute Requirement
MEMORY At least 4G
CPU Intel Core i5 and Up (2 core minimum)
Disk 10 G Free Space (50G Recommended)
Window 7(32/64 bit), Window 8 (32/64 bit)
Operation System Window Server 2008 SP2
Any Linux(64 Bits)
Virtual Machine SNE can be installed in VM running above OS
One 100M/1Gigabit Ethernet Port is required. Two Ethernet
Network Port Ports are recommended(One port for traffic monitoring and
one for web server)
Application Server Apache Tomcat 7.0.xx or Weblogic
Application SNE v1.0
10235028 6-8
Table 6-2
SNE client requirements
Attribute Requirement
MEMORY At least 4G
CPU Intel Core i5 and Up (2 core minimum)
Disk 10M Free Space
Operation System Window 7(32/64 bit), Window 8 (32/64 bit)
Network Port One 100M Ethernet Port is required.
Web Browser Chrome(Recommended), IE and Firefox(Supported)
6.4 Network Topology Visualization

The network topology view is automatically generated from the topology.xml file. This file is the
combination of SCD IED network information and network topology information from Switches.
Figure 6-4
Network topology view
The following is part of topology XML description; the entail file is in the appendix.
10235028 6-9
Figure 6-5
Topology XML description
6.5 Resource Monitoring Use Case

SNE uses the System Summary View to show all Network and IED device operational status.
That view is using the SNMP manager to get the necessary information about network device’s
condition. The following MIBS are used to generate the visualization:
10235028 6-10
Table 6-3
MIBs utilized for SNE
CISCO MIBs Utilized

The table of power supply status
ciscoEnvMonSupplyStatusTable <CISCO >.13.1.5 maintained by the environmental
monitor card.
The table of ambient temperature
ciscoEnvMonTemperatureStatusTable <CISCO>.13.1.3 status maintained by the
environmental monitor.
cpmCPUTotalTable <CISCO >.109.1.1.1 A table of overall CPU statistics
A table of memory pool monitoring
ciscoMemoryPoolTable <CISCO >.48.1.1
entries.
A table of memory pool utilization
entries. Each of the objects provides
a general idea of how much of the
ciscoMemoryPoolUtilizationTable <CISCO >.48.1.2
memory pool has been used over a
given period of time. It is determined
as a weighted decaying average.
RuggedCom MIBs Utilized
The percentage in tenths of percent of
available CPU cycles used for device
rcDeviceStsCpuUsage <RuggedCom >4.2.2.1
operation as measured over the last
second when object was retrieved.
The total number of bytes of RAM still
rcDeviceStsAvailableRam <RuggedCom >4.2.2.2
available in the system control CPU.
The temperature measured in the
rcDeviceStsTemperature <RuggedCom >4.2.2.3
device.
Indicates the status of Power Supply
Module 1. Whenever the value of this
object changes from functional(2) to
rcDeviceStsPowerSupply1 <RuggedCom >4.2.2.4 notFunctional(3), or from
notFunctionl(3) to functional(2), the
device will generate powerSupplyTrap
notification.
Indicates the status of Power Supply
Module 2. Whenever the value of this
object changes from functional(2) to
rcDeviceStsPowerSupply2 <RuggedCom >4.2.2.5 notFunctional(3), or from
notFunctionl(3) to functional(2), the
device will generate powerSupplyTrap
notification.
The percentage of available CPU
cycles used for device operation as
rcDeviceStsCpuUsagePercent <RuggedCom >4.2.2.6
measured over the last second when
object was retreived.
10235028 6-11
Table 6-3 (continued)
RuggedCom MIBs Utilized (continued)

Indicates status of fail-safe relay in the
device. Fail safe relay is
rcDeviceStsFailSafeRelay <RuggedCom >4.2.2.7
deEnergized(2) if there is at least one
active alarm recorded in the device.
Indicates that at least one alarm of
rcDeviceStsErrorAlarm <RuggedCom >4.2.2.8 level ERROR, ALERT or CRITICAL is
active in the device.
Number of active alarms currently
rcDeviceStsNoOfActiveAlarms <RuggedCom >4.2.2.9
recorded in device.
<RuggedCom > Indicates the status of Fan Bank
rcDeviceStsFanBank1
4.2.2.10 Module 1.
<RuggedCom > Indicates the status of Fan Bank
rcDeviceStsFanBank2
4.2.2.11 Module 2.
Indicates if any of passwords is
<RuggedCom > configured as ‘weak.’ Setting bit in the
rcDeviceStsPwdsWeak
4.2.2.12 value of this object will generate
weakPasswordTrap.
The total number of bytes of RAM in
rcDeviceInfoTotalRam <RuggedCom>.4.2.3.5
the system control CPU.
An entry containing additional
ifXEntry < ifMib >1.1.1 management information applicable to
a particular interface.
Radiflow MIBs Utilized
<Private Enterprise > System name used for identification
issSwitchName
100.1.81.1.1 of the device.
Indicates the minimum threshold
temperature of the switch in Celsius.
<Private Enterprise > When the current temperature drops
issSwitchMinThresholdTemperature
100.1.81.1.64 below the threshold, an SNMP trap
with maximum severity will be sent to
the manager.
Indicates the maximum threshold
temperature of the switch in Celsius.
<Private Enterprise > When the current temperature rises
issSwitchMaxThresholdTemperature
100.1.81.1.65 above the threshold, an SNMP trap
the manager.
.<Private Enterprise > Indicates the current temperature of
issSwitchCurrentTemperature
100.1.81.1.66 the switch in Celsius.
10235028 6-12
Table 6-3 (continued)
Radiflow MIBs Utilized (continued)

Indicates the maximum CPU usage of
the switch in percentage. When CPU
.<Private Enterprise >
issSwitchMaxCPUThreshold load exceeds the threshold value, an
100.1.81.1.67
SNMP trap with maximum severity
will be sent to the manager.
.<Private Enterprise > Indicates the current CPU threshold
issSwitchCurrentCPUThreshold
100.1.81.1.68 of the switch in percentage
Indicates the maximum power supply
of the switch in volts. When the
.<Private Enterprise >
issSwitchPowerSurge current voltage exceeds the threshold
100.1.81.1.69
value, an SNMP trap with maximum
severity will be sent to the manager.
Indicates the minimum power supply
of the switch in volts. When the
.<Private Enterprise > current voltage drops below the
issSwitchPowerFailure
100.1.81.1.70 threshold value, an SNMP trap with
maximum severity will be sent to the
manager.
.<Private Enterprise > Indicates the current power supply in
issSwitchCurrentPowerSupply
100.1.81.1.71 volts.
Indicates the maximum RAM usage
of the switch in percentage.
.<Private Enterprise > When the RAM usage crosses the
issSwitchMaxRAMUsage
100.1.81.1.72 threshold percentage, an SNMP trap
the manager.
.<Private Enterprise > Indicates the current RAM usage of
issSwitchCurrentRAMUsage
100.1.81.1.73 the switch in percentage
Indicates the maximum flash usage of
the switch in percentage. When the
.<Private Enterprise > flash usage crosses the threshold
issSwitchMaxFlashUsage
100.1.81.1.74 percentage, an SNMP trap with
maximum severity will be sent to the
manager.
.<Private Enterprise > Indicates the current flash usage of
issSwitchCurrentFlashUsage
100.1.81.1.75 the switch in percentage
An entry containing additional
ifXEntry .<Std ifMib >1.1.1 management information applicable
to a particular interface.
10235028 6-13
The above OID is polled at a fixed interval; the sne.property file defined the configuration of the
SNMP manager and the key configuration is:
snmp.version: SNMP version number
snmp.server: Switch IP address
snmp.community: community strings
cron.expression: SNMP Polling rate, */30 means every 30 seconds.
Figure 6-6
SNMP manager configuration
SNMP collected values are visualized in System Summary View:
Figure 6-7
Summary view using SNMP
10235028 6-14
6.6 Power Supply Failure Use Case
The following figure shows the possibilities of using SMMP to monitor for power supply
failures.
Figure 6-8
Power supply failure display using SNMP
6.7 Other Visualizations

This section contains several additional use case visualizations using SNMP.
6.7.1 Lost Packet Detection
From the GOOSE protocol format, the StNum are always the same and SqNum are always
incremental by 1 in normal condition. This behavior could be used in real-time to track the
package loss of the network. The protocol analysis module is looking at the following section of
GOOSE message:
10235028 6-15
Figure 6-9
Example of GOOSE protocol packet
Figure 6-10
Example of detecting GOOSE packet loss
6.7.2 IED Clock Error Detection

The IEC 61850 GOOSE packet contains a precision timestamp of when a state change was
detected. Using this information, it is possible to compare the GOOSE timestamp with a
precision local clock source and determine if there is a clock issue in the IED. This information
was exposed through the IEC 62351-7 MIB regarding clock information.
Figure 6-11
Time accuracy detection
10235028 6-16
6.7.3 Intrusion Detection
Unexpected GOOSE packets can be detected based upon a pre-defined knowledge of what
GOOSE messages should be sent within a specific substation. This information is provided by
the System Configuration Description (SCD) file. Using this information, rogue transmitters can
be detected through a real-time comparison of the source/destination MAC address pair for a
specific GOOSE.
Figure 6-12 depicts the possibility of detecting the source address associated with a GOOSE
message which can eventually lead to the ability of detecting intrusion.
Figure 6-12
GOOSE intrusion detection proof of concept
6.7.4 Traffic Statistics

IEC 62351-7 provides MIBs that allow the exposure of protocol specific network traffic
information. The following figures show possible displays of such information on a per IED or
aggregate basis.
10235028 6-17
Figure 6-13
GOOSE traffic statistics (per IED)
Figure 6-14
GOOSE network traffic aggregate view
6.7.5 Summary Display

The individual displays can be combined into a summary display that shows the protocol related
information the relevant information for a particular IED.
10235028 6-18
Figure 6-15
Protocol summary for an IED
10235028 6-19
10235028
7
CURRENT STATE OF IEC 62351-7
IEC 62351-7 Edition 1 is currently under revision to become IEC 62351-7 Edition 2. The basis
of the Edition 2 work is the previous EPRI work to refine and give definition to the MIB objects.
The EPRI definitions have been translated into UML and augmented/further refined.
Figure 7-1
Overview of structure of IEC 62351-7 edition 2
Each UML package contains further definitions of the MIB objects. As an example, the Device
Agent is currently defined as:
10235028 7-1
Figure 7-2
IEC 62351-7 edition 2 MIB structure for the device agent
The UML will be used to auto-generate the actual IEC standard. Publication of the document is
due for a ballot in the December 2014 timeframe.
10235028 7-2
8
SUMMARY
This report has set forth to discuss the history of NSM in the current utility environment. To
date, most NSM infrastructures deal with intermediate systems (e.g. routers, switches, firewalls,
etc.) and host server computers. The lack of NSM and security monitoring including the IEDs
leaves a gap in visibility. This gap means that certain use cases can’t be achieved and leaves the
potential of security threats to go undetected.
Several use cases were developed that showed how closing the visibility gap could improve the
overall integration of security and IT with OT. The use cases development the interactions and
exchanges of information that could occur and how such exchanges could be beneficial to the
overall operation of a utility. As part of the use case development, differences between current
utility operations and the possible future were discussed.
The advent of an implementable set of IEC 62351-7 MIBs and definitions allows the NSM
visibility gap to be closed and to construct demonstrations of the use cases. During the
development of the demonstrations, several observations were made:
• The current generation of IEDs do not support the information required to create the IEC
62351-7 MIBs. It is hoped that this will change as IEC 62351-7 is published and gains wider
visibility.
• It was found that the switches and routers had proprietary MIB information that could be
mapped into IEC 62351-7 MIBs to create a demonstration that utilized standardized MIB
definition.
• The use of the standardized MIBs allowed the use case demonstrations to be successful. In
some cases, proprietary MIBs were utilized unmapped for the purpose of convenience and
meet the timeframe.
• During the creation of the demonstrations, it was discovered that IEC 62351-7 did not have
information regarding clock synchronization or clock tampering. These MIB definitions were
developed and forwarded to IEC for standardization. IEC has tentatively accepted the
definitions that were developed.
Staging demonstrations of the use cases allowed for people to see the benefits of the extended
NSM architecture and to provide feedback and ideas for future demonstration expansion. The
feedback was positive with several utilities expressing interest for further work in the future.
10235028 8-1
10235028
9
REFERENCES
1. Network Security Management for Transmission Systems. EPRI, Palo Alto, CA: 2012.
1024421.
2. IEC 62351-7, “Network and system management (NSM) data object models,” Edition
1.0, 2010-07.
10235028 9-1
10235028
A
MIB UTILIZATION
proprietary MIBs, IEC 62351-7 MIBs, and other standardized MIBs. Since there were no demo
devices with native IEC 62351-7 MIBs, the mappings of vendor information to those MIBs are
shown.
A.1 CISCO
This section details the CISCO CGR/CGS information exposed via SNMp
A.1.1 CISCO Proprietary MIBs used directly
This section shows the proprietary CISCO MIBs that were utilized as part of the demonstration.
The following table shows that some of the proprietary MIBs could have been mapped to IEC
62351-7 MIBs, but this did not occur within the demo environment.
10235028 A-1
MIB OID
MIB Name Description IEC 62351-7 Equivalent Avaliable
(MIB Root <CISCO>)
The table of power supply status
ciscoEnvMonSupplyStatusTable <CISCO>.9.13.1.5 maintained by the environmental monitor Yes
card.
The table of ambient temperature status
Yes (IED.InternalTemp). This is a
ciscoEnvMonTemperatureStatusTable <CISCO>.9.13.1.3 maintained by the environmental
single value instead of a table.
monitor.
Yes, but has a restricted set of CPU
cpmCPUTotalTable <CISCO>.9.109.1.1.1 A table of overall CPU statistics
statistics.
A table of memory pool monitoring Yes (IED.MemUsage). This is a single
ciscoMemoryPoolTable <CISCO>.9.48.1.1
entries. value instead of a table.
A table of memory pool utilization
entries. Each of the objects provides a
general idea of how much of the
ciscoMemoryPoolUtilizationTable <CISCO>.9.48.1.2
memory pool has been used over a
given period of time. It is determined as
a weighted decaying average.
Yes, but indirectly through the
SysUpTime <SNMP MIB-2>.1.3 System Up Time from last Reboot
timestamp provided by IED.PwrOnCnt.
A.1.2 Standardized MIBs

This section details the non-IEC 62351-7 standardized MIBs that were utilized in the demo.
MIB Name MIB OID Description IEC 62351-7 Equivalent Available

Yes, but indirectly through the timestamp
SysUpTime SNMP MIB-2.1.3 System Up Time from last Reboot
provided by IED.PwrOnCnt.
10235028 A-2
A.1.3 IEC-62351 MIBs
This section details the IEC 62351-7 standardized MIBs that were utilized in the demo as well as the CISCO specific information that
was used by the gateway to map into the standardized MIB.
Agent MIB Name MIB OID Description CISCO Information Mapped

Timestamp of the change of state for
envPSPIedTS <IEC 62351-7>.1.1.4 Console Connection time
EnvPSPIed
envPSPPanelTS <IEC 62351-7>.1.1.6 N/A
EnvPSPanel.
envPSPBldTS <IEC 62351-7>.1.1.7 Timestamp of last Building Access. Use CGS HW Alarm
envPSPPerimiterTS <IEC 62351-7>.1.1.8 Timestamp of last Perimeter Access. User CGS HW Alarm
Status of application, sensors, or
envAppDatSt <IEC 62351-7>.1.1.1 <SNMP MIB-2>.1.3.0
software.
TimeStamp of the last power supply
envPSUPLosTS <IEC 62351-7>.1.1.13 <CISCO>.9.13.1.5
loss detected from the PSUP Table.
Environment
TimeStamp that the last power supply

envPSUPOnTS <IEC 62351-7>.1.1.11 <CISCO>.9.13.1.5
came on.
TimeStamp of the last AppDatSt
envAppDatStTS <IEC 62351-7>.1.1.2 <SNMP MIB-2>.1.3.0
change.
Status of application, sensors, or
envAppDatSt <IEC 62351-7>.1.1.1 <SNMP MIB-2>.1.3.0
software.
TimeStamp of the last AppDatSt
envAppDatStTS <IEC 62351-7>.1.1.2 <SNMP MIB-2>.1.3.0
change.
Indicates that direct access to the IED
envPSPIed <IEC 62351-7>.1.1.3
is in process
envPSPIedTS <IEC 62351-7>.1.1.4
EnvPSPIed.
Indicates that panel access is occurring
envPSPPanel <IEC 62351-7>.1.1.5
to the IED.
10235028 A-3
envPSPPanelTS <IEC 62351-7>.1.1.6
EnvPSPanel.
envPSPBldTS <IEC 62351-7>.1.1.7 Timestamp of last Building Access. Using HW Alarm
envPSPPerimiterTS <IEC 62351-7>.1.1.8 Timestamp of last Perimeter Access. Using HW Alarm
Index of the power supply that was the
envPSUPOnIndex <IEC 62351-7>.1.1.10 <CISCO>.9.13.1.5
last to come on.
TimeStamp that the last power supply
envPSUPOnTS <IEC 62351-7>.1.1.11 <CISCO>.9.13.1.5
came on.
pSUAPTable Entry Index of the last
envPSUPLosIndex <IEC 62351-7>.1.1.12 <CISCO>.9.13.1.5
power supply that was detected as lost.
Environment (continued)
TimeStamp of the last power supply

envPSUPLosTS <IEC 62351-7>.1.1.13 <CISCO>.9.13.1.5
loss detected from the PSUP Table.
envPSUPTable <IEC 62351-7>.1.1.14 A list of Power Supply entries <CISCO>.9.13.1.5
Information for a particular power
envPSUPEntry <IEC 62351-7>.1.1.14.1 <CISCO>.9.13.1.5
supply.
pSUPName <IEC 62351-7>.1.1.14.1.1 Name of the power supply. <CISCO>.9.13.1.5
pSUPDescr <IEC 62351-7>.1.1.14.1.2 Description of the power supply. <CISCO>.9.13.1.5
Indicates if the PowerSupply state is
pSUPPwrOn <IEC 62351-7>.1.1.14.1.3 currently on and operating approriately <CISCO>.9.13.1.5
(e.g. TRUE).
Timestamp of the last power-on.. This
pSUPPwrOnTS <IEC 62351-7>.1.1.14.1.4 value does not reset if CntRs is set <CISCO>.9.13.1.5
True.
Timestamp of the last power-down..
pSUPPwrLosTS <IEC 62351-7>.1.1.14.1.6 This value does not reset if CntRs is set <CISCO>.9.13.1.5
True.
pSUPCntRs <IEC 62351-7>.1.1.14.1.7 Reset structure for counters. <CISCO>.9.13.1.5
pSUPCntRsTm <IEC 62351-7>.1.1.14.1.8 Time of last counter reset. <CISCO>.9.13.1.5
10235028 A-4
Timestamp of the last detected power

pSUPwrLosCntTm <IEC 62351-7>.1.1.14.1.9 <CISCO>.9.13.1.5
loss.
Environ-
(cont’d)
ment
Counter of the number of times the

pSUPPwrLosCnt <IEC 62351-7>.1.1.14.1.5 <CISCO>.9.13.1.5
power supply has lost power.
Attribute of the status of an application,
iEDAppSt <IEC 62351-7>.1.2.3 <SNMP MIB-2>.1.3.0
end system, or software module.
Provides a count of the number of
iEDAppStrCnt <IEC 62351-7>.1.2.5 transitions to the running or resetting <SNMP MIB-2>.1.3.0
states of AppSt.
TimeStamp of the last IEDAppSt value
iEDAppStTS <IEC 62351-7>.1.2.4 <SNMP MIB-2>.1.3.0
change.
Attribute that provides the number of
iEDAtkCnt <IEC 62351-7>.1.2.7
cyber- attacks that have been detected.
Timestamp of the last count change of
iEDAtkCnttTm <IEC 62351-7>.1.2.8 iEDAtkCnt not due to a commanded
reset.
Provides the last known attack vector
IED
iEDAtkTyp <IEC 62351-7>.1.2.9

that was encountered
Count of the number of buffer overflows
detected. Buffer overflow is defined as
iEDBufOvCnt <IEC 62351-7>.1.2.10 the attempt to allocate a resource,
typically a memory buffer, and the
resource is not available.
iEDBufOvCntTm <IEC 62351-7>.1.2.11 iEDBufOvCnt not due to a commanded
reset.
Count of the number of buffer
underflows detected Buffer overflow is
iEDBufUnCnt <IEC 62351-7>.1.2.12 defined as the attempt to allocate a
resource, typically a memory buffer,
and the resource is not available.
10235028 A-5
iEDBufUnCntTm <IEC 62351-7>.1.2.13 iEDBufUnCnt not due to a commanded
reset.
attempts to invalid control requests.
Invalid control counts includes control
iEDCntlPrivFailCnt <IEC 62351-7>.1.2.24 requests to non-existent control objects, Maybe
incorrect control service (e.g. SBO vs.
SBO with enhanced security) and
access role violation.
iEDCntlPrivFailCntTm <IEC 62351-7>.1.2.25 iEDCntlPrivFailCnt not due to a
commanded reset.
Provides capability to reset statistics
IED (continued)
iEDCntRs <IEC 62351-7>.1.2.39 and Does not reset the timestamps Maybe
associated with the counters.
iEDCntRsTm <IEC 62351-7>.1.2.40 iEDCntRs not due to a commanded
reset.
Provides an attribute that indicates the
iEDCPU <IEC 62351-7>.1.2.19 percentage of processing resources <CISCO>.9.109.1.1.1
that is in use. Range is 0-100
attempts to write invalid data. Invalid
iEDDataInvCnt <IEC 62351-7>.1.2.16
data includes out-of-range and access
role violation.
iEDDataInvCntTm <IEC 62351-7>.1.2.17 iEDDataInvCnt not due to a
commanded reset.
This entry provides a textual description
iEDDescr <IEC 62351-7>.1.2.2 the describes the IED or device Description
provider.
10235028 A-6
Indicates the number of expired
iEDExpCerts <IEC 62351-7>.1.2.35 certificates that are in the certificate
storage configured for local use.
iEDMem <IEC 62351-7>.1.2.18 percentage of memory that is in <CISCO>.9.48.1.2
use.Range is 0-100
Provides an attriubute that represents
iEDMisEvCnt <IEC 62351-7>.1.2.20
the count of missed events or data.
iEDMisEvCntTm <IEC 62351-7>.1.2.21 iEDMisEvCnt not due to a commanded
reset.
iEDName <IEC 62351-7>.1.2.1 Name of the device or IED. Hostname
Indicates the number of kilobytes of
IED (continued)
iEDNvStore <IEC 62351-7>.1.2.37 <CISCO>.9.48.1.2

storage allocated for local use.
Indicates the percentage of NvStore
iEDNvStoreRem <IEC 62351-7>.1.2.38 <CISCO>.9.48.1.2
available for storage. Range is 0-100
Number of access requests that failed
iEDPrivFailCnt <IEC 62351-7>.1.2.22 Maybe
due to insufficient privilege.
iEDPrivFailCntTm <IEC 62351-7>.1.2.23 iEDPrivFailCnt not due to a
commanded reset.
iEDPwrOnCnt <IEC 62351-7>.1.2.28 Number of power-ons detected. <CISCO>.9.13.1.5
iEDPwrOnCntTm <IEC 62351-7>.1.2.29 iEDPwrOnCnt not due to a commanded <CISCO>.9.13.1.5
reset.
Indicates the number of revoked
iEDRevCerts <IEC 62351-7>.1.2.36 certificates that are still configured for
local use.
Precision of time synchronization in
iEDSynPrec <IEC 62351-7>.1.2.32 maybe
nano-seconds.
10235028 A-7
Indicates if there is a problem with time
iEDTimSync <IEC 62351-7>.1.2.33 maybe
synchronization.
iEDTimSyncSrc <IEC 62351-7>.1.2.34 source of time synchronization that is in maybe
use.
IED (continued)
Provides an attribute that represents

iEDUnAuthAcsCnt <IEC 62351-7>.1.2.26 the count of un-authorized access Maybe
attempts.
Timestamp of the last un-authorized
iEDUnAuthAcsCntTm <IEC 62351-7>.1.2.27 Maybe
access attempt.
iEDWrmStrCnt <IEC 62351-7>.1.2.30 Number of warm starts detected. <CISCO>.9.13.1.5
iEDWrmStrCntTm <IEC 62351-7>.1.2.31 iEDWrmStrCnt not due to a Maybe
commanded reset.
10235028 A-8
10235028
Export Control Restrictions The Electric Power Research Institute, Inc.
Access to and use of EPRI Intellectual Property is granted (EPRI, www.epri.com) conducts research and
with the specific understanding and requirement that development relating to the generation, delivery
responsibility for ensuring full compliance with all applicable and use of electricity for the benefit of the public. An
U.S. and foreign export laws and regulations is being independent, nonprofit organization, EPRI brings
undertaken by you and your company. This includes an
together its scientists and engineers as well as
obligation to ensure that any individual receiving access
hereunder who is not a U.S. citizen or permanent U.S. experts from academia and industry to help
resident is permitted access under applicable U.S. and address challenges in electricity, including
foreign export laws and regulations. In the event you are reliability, efficiency, affordability, health, safety and
uncertain whether you or your company may lawfully obtain the environment. EPRI also provides technology,
access to this EPRI Intellectual Property, you acknowledge policy and economic analyses to drive long-range
that it is your obligation to consult with your company’s legal
research and development planning, and supports
counsel to determine whether this access is lawful.
Although EPRI may make available on a case-by-case research in emerging technologies. EPRI’s
basis an informal assessment of the applicable U.S. export members represent approximately 90 percent of the
classification for specific EPRI Intellectual Property, you and electricity generated and delivered in the United
your company acknowledge that this assessment is solely States, and international participation extends to
for informational purposes and not for reliance purposes. more than 30 countries. EPRI’s principal offices and
You and your company acknowledge that it is still the
laboratories are located in Palo Alto, Calif.;
obligation of you and your company to make your own
assessment of the applicable U.S. export classification and Charlotte, N.C.; Knoxville, Tenn.; and Lenox, Mass.
ensure compliance accordingly. You and your company
Together…Shaping the Future of Electricity
understand and acknowledge your obligations to make a
prompt report to EPRI and the appropriate authorities
regarding any access to or use of EPRI Intellectual Property
hereunder that may be in violation of applicable U.S. or
foreign export laws or regulations.
© 2014 Electric Power Research Institute (EPRI), Inc. All rights reserved.
Electric Power Research Institute, EPRI, and TOGETHER…SHAPING THE
FUTURE OF ELECTRICITY are registered service marks of the Electric
Power Research Institute, Inc.
3002003738
Electric Power Research Institute

3420 Hillview Avenue, Palo Alto, California 94304-1338 • PO Box 10412, Palo Alto, California 94303-0813 • USA
10235028800.313.3774 • 650.855.2121 • askepri@epri.com • www.epri.com

Network System Management - Implementation and Applications of The IEC 62351

Încărcat de

Informații document

Titlu original

Drepturi de autor

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Network System Management - Implementation and Applications of The IEC 62351

Încărcat de

Drepturi de autor:

Network System Management: Implementations and

Applications of the IEC 62351-7 Standard

EPRI Project Manager

ELECTRIC POWER RESEARCH INSTITUTE

This report describes research sponsored by EPRI.

In order to achieve the integrated/smart grid, there is an expansion of communication capability

2.2 Abbreviations and Acronyms

Notation MIB Root OID

<MIB-2 Interfaces> 1.3.6.1.2.1.2

<Private Enterprise> 1.3.6.1.4.1.35406

3.1 NSM Comparison

FCAPS FCAPS IEC 62351-7 Event Detection and

3.2 Legacy Power Device Monitoring and Management Strategies

Primary and Intermediate

Primary and Intermediate

SCADA/ NSM Maintenance

SCADA/ NSM Maintenance

Application Entity Device/IED

DNP, IEC 60870-5,

The MIB hierarchy is shown in Figure 3-11.

IEC 6235-7 IEC 6235-7

4.1 Use Case Development and Documentation

4.2 Implementation Architectures

IED or Intermediate Systems or

IED with no SNMP SNMP

5.2 Traffic Analysis

From RxPDU, TxPDU

From RxPDU, TxPDU

IT Policy Defined Sequence

5.3 Resource Exhaustion

From Resource Information

5.4 Component Failure and Degradation Alarms

From Component Infromation

Component Information (Failure)

The following IEDs are connected to test environment:

Name AREVA P145

Name GE D60 1 CTRL

6.2 SNE Software Architecture

6.3 SNE Deployment Requirements

6.4 Network Topology Visualization

6.5 Resource Monitoring Use Case

CISCO MIBs Utilized

RuggedCom MIBs Utilized (continued)

Radiflow MIBs Utilized (continued)

SNMP collected values are visualized in System Summary View:

6.7 Other Visualizations

6.7.2 IED Clock Error Detection

6.7.4 Traffic Statistics

6.7.5 Summary Display

A.1.2 Standardized MIBs

MIB Name MIB OID Description IEC 62351-7 Equivalent Available

Agent MIB Name MIB OID Description CISCO Information Mapped

TimeStamp that the last power supply

TimeStamp of the last power supply

Timestamp of the last detected power

Counter of the number of times the

iEDAtkTyp <IEC 62351-7>.1.2.9

iEDNvStore <IEC 62351-7>.1.2.37 <CISCO>.9.48.1.2

Provides an attribute that represents

Electric Power Research Institute

S-ar putea să vă placă și