End to End Testing - What Should You Know?

A. Apostolov, IEEE Fellow, B. Vandiver, Member IEEE (OMICRON electronics)

Abstract - This paper discusses the increasing and hundreds into one black box that only had the
interest in performing End to End testing on base required Inputs and Outputs. Segregation of
modern protection systems. There are many the protection functions for the purpose of testing
and varied approaches to this test method, from
was not the primary design requirement; the
simplistic to overly complex. What is really
economy for creating the overall protection
required and what really makes this test
method valuable? What are the technical scheme through mUltiplexing internal functions

challenges and how to resolve them? These and and leveraging programmable logic was however.
other questions are answered. Using the legacy testing method of static
secondary injections proved laborious for a multi­
This paper makes a comparison of the test
function protection relay that had become the
tools/approaches that are commonly used. It
complete protection system.
discusses the payback levels of investing in
proper preperation and the information
Back in the late 1980's, some utilities began to
required for each type of test method. It
explore using a system testing method on their
presents real case studies of various techniques
used on Step Distance / POTT / Current transmission line applications that potentially

Differential and issues associated with time could reduce the overall testing workload. This
sync that can make or break the test. method was dubbed panel injection testing since it
required a 3-phase injection of voltage and current
Index Terms- Protection system, and was the prelude to End-to-End Testing (E2E).
maintenance, testing, reliability, NERC, FERC, Mastering this method required more preparation
IEC 61850 time and extra equipment over typical I-phase
injection methods, but the successful execution in

1 Introduction the field would significantly reduce the man-hours

on site. Further, the test results could provide
The commissioning/testing of a Protection & insight into the overall protection panel scheme
Control System is the last step in the overall health and performance.
engineering process of system study, protection
concept, design, purchase, build, and installation
2 Why End to End Testing?
of such a system. In decades past, the needs of
testing were inherently designed into the various When a protection system is applied there are
components that made up the protection & control many components and apparatus that make up the
systems of the day. The majority of these devices complete Fault Clearing System (FCS) as seen in
could be discretely tested since they were designed Fig. 1. In an ideal world we want to prove the
as draw out case devices. complete integrity of the FCS before we put it into
service (Commissioning), periodically to ensure
The testing method used was naturally a discrete
continued health and performance (Routine
functional test approach; this meant a single­
testing), and after an unexpected misoperation
function test for a singular device. (We equate this
(Troubleshooting). Both manufacturers and
to electro-mechanical [E/M] schemes.) As
utilities used the system testing method
protection devices evolved from electro­
(Acceptance testing) on a new protection concept
mechanical designs to modern digital complex
or product as due diligence to ensure specification
systems, they typically lost several degrees of
compliance and a successful application in their
testability. This due to the fact that now most of
the single-functions were combined by the tens

978-1-4799-4739-3/14/$31.00 ©2014 IEEE 125 ProRelay 2014

 End to End Testing - What Should You Know?
grid. So it makes sense to carry this testing method
over to all testing requirements.
� Fault Clearing sy7t e";;; -
---
-
-------
� •
Figure 1: Scope of System Tests on FCS
As primary injection and actual operation of the
switches, breakers, and other primary apparatus
was cost prohibitive, covering as much scope as
possible of the FCS resulted in secondary injection
test equipment being up scaled to the portable
power system simulators we have today.
In the 1980's performing a 3-phase injection to a
protection system was accomplished by using 3-
single phase test sets ganged together. By adding a
GPS clock to sync the PC clock and/or test sets,
early E2E tests were attempted. (Fig. 2) But just
adding a GPS (Global Positioning System) input
did not instantly make it work, it goes a little
deeper than that.
[Q[Q[Q]
DOD
���
�
3V, 31, Trip
Fig. 2 - ElM Panel and Early GPS Test
Technical challenges aside, utilities could really
benefit from making the system test method the
preferred test process because it can detect:
• a bad or failing relay output contact,
• a failed 52B contact (52A contact too),
• failing wire insulation or loose connection,
- - - - -
bad turns of an interposing CT,
• inadvertent change of settings,
• an event log or report fails to generate,
• proper interlocking with another device,
• correct operation of a remote device,
• or measurement inaccuracies.
Most of which are more common problems than a
single function setting error in a digital relay. As a
rule, system testing takes, as its input, all of the
"integrated" components (functions) that should
have passed "Acceptance" testing including the
interconnected 110 of the fault clearing system.
This may be inclusive of the digital relay only or
be a combination of relays as a distributed
protection system. This is also true of any
communication based protection scheme or remote
device in a dedicated application such as Line
Current Differential.
Moreover, system testing can be used to
investigate the protection system's performance or
operations. Did it trip correctly? What condition
caused it to trip? Why did the relay's protection
logic misoperate? The behavior and the expected
performance of the relay in its designed power
system application can never be verified with a
single function test. Both positive and negative
"system" test cases are therefore required.
The best benefit is that a properly defined series of
system tests can be used to identify all of these
conditions and document them without settings
changes to the relay(s). All of this being true, why
is the legacy test methods of the 1980's still being
used?
3 Engineering Preparation
Perception is often taken as reality and in the
discussion of system testing or preparing for an
E2E test, most would say that it requires too much
126
 End to End Testing - What Should You Know?

engineering preparation or is too difficult to
execute successfully. Well, not really. It involves
three basic things; understanding the problem
domain, access to information, and using the
proper tools.
critical part is that each protection terminal can
properly identify the fault location of each test
case. Using a little discretion, we can allow a
proper margin based on the worst case phase error
accuracy.

Reference the 3-Zone POTT scheme of Fig. 3. The

problem domain is defining the test cases to prove
�Z21 BZ1 BZ3

the overall scheme operation/coordination with : 5 ubstation A

,(

5
I
ubstation 8 :
proper fault detection. By definition, we should : ;
F1
__ _ _ ____ _ ___ ____ ____ _ :
F2 F3

,-- -----------1 --- - -- - -

know and understand the operational limits of the

: :"'@}'JZone11 ; : IZone1f"0"': :
protection scheme and that defines the test set :
: : ,·fiXl··; ···· . /··;··rrxr··, : :
: ! �-�--�-�---�·� ���---:--�- � --� ! :
'

performance and accuracies we will need to

perform the tests. :�-IZ�ne 21 ; IZone'21-� :
:.[gtlJ�o�e 3.1. .1 z.on e?JIg!L
..
___

.
. . u . . .'

To make the best test cases we need:

Fig. 3 - Typical3-Zone w/POTT scheme
• Correct secondary voltage used
• Nominal CB operation times With a Fault Model Tool, we can use the Positive
• Proper 52a152b simulation Sequence Line Impedance and create simple
• Line Impedance Zl and ZO (Pos/Zero Seq) PreFaultlFaultiPostFault states for each test case
values we need. From Fig. 3 we see five fault zones
identified, F 1-F5. Using Substation A as the
• Zone 1, 2, 3 reach and delay times
reference we could create a test sequence based on
• Typical propagation delay of the Comm
percent of line length and create the reciprocal test
channel
states for Substation B. The table below shows the
• Synchronized start of the test sets within
synchronized injections at both A and B with
our application limits, especially at fault
expected operation results.
inception
Fault Case %ofLL(A) SubAOp SubATime %ofLL(B) SubBOp SubBTime
But even ballpark numbers can allow us to F1 10 Zl+PTT-Send <30mS 90 Z2+PTT-Recv <50mS

F2 50 Zl+PTT-Send <30mS 50 Zl+PTT-Send <30mS

construct the test cases and for this application the F3 90 Z2+PTT-Recv <SOmS 10 Zl+PTT-Send <30mS

F4 120 Z2+No PIT Notrip ·20 Z3+No PIT NoTrip

simulation can even be quite basic. For instance, F5 ·20 Z3+No PIT Notrip 120 Z2+No PIT NoTrip

SPECIAL CASES - PIT Failure / Out ofSeg:ment Failure

the nominal secondary voltage for HV line F1·NoPTI 10 Zl+PTT-Send <30mS 90 22T <330mS

F3·NoPTI 90 22T <330mS 10 Zl+PTT-Send <30mS

applications is typically 67V, the CB operate times F4·No Blk 120 22T <330mS ·20 23T <630mS

F5·No Blk
are 2, 3, or 5 cycle depending on make and system ·20 23T <630mS 120 22T <330mS

voltage, Zone settings are known from the relay(s)

and the Line impedance is typically included there
too. Propagation delays on Comm channels can be
generalized; 12-20mS for power line carrier, 1-
4mS for fiber links, 4-8mS for microwave links.
Because this is a Step-Distance scheme, the timing
accuracy of the synchronized start can be +/­
O.5mS and still be successful, as this equates to a
phase error of about 10 degrees @ 60Hz. The
Table A - Test Cases for POTT Scheme
For this application, we should be able construct a
typical step change simulation sequence
(PreFaultlFaultlPostFault) for the power system
events above that would cause deterministic
operations as described (Table A). It would also be
a good idea to vary which fault loop is used per
test case, and of course match it to the other end.

127
 End to End Testing - What Should You Know?
Hint: Some relay algorithms may be sensitive to a
phase and magnitude step change in the currents,
to avoid this keep the current's phase unchanged
between PrefauItlFauIt states only changing the
magnitude. Instead change the voltage phase
reference for the specific fault impedance phase
angle.
4 Better Testing Tools
For our POTT example it should be explained that
in order to get the most out of the test cases
described, we need to be able to monitor more
than just the General Trip from the protection
system. In this application we should also monitor
at the same time, the PTT send/receive status, the
52a1b status (even if simulated), and if possible
each Zone 1, 2, 3, Pickup and Timed operations.
There are others as well, CB Close, Breaker
Failure Initiate, Reclose Enable just name a few.
Obviously, this requires access to both dry
contacts; control voltage wetted signals, and
maybe LED status implying optical pickups.
Based on these requirements, we would need a
minimum of eight (8) status inputs to possibly 12
inputs for our test set. What if you are now testing
a distributed protection scheme with independent
pole operation, covering six protection systems at
different physical locations like Fig. 4? We
definitely need GPS sync, and even more status
monitoring capability.
Fig. 4 - Distribution Automation Loop Test
128
We would probably need new test tool capabilities
too that can generate the required test cases for
each location based on the system parameters.
For testing a Line Differential scheme using E2E
test methods; the test sync start accuracy now must
be improved over the POTT or DA scheme. As
noted, the 10 degree phase error was fine for the
POTT and even the DA scheme, but for the Line
Diff it would likely need to be much more
accurate. With a sync start of +/- O.l mS we get a
phase error of 2. 18 degrees @ 60Hz. So if we
intend on testing the load swing limits of our Line
Diff we would definitely need at least this
accuracy, better if possible. Verifying a sensitive
blocking zone on a short line for this scheme could
require even higher sync accuracy. With an order
of magnitude better accuracy (O.Ol mS or l OuS) we
could realize a phase error of +/- 0.2 18 degrees.
Most would consider this really good, but today
we have many schemes and applications that
require accuracies of +/- IuS. (PMU's, PDC's, SV
of IEC 6 1850-9-2)
The facts are a really good GPS receiver can be +/-
15 nanoseconds accurate when locked on 6+
satellites and implementing some good
compensation algorithms. A common commercial
GPS receiver is 1- 10uS accurate. From that point,
every node, transition, or translation will add to
the delay and uncertainty. Using a time code based
clock distribution (i.e. IRIG-BI22) we can realize
a native +/- l OOuS accuracy. (Ius if a PLL is used
to sync to the carrier) Even so, a modem test set
can introduce 400-1200 microseconds delay on a
start trigger due to its DAC initialization, edge
detection, and its reconstruction filter. So to
achieve +/- 1 uS accuracy we need a new
technology, we need to be 4-5 times better. IEEE
1588-2008 and IEEE C37.238-20 1 1 provide this
capability. Both are currently in revision again, but
already products using these standards are
achieving the desired results.
 End to End Testing - What Should You Know?

Hint: It is always best practice to use matched test
set hardware when executing any synchronized
test method. Each technology employed has
different limitations and benefits. It is possible to
use mixed hardware and even mixed test set
manufacturers, but it is always advised to make a
bench test, configured exactly as they will be used,
to document the start delays and phase offsets in
order to properly compensate for them in the test
software and test cases used.
5 Test Simulation Requirements
Of course it takes more than the hardware
specifications to achieve the testing acumen we
need. This falls on the software tools we use to
both configure and control the test set hardware. If
we concede that the hardware is not an issue, then
using the best simulation tools should make the
testing process simple. (With one small caveat.)
location allows for greater test flexibility,
especially for troubleshooting. Last, the ability to
add S2a/b or other contact simulations or alter
them in the file is a big plus to get good results.
When using a Network Simulation tool the output
is typically a sample stream similar to the
COMTRADE file. But the main difference is that
the simulation will alter its outputs based on the
parameters used and what the relay does, like CT
saturation, DC offset, or stops the currents and
voltages when the relay calls for trip and react
with a proper or improper breaker operation based
on the test case used. This allows the relay to react
as it is programmed. So the test results are more in
line with the real world situation. Of course this
requires more detailed system study information
and power system parameters, but if the goal is to
obtain confidence in the given protection system
as installed, then it can prove invaluable.

Any power system simulation software tool

requires information to achieve a usable result
from it. The old adage, "garbage in equals garbage
out" rings very true in this case. We can adapt and
generalize some parameters to get baseline or
worst case operational results, but since most of
the data is available from a system study we __ Tl!!!1C2!!II'!llLnI�lQ!L __
lmk

should just get it and use it. Additionally, the

software tool used directly contributes to the ease
of use in carrying out a given E2E test. In many
cases, the ability to adapt to actual field conditions
is key to a successful E2E test.
Fig.5 - Typical End-ta-End Test setup
For instance, when using COMTRADE files as
test cases, the two ends must both see the fault
inception at the same time. COMTRADE is a 6 Critical Success Factors
sample stream, so when the file begins playback
that is exactly what you get, start to finish. Requirement # 1 - put your E2E test method into
the Engineering Process so that it can be used
For many relays, they must see enough PreFault in consistently and regularly across the organization.
order to initialize their routines and logic before Training is a requirement, not an option.
the fault occurs. So having a software tool that can Document each step for getting correct
adjust the amount of PreFault sample time is information and where it comes from; and then
important. (Remember to do it to files at both also where the test results go.
ends) Additionally, if the COMTRADE file comes
Requirement #2 - Perform Acceptance tests on all
from a fault recording, the ability to adapt it for
discrete components in a lab or controlled
different CTNT ratios and add delays to change
environment. Prove the device meets spec and
the fault inception time relative to the desired fault
accuracy claims and is suitable for your protection

129
 End to End Testing - What Should You Know?

application. This eliminates the need for further
functional testing unless it is E/M.
Requirement #3: Use the same system test cases
regularly on new or existing relays of the same
application for both commissioning and routine
testing.
This process will more quickly verify any new
firmware update, setting changes or a substitute
manufacturer's relay to be used in that same
application.
Requirement #4: verify the health and availability
of the digital protection relay/system in situ
without violating its commissioned status or
performing excessive testing.
This can only be accomplished by understanding
the device's design, algorithm's, and performance.
This is directly related to how it is to be used in
the designed protection application
Requirement #5: when possible, use matched test
equipment and the best accuracy clocks available,
and match the equipment capabilities to the
protection application requirements. Know your
gear.
requirements for intrusive functional testing and
covers not only the protection devices themselves,
but also most components of the FCS. Both test
equipment and the software tools that run them
can provide these advantages for most protection
applications.
8 References
[ 1] A. Apostolov, B. Vandiver III: "Maintenance
Testing of Multifunctional Distance Protection
IEDs", IEEE T&D Conference, New Orleans, LA,
April 20 10
[2] A. Apostolov, B. Vandiver III: "The Primary
Reason for Adopting System Testing Methods",
PAC World Conference, Dublin, Ireland, June
20 1 1.
[3] A. Apostolov, B. Vandiver III: "How Will PTP
Impact Protection Automation & Control
Testing?", Power Energy & Automation
Conference, Spokane, WA, 2014.

Requirement #6: when executing E2E tests

coordinate the test case sequences for each end
and ensure the power system will be properly
modeled or represented. Know what the protection
system does and how it should respond.

Requirement #7: automate as much of the data

exchange into the test software and/or test cases.
Manual manipulation of settings and parameters
leads to errors.

Requirement #8: Share your experiences and

knowledge.

7 Conclusions

In summary, it cannot be stressed enough that to

properly maintain a modern protection system one
must begin at the planning and engineering phase
in order to provide the proper design, components,
programming and configuration of the protection
system. New standards and technology coupled
with proper system test methods like End-to-End
testing allow a significant reduction in the

130
 End to End Testing - What Should You Know?

Biographies

Alexander Apostolov received a

MS degree in Electrical
Engineering, MS in Applied
Mathematics and Ph.D. from the
Technical University in Sofia,
Bulgaria. He has more than thirty
years experience in the field of electric power
systems protection, control and communications.

He is presently Principal Engineer for OMICRON

electronics in Los Angeles, CA. He is an IEEE
Fellow and Member of the Power Systems
Relaying Committee and Substations CO
Subcommittee. He is the past Chairman of the
Relay Communications Subcommittee, serves on
several IEEE PES Working Groups and is
Chairman of Working Group D2l : Investigate
Supporting of IEC Standard for Distance Relay
Characteristics.

He is member of IEC TC57 and Convener of

CIGRE WG B5.27 and member of several other
CIGRE B5 working groups. .He is Chairman of
the Technical Publications Subcommittee of the
UCA International Users Group. He holds three
patents and has authored and presented more than
300 technical papers. He is also Editor-in-Chief of
the PAC World magazine.

Benton Vandiver III received

BSEE from the University of
Houston in 1979.

He was with Houston Lighting &

Power for 14 years and Multilin Corp. for 4 years
before joining OMICRON electronics in 1995
where he is currently Technical Director in
Houston, TX. A registered Professional Engineer
in TX, he is also an IEEE / PSRC member, USNC
member, CIGRE corresponding member. He holds
a US Patent for "Communication-based Testing of
lED's" and has authored, co-authored, and
presented over 90 technical papers.

131