Documente Academic
Documente Profesional
Documente Cultură
Coverage Metrics for Device Level Validation of SATA and SAS Devices -An
Approach
Raja Subramani, Ramya Penneru, Glamar Selvaraj, Bharath Radhakrishnan, Krishnamurthy Puttaiah
Integrated Engineering Services
Tech Mahindra
India
Raja.Subramani@TechMahindra.com, Ramya.Penneru@TechMahindra.com
Glamar.Selvaraj@TechMahindra.com Bharath.Radhakrishnan@TechMahindra.com,
Krishnamurthy.Puttaiah@TechMahindra.com
Abstract- This paper highlights the significance of Coverage state drive possible. Coverage is quantitative test metrics
analysis with respect to the functionality and adherence to the and can be utilized for the mentioned cause and can play a
standard specification of SATA, SAS, NVME & eMMC significant role in determining the superiority of the
Devices (SSD’s, HDD’s). Coverage analysis aims at achieving particular storage device. Coverage is a statistical
an effective way of testing the intrinsic functionality of the representation of the number of scenarios or test cases or the
storage devices. Commands are issued to the Device under test various combinations the parameter under coverage has
to validate the attributes such as methods of transfer, latest attained. Coverage Analysis is used as a benchmark for
technology involved to structure the storage device
functionality, performance and the adherence to the protocol
validating whether the device under test is completely
(SATA protocol). Validation environment of the storage device adhering to the functionality described as per the standard
will include innumerable test cases which lead to redundancy specification.
of test scenarios and may cause gaps in the whole validation During the course of research the following were utilized
procedure rendering the goal to achieve a thorough validation for obtaining the results and updating the Coverage
output. Validation of each Target Connected device.
Validating Tool: MSDV 2013 Pro SATA, SAS, NVME &
Keywords: Functional Coverage, Command Coverage, HDD, eMMC in-house proprietary tool by TechMahindra, Inc.
SSD, SATA, SAS, NVME, eMMC, Coverage, Coverage Analysis Protocol Analyser: SerialTek BusXpert 4.0.883 by Agilent
I. INTRODUCTION Technologies, Inc.
The non-linear growth of the SSD market has led to a II. NEED FOR COVERAGE
significant influx of new SSD-Solid State Drive vendors Functional requirements are described as metrics to make
into the market. In 2002 there were less than 10 SSD the test quality measurable to obtain a high functional
vendors; today there are over 100 vendors, and this number coverage in the given amount of time that can be used to test
is growing exponentially. The product attributes of the given the system. During a test run on the chosen target, the
SSD product alone cannot be considered as a differentiating functional coverage is collected and back annotated to the
factor to distinguish which of the three emerging SSD metric describing the test quality. Measuring test
categories the product belongs to: Consumer, Commercial completeness allows the test engineer to concentrate on
or Industrial Grade. Consumer Grade SSDs are the most areas not covered by current tests. To increase test coverage,
visible in the market since they are optimized for the high- each test can be varied with random data and control flow
volume, low-cost per gigabyte applications such as laptop elements. This allows even reaching test scenarios which are
computing. The distinction between Commercial and not described in the specification, but are possible scenarios
Industrial Grade drives is far less clear since SSD vendors if the software interface is used as specified in the legal
can deliver very similar market outputs. An efficient way to ranges. This avoids holes in the test environment, where
understand whether an SSD is truly Industrial Grade is by only features are tested that have been specified before.
comparing the design, testing and manufacturing processes This provides a deterministic metric of progress for test
used by the vendors being evaluated. SSD has no moving development.
parts and is essentially an HDD-Hard Disk Drive emulator. The coverage efficiency of two test suites may also be
It is basically comprised of a printed circuit board, a set of compared. The most useful measure is probably the amount
NAND flash memory chips, SDRAM cache, a memory of simulation resources required, for example the total
controller, an interface controller and an interface connector length of all the test cases. By knowing the correspondence
such as IDE or SATA or SAS or even fiber channel. between the micro-architectural events at the boundary of
Thorough design verification on operational devices, the unit under test and architectural events it is possible to
extensive burn-in testing during manufacturing on every translate abstract tests into architectural level test
single device (vs. lot sampling), and comprehensive quality specifications. Coverage analysis provides any validation
assurance procedures will lead to the highest quality solid
Fig.3 BusXpert Protocol Analyzer set up Options for efficient coverage
Validation
Fig.1 MSDV Tool Setup for coverage Validation
IV. COVERAGE METRICS CLASSIFICATION
A. Mass Storage Device Validator (MSDV)
Task File interface is used for the command delivery and Coverage analysis is based on the set of variables defined
data transfer. SATA-Serial Advanced Technology for coverage. Variables defined are utilized to differntiate
Attachment, interface is built on the foundation of this Task between coverage scenarios and can be merged to get a
File architecture. SATA uses NCQ which allows systems to complete coverage analysis. The different types of coverage
take advantage of a queuing structure in the SATA interface. depend upon each other and may sometimes may be
It lets device to reorder multiple commands in order to mutually exclusive to their own context. These Coverage
minimize the rotational latency and track access overhead. metrics provides the validation engineer more pipelined and
AHCI is a host system interface designed for this purpose: more accurate results to improve test scenarios. The
to support NCQ and to provide a platform for hardware and coverage merics are broadly classified as follow:
software vendors to use. Intel has leaded the industry to A. Specification coverage
define a NEW TASK FILE: AHCI for the next generation B. Functional coverage
storage interface. It prepares a command base for hardware C. Command coverage
vendors and software drivers to build a higher performance D. Parameter coverage
storage interface. The mentioned Tool provides the features E. Test scenario Coverage
required for device level testing. A. Specification Coverage
MSDV provided the feature of calculating the command This Coverage analysis signifies the measure upto which
coverage and the parameter coverage. It calculated the the device under test is adhering to the standard
number of commands which were executed on the device specification. This coverage is calculated with respect to the
and the various possible input combinations of the scenarios mentioned in the specifications. The standard
parameters as the parameter coverage. specifications imply the protocols to be followed and
specification coverage is aimed at determining whether the
164
device is following those protocols and is giving expected V. COVERAGE METRICS FOR SATA DEVICES.
results.
SATA is used to transfer information between host and
B. Functional Coverage target connected device and vice-versa. This section
Functional Coverage aims at providing the measure upto describes the coverage metrics defined or validated for all
which the the functions or the features provided in the devices which follows SATA protocol. The coverage
device adheres to those present in the standard specification. defined for all ATA Commands, Interface commands,
Whether the features metioned in the specifations are SATA Specification, Functionality, Test cases, etc.
enabled or supported is determined by the device vendors
and through this coverage we can analyze the coverage of A. Specification Coverage:
these features. This coverage is purely dependent on the The Specification Coverage structures are:
device manual and on the device. The validation engineeer i. ATA Commands
should ensure that functional coverage should be achieved ii. Transfer Lengths
maximum since this truly detemines the adherence of the iii. Corner LBA(Logical Block Addressing)
vendor devices to the stndard protocol. iv. Security Feature Set
v. General Purpose Feature Set
C. Command Coverage vi. Software Setting Preservation
Command Coverage depicts the number of command run vii. SMART (Self-Monitoring, Analysis, and
or tested on the device against the total number of Reporting Technology
commands supported by the device. This coverage type can viii. Resets
be utilized to determine the number of command the device ix. Simulation Tests
actually processed and the number of commands which x. Multiple Threads
failed to execute and can provide an accurate and more xi. NCQ Queue Depth
efficient test case and test scenarios. The total number of xii. NCQ Error Injection
command may differ in various protocols and hence the xiii. General Error
parameter to obtain this coverage has to be fixed after Achieving high specification coverage depends upon the
thourough understanding of the specification and the device extent to which the device vendor has followed the standard
under test. specifications and which features the vendors have
eliminated or coined them as obsolete set of features. The
D. Parameter Coverage device under test has to be completely tested against every
Parameter Coverage is calculated for commands having feature and then the calculation of the specific coverage is
parameters. Device commands are tested on the device done with respect to the features actually present against the
which may have arguments in this case parameters and these features which the vendor has claimed to be present in their
parameters can achieve various input comibantions and it device. Specification coverage was calculated by writing
becomes mandatory to check the behavior of the device number of scripts which emulated the scenarios which were
when each command is executed with all the possible input required to test the features supported in the device under
combinations to ensure that there are no holes in the test. Specification Coverage may vary from vendor to
verification environment. Parameter coverage works in vendor.
tandem with Command Coverage.
165
Fig.5 Coverage percentage with respect to SATA – Functional
Fig.5 illustrates how SATA Functional coverage is Fig. 7 BusXpert-Transaction View Shows trigger point (256 GB SSD)
calculated over the SATA Specification with respect to the
specification supported by the target connected device. The The Transactional view of BusXpert shows all the 5
X axis shows the SATA Specification and Y axis shows the commands how it is transferred from host (SATA Controller
percentage of specification supported by the device. HBA) to the target connected SATA Device. These were the
Functional Coverage can be increased by identifying the basic commands used for the generation of the command
boundary cases and error response cases. The accuracy of coverage for SATA Devices and other commands could be
the device behavior will be determined if we achieve high leveraged to increase the coverage.
functional coverage.
C. Command Coverage
Command coverage aims at running test cases which will
cover all the 140 SATA commands along with their
respective sub-commands, Interface commands, NCQ
Commands, etc. If the test case on commands increasing
more and more, the coverage will increase and it’s
maximum when all 140 commands executed. 0
Parameters Set (Fig. 6, Fig. 7, and Fig. 8):
Device Settings Stop Mode : Trigger
Trigger Position: 30%
Buffer Size : 100 %( 2.25 GB)
Speed - Up to 3.0 Gbps
Trigger - IDFY, Read Sector, Write Sector, Check Fig. 8 MSDV window Shows Coverage for 5 SATA Commands
Power mode, Read FPDMA (first part DMA) Queued and
Enable Smart In this section , MSDV window shows the percentage of
command coverage achieved for 5 SATA Commands.
Commands used for the above coverage generation were
IDFY (Identify Commands), Read sector Command, Write
Sector Command, Power Management command (Check
Power Mode). Command Coverage was calculated by
executing the 140 SATA commands and their sub-command
which varied to the Features and Length field. These
commands are mentioned in the standard specification.
D. Parameter Coverage
Parameter coverage is calculated with respect to the
commands having parameters, supported by the SATA
device. For example, Read multiple command have
Fig. 6 BusXpert-Spreadsheet \View Shows Commands trigger point parameters such as LBA, sector number.
e.g. READ_MULTIPLE 10, 10
The Spreadsheet views of BusXpert shows the commands
LBA count varies from 0 to 250045779 and sector count
which were used for the generation of the command
varies from 0 to 65535 for 48 bit 128 GB Device. Parameter
coverage for SATA Devices.
coverage defines the coverage for all the LBA ranges
166
Bus Analyser Trace Window showing the Trigger point and
Views is displayed in Fig. 9, Fig. 10 and Fig. 11. The
Capture window shows all the setup of BusXpert and also it
shows the triggering point.
BusXpert Settings:
Device Settings Stop Mode : Trigger
Trigger Position : 30%
Buffer Size : 100 %( 2.25 GB)
Speed - Up to 3.0 Gbps
Trigger - Read Multiple, Write Multiple,
Set Feature Command, TRIM Command,
Read DMAEXT Command
Fig. 11 MSDV Tool GUI Shows Coverage for 5 SATA Commands with
their parameters for parameters coverage
167
SAS Command Coverage aims at identifying and VIII. HARDWARE REQUIREMENT
calculating how much the SAS commands were covered Below is the system configuration requirement for SSD
with respect to the Validation Environment. The SAS testing and validation.
Commands are broadly classified as:
1. Multimedia commands TABLE 1 HARDWARE REQUIREMENTS FOR DEVICE LEVEL
2. SCSI Block Commands VALIDATION SET UP
PROPERTY REQUIRED USED
3. SCSI Controller Commands
4. SCSI Media Commands Operating system Windows 8, Windows 7, Windows 7
Windows Vista , Windows Enterprise
5. SCSI Primary Commands XP, Windows
6. SCSI Stream Commands Server 2008 or Windows
Server 2003 Operating
System.
168