See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/273393397

The cache and memory subsystems of the IBM POWER8 processor

Article  in  Ibm Journal of Research and Development · January 2015

DOI: 10.1147/JRD.2014.2376131

CITATIONS READS

34 3,309

10 authors, including:

William J. Starke David Daly

IBM IBM
37 PUBLICATIONS   659 CITATIONS    43 PUBLICATIONS   804 CITATIONS   

SEE PROFILE SEE PROFILE

B. Blaner
IBM
22 PUBLICATIONS   364 CITATIONS   

SEE PROFILE

All content following this page was uploaded by B. Blaner on 01 April 2015.

The user has requested enhancement of the downloaded file.

 The cache and memory W. J. Starke
J. Stuecheli
subsystems of the IBM D. M. Daly
POWER8 processor J. S. Dodson
F. Auernhammer
P. M. Sagmeister
In this paper, we describe the IBM POWER8i cache, interconnect, G. L. Guthrie
memory, and input/output subsystems, collectively referred to as the C. F. Marino
Bnest.[ This paper focuses on the enhancements made to the nest to
M. Siegel
achieve balanced and scalable designs, ranging from small 12-core
single-socket systems, up to large 16-processor-socket, 192-core B. Blaner
enterprise rack servers. A key aspect of the design has been
increasing the end-to-end data and coherence bandwidth of the
system, now featuring more than twice the bandwidth of the
POWER7A processor. The paper describes the new memory-buffer
chip, called Centaur, providing up to 128 MB of eDRAM (embedded
dynamic random-access memory) buffer cache per processor,
along with an improved DRAM (dynamic random-access memory)
scheduler with support for prefetch and write optimizations,
providing industry-leading memory bandwidth combined with
low memory latency. It also describes new coherence-transport
enhancements and the transition to directly integrated PCIeA
(PCI ExpressA) support, as well as additions to the cache subsystem
to support higher levels of virtualization and scalability including
snoop filtering and cache sharing.

Introduction and the POWER8 processor builds upon that history [2].
The IBM POWER8* processor, shown in Figure 1, is IBM’s The POWER8 design provides industry-leading memory
latest generation of POWER* processors, boasting significant bandwidth and capacity, allowing the cores to run at
increases in thread, core, and system performance, scaling full speed, while minimizing average memory latency
to large core-count SMPs (symmetric multiprocessors) in and power consumption.
order to support big data analytics, cognitive computing, This paper describes the architectural enhancements
and transaction processing. These workloads require large made to the POWER8 cache, interconnect, memory, and
memory capacities with high bandwidth, accessed with low input/output subsystems, collectively referred to as the
latency, and optimally supporting locking and shared data. Bnest[ (shown in Figure 2), to achieve a balanced and
The performance of the systems, from 12-core single-socket scalable design. A major focus of the design has been
systems, up to the largest 192-core 16-processor-socket increasing the end-to-end data and coherence bandwidth of
SMPs depend on caches, coherence, data interconnects, the system (more than twice that of the POWER7 processor).
memory subsystems, and I/O subsystems to provide The paper describes the new memory-buffer chip, called
the cores with all the data they demand. Additionally, Centaur, with up to 128 MB of eDRAM buffer cache
the POWER8 core has roughly doubled in performance per processor along with an improved DRAM scheduler
compared to the IBM POWER7* core, with roughly with support for prefetch and write optimizations,
1.5 times the single-thread performance, and supporting up to providing industry-leading memory bandwidth combined
8-concurrent threads (SMT8) [1]. The POWER processor with low memory latency. It also describes new
family has consistently excelled at large SMP systems, coherence-transport enhancements to optimize the
performance on shared data, and integrated PCI Express**
(PCIe**) support to enable higher bandwidth, lower latency
Digital Object Identifier: 10.1147/JRD.2014.2376131 I/O (input output) operations.

ÓCopyright 2015 by International Business Machines Corporation. Copying in printed form for private use is permitted without payment of royalty provided that (1) each reproduction is done without
alteration and (2) the Journal reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this paper may be copied by any means or distributed
royalty free without further permission by computer-based and other information-service systems. Permission to republish any other portion of this paper must be obtained from the Editor.

0018-8646/15 B 2015 IBM

IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015 W. J. STARKE ET AL. 3:1

Figure 1
Annotated die photo of the POWER8 chip showing 12 processor cores
each with local L2 and L3 cache, on-chip interconnect, memory
controllers, onboard PCIe, and off-chip SMP interconnects.
The rest of this paper is organized as followed. The BCache
hierarchy[ section describes the L2 (level 2) and L3 (level 3)
caches. The BMemory subsystem[ section describes the
memory subsystem, including the new Centaur chip with L4
(level 4) cache and technology-agnostic memory channel.
The BI/O subsystem[ section describes the I/O subsystem,
while the BSMP interconnect[ and BOn-chip accelerator[
sections describe the processor data and coherence
interconnects with coherence filtering and the nest
accelerators available in the POWER8 processor, before
providing a brief conclusion.
Cache hierarchy
The cache hierarchy for the POWER8 processor builds
on the basic organization created for the POWER7
architecture [3]. The cache hierarchy local to a processing
core comprises a store-through L1 (level 1) data cache [1]
and a relatively small but low-latency and high-bandwidth,
private, store-in L2 (level 2) cache built from SRAM
(static random access memory). Additionally, the
POWER8 processor chip has up to 96 MB of a large,
dense, shared L3 (level 3) cache, comprised of 8 MB
cache regions built from eDRAM [4] and local to
the processor cores. The L3 cache-management algorithm
selectively migrates data to the local L3 region attached
to the core that is using the data, even cloning heavily
used shared data copies as needed and collapsing them
as they fall out of use. The same 13-state coherence
protocol developed for the POWER6* architecture [5]
and the POWER7 architecture is utilized for the
POWER8 architecture.
3:2 W. J. STARKE ET AL.
While similar in form, the POWER8 processor’s cache
hierarchy has been significantly enhanced to accommodate
the computational strength of the core; the core has roughly
doubled in performance from the POWER7 processor core
for most workloads [6], supports twice as many hardware
threads, and increases the L1 data cache capacity from
32 KB to 64 KB [1]. As shown in Table 1, the L2 capacity
per core has grown from 256 KB to 512 KB, while the
L3 capacity per core has increased from 4 MB to 8 MB.
The number of cores has grown from 8 to 12, and the
aggregate L3 capacity per chip was extended from 32 MB
to 96 MB. The latencies to each level of cache remain
roughly the same as in the POWER7 processor [3]. Finally,
the POWER8 processor supports up to 128 MB of shared
L4 (level 4) cache per processor, included in the new
Centaur chip. The L4 memory cache will be detailed in
the BMemory Subsystem[ section of this paper.
Designed for big data workloads, most data paths
throughout the L1 data cache and core-execution pipelines
are twice as wide (providing twice the data width per
processor clock) as those found in the POWER7 processor
[1]. As described in [1], this double-wide dataflow extends
through the L2 load and store paths, L2 cache arrays,
local L3 region read/write paths and cache arrays, and
seamlessly through the on-chip interconnect to the
memory read/write interfaces.
More in-flight requests must be tracked to manage the
increased traffic flow enabled by the double-wide dataflow.
Table 1 depicts the growth from the POWER7 processor
to the POWER8 processor for the major classes of L2
and L3 cache resources to manage the increased flow.
The L2 core store-gather cache consolidates store-thru
traffic from the core to optimize updates to the L2 cache.
L2 core read/write state machines manage coherence
negotiations with the system and L2 cache reads
and writes. L2 cache reads and writes can originate
from consolidated core-store traffic, core data loads,
instruction fetches, and address-translation fetches.
L2 castout state machines manage capacity related
migration of L2 data to the L3 cache or memory, while
L2 system-read/write state machines manage coherence
and data requests from the rest of the system. L3
core-read state machines handle core-fetch traffic for
L3 hits, while L3 write state machines manage L2
castouts (data evicted from L2), lateral L3-region castouts
(data evicted from other L3 regions to another region),
and cache injection (data installed directly into the L3
from a remote agent). L3 prefetch-write state machines
stage prefetch data from memory to the L3 cache. L3
castout and L3 system-read/write state machines perform
the same functions as their L2 counterparts.
In addition to the fundamental scaling improvements
afforded by increased bandwidth, the POWER8 processor
has significant hardware-based improvements to enhance
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
 Figure 2
POWER8 Nest is composed of up to 12 cores with local caches, connected through an on-chip coherence and data interconnect to off-chip interconnect,
accelerators, PCIe host controllers, and memory controllers. The accelerators include a true random number generator (RNG), cryptography accelerator
(Crypto), CAPP (coherent attached processor proxy), and compression accelerator (CMP).

Table 1 Comparison of POWER7 and POWER8 cache hierarchy capacities, resources, and bandwidth.

IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015 W. J. STARKE ET AL. 3:3
 Figure 3
The POWER8 memory subsystem includes a high-speed memory-technology-agnostic memory channel connected to the new Centaur chip, which
includes an eDRAM cache and 4 DDR3/DDR4 DRAM ports.
uncontested-lock performance (time to access a lock
that is free), highly-contested-lock scaling (handling
multiple threads competing for the same lock),
and address translation management scaling. For example,
uncontested-lock performance benefits from two changes.
Atomic operations are able to make binding coherence
commit decisions in the L1 data cache instead of the
L2 cache, reducing commit latency for uncontested locks.
Additionally, hardware transactional memory has been
added to the POWER8 processor enabling significant
software-scaling capabilities with software lock elision
for software that previously would have used locking [7],
further reducing uncontested-lock latency.
Beyond these foundational protocol improvements to
single-thread and system-wide scaling for shared data and
volatile-translation contexts, several hardware capabilities
have been added to the cache hierarchy to directly enable
high-value system software, middleware, and application
features, as described in [8]. Such features include
micro-partition prefetch and support for multiple
concurrent partitions on a core, both of which enable higher
performance for many smaller partitions, as well as the
hardware-managed reference-history array to improve
virtual-memory management.
3:4 W. J. STARKE ET AL.
Memory subsystem
The memory subsystem has been a major area of our focus
for the POWER8 processor, resulting in a substantial increase
in bandwidth (roughly 230 GBs/socket in the POWER8
processor compared to roughly 100 GB/s for the POWER7
processor) while reducing the latency to memory from over
100 ns in the POWER7 processor to around 80 ns in the
POWER8 processor.
The POWER8 processor chip has 8 memory controllers
(MCs) that operate synchronously with the data and
coherence interconnects. Figure 3 shows the memory
subsystem for one MC. Each memory controller can track
up to 32 requests at a time. Memory is interleaved at a
cache-line granularity of 128 B across memory controllers
to support low-latency operation. In contrast to the POWER7
processor, which spread a cache line over two memory
channels, every POWER8 memory channel supplies a full
128 B cache line of data.
Memory-technology-agnostic memory channel
The POWER7 and POWER8 processors both use a
high-speed memory channel and memory-buffer chip to
increase both the memory bandwidth and memory capacity
above what could be supported by a direct-attached DRAM
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
configuration. The POWER8 Centaur chip itself is
manufactured in the same 22 nm SOI (silicon on insulator)
process as the main processor chip, using the same design
rules, enabling a high-performance solution. Each memory
controller can connect to a memory-buffer chip (Centaur
chip) over a high-speed, differential, memory channel
providing 2 B wide read and 1 B wide write width with a
bit rate of 9.6 Gb/s. This results in a peak efficiency of over
28 GB/s of usable memory bandwidth per memory channel.
The 8 high-speed memory channels provide bandwidth
equivalent to 32 direct-attached DRAM channels. The
Centaur chip will be included directly on the custom DIMM
(dual-inline memory module) for some systems, and can
be soldered onto the backplane or a riser card for use with
industry-standard DIMMs (ISDIMMs) for other systems.
The Centaur chip also enables higher efficiency on the
memory channel, greatly increasing the total usable bandwidth
compared to the POWER7 design. The memory-channel
efficiency increase is due to architectural improvements.
The POWER7 processor had the memory scheduler on the
processor chip and the scheduler was DDR3 specific. The
POWER7 buffer chip merely received requests from the
processor chip, transmitting them to the appropriate DRAM
channel, and returning data to the processor chip when it
came back from the DRAM. The memory controller on the
processor chip was responsible for scheduling all operations,
and ensuring that there were no collisions on the channel
back to the processor chip. With the scheduling tightly
controlled from the processor chip, the efficiency of the
memory channel to the buffer chip was tied to the efficiency
of the DRAM channel itself. In the POWER8 memory
subsystem, the memory channel is agnostic with respect to the
memory technology used by the buffer chip. Requests sent to
the Centaur chip are high-level commands (e.g., cache-line
read, cache-line write) and include a tag identifying the
request. The high-level commands contrast with the low-level
DDR commands sent across the POWER7 memory channel.
The Centaur chip processes each request, and sends back
data as quickly as possible with the appropriate tag, possibly
sending data back in a different order than the requests arrived.
Requests might be reordered because of an L4 hit passing
an earlier request. Additionally, the Centaur chip can reorder
requests to the DRAM for optimal efficiency. The scheduling
flexibility and L4 cache allow us to drive the memory
channel to the processor chip at an efficiency of over 90%,
representing a significant performance improvement over
previous designs.
The memory-technology-agnostic memory channel has
another benefit: it enables possible upgrades to new memory
technologies without requiring a new processor chip. For
instance, we could develop a new variant of the Centaur chip
to support novel memory technologies (e.g., phase-change
memory, STT-MRAM (spin-transfer torque magnetoresistive
random-access memory), flash) or a newer DDR specification
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
by changing the scheduler logic. New custom DIMMs
with this new buffer chip could be used seamlessly in
POWER systems built with POWER8 processors.
L4 buffer cache
The memory-technology-agnostic memory channel also
enables another major change in the Centaur chip: the
inclusion of an L4 buffer cache per Centaur providing
up to 128 MB of L4 buffer cache per processor chip. The
cache is organized as a 16-way set-associative cache, with the
data stored in eDRAM, and the directory in SRAM. The
cache serves as a buffer for the memory and is only accessed
on memory-system accesses. As such, it is not snooped
on every bus request like the processor caches, and does
not participate in the coherence protocol. The L4 cache
provides several memory improvements: lower average
memory-read latency, lower write latency, more efficient
memory scheduling, lower DIMM power, and prefetch
extensions. The latency and power impacts are the most
obvious of the improvements. Read requests that are satisfied
by the L4 cache merely pay the cost of the eDRAM access
(latency and power), and do not have to perform a DRAM
access. In general an L4 hit reduces the latency of an L3
miss by over 35 ns and requires less energy to complete.
The other features enabled by the L4 cache are also
significant. The first feature is reduced write latency. All
writes received by the Centaur chip are installed into the
L4 cache, whether they hit the cache or not. This allows the
write to be retired quickly, freeing the memory-controller
resources associated with the write for use for the next
command. Additionally, the cache also enables much more
efficient scheduling of writes to the DRAM. The Virtual Write
Queue [9] introduced the idea of using the last-level cache
as a large write queue. We use the L4 cache for the same
purpose. We have added machinery, called the cache cleaner,
to track how many dirty lines are in the L4 cache, and to
scan the cache for lines to write back to memory. The cache
cleaner attempts to keep the DRAM write queue mostly
full and to schedule bursts of writes to a page on each rank
when writes are scheduled. This active process of scanning
the cache for page-mode writes enables more efficient use of
the DRAM data bus, as multiple writes to the same DRAM
page do not hit any DRAM-scheduling constraints, and
additionally, the page-mode writes save energy as the page
only needs to be activated once. In previous designs, we
would let writes accumulate in the DRAM write queue, and
perform a burst of writes when either the read queue was
empty or if the write queue became too full. In the current
design, we try to keep the write queue mostly full. Instead
of forcing writes when the write queue is mostly full, the
cache cleaner monitors the number of dirty lines in the cache,
and switches to a burst of writes when the number of dirty
lines in the cache exceeds a threshold. In this way, we allow
reads to pass writes and perform efficient bursts of page mode
W. J. STARKE ET AL. 3:5
writes when necessary to drain writes from the system,
reducing the likelihood of writes delaying critical reads.
Extended prefetching is also enabled by the L4 cache.
As prefetch requests are sent from the cores, extra information
in the form of hints is included with the prefetch requests.
The extra information indicates if the prefetch is part of a
stream of prefetches, and the confidence associated with the
prefetch stream. For high-confidence streaming prefetches,
the Centaur chip can prefetch extra data into the L4 cache
before it is requested by the core, reducing latency of future
prefetch requests in that stream. The extra prefetching has
two forms. The first involves fetching two 128 B lines of data
for one request. The second line of data is installed into the
L4 cache for later use. Additionally, the two lines can be
fetched together from the DRAM, using an open-page
access for the second line. The open-page access increases
efficiency and lowers DRAM-power usage. The second
form involves fetching the next N (e.g., 4) pairs of lines
in the stream into the L4 cache. This is done for the
highest-confidence streams, and can essentially transform
all of the prefetch requests into L4 cache hits.

DRAM interface
Each Centaur chip has four 9 or 10 B (9 B with up to
1 B of spare data) DRAM ports, supporting the DDR3
(double-data rate) [10] and DDR4 [11] JEDEC standards,
with the DRAM scheduler included on the Centaur chip.
Each port can address up to 16 logically independent ranks
of DRAM, in order to support multiple physical ranks of
DRAM as well as stacked DRAM. A rank of DRAM is a
collection of DRAM chips (e.g., 8) that are accessed together
in lockstep to provide a cache line of data. Additionally,
the Centaur chip contains an eDRAM buffer cache, providing
up to 128 MB of aggregate L4 memory buffer to the
processor chip, enabling lower latency and improved
scheduling opportunities. The DRAM ports are accessed
in pairs to provide data, fetching a full 128 B cache line
from a port pair. The die photo in Figure 4 shows the
four DRAM ports arranged around the outside of the
Centaur chip, with the high-speed channel to the processor
chip taking up the remaining perimeter on the right. The
eDRAM cache and the related control structures fill the
center of the chip, along with the DRAM scheduling logic.
DRAM scheduling must satisfy a number of scheduling
constraints, including the need for regular refresh operations.
While a refresh operation is occurring, no read or write
commands can be issued to the rank in refresh. In traditional
systems, this can have a noticeable performance impact. We
have included a number of features to mitigate the refresh
impact. Before the memory controller issues a refresh to
a rank, it does a number of things. First it checks the
high-confidence prefetch streams that are being fetched into
the L4. If any of the streams will be impacted by the upcoming
refresh, the stream is allowed to prefetch further ahead
Figure 4
Annotated die photo of the Centaur buffer chip with high-speed link to
the processor chip, eDRAM memory buffer cache, DDR interfaces, and
control logic.
than normal, in order to fetch lines from the soon to be
refreshed rank. After the prefetcher is allowed to prefetch
ahead for a period of time, no new requests to that rank are
allowed into the read queue (from the prefetcher or from
L4 misses). The reads that remain in the read queue are
drained from the queue with high priority, and the refresh
is performed after the reads are drained. Extended prefetching
before performing refresh allows the stream to continue for
longer before being disrupted by the refresh. Draining the read
queue has a different effect: It keeps the read queue from
becoming filled with reads that cannot be serviced at the
expense of reads that can be serviced.
Memory RAS
POWER7 systems had industry leading RAS (reliability,
availability, and serviceability) properties, using a 72 B
marking code to support chipkill (failure of a complete
DRAM chip) and sparing. For POWER7 the full 128 B access
used two memory channels, and the ECC (error correcting
code) correction was performed on the processor chip. In the
POWER8 processor using the memory-technology-agnostic
memory channel, the full 128 B cache line comes from one
memory channel, allowing the ECC check and correction

3:6 W. J. STARKE ET AL. IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
to be performed on the buffer chip. The Centaur chip supports
10 B wide DRAM ports, consisting of 72 b of data and
ECC plus 8 b of spare data. The data is arranged into a 72 B
code word for 64 B of data. The DRAM is accessed in a burst
length of 8 beats, with the first four beats of data from the
two DRAM ports forming one code word, and the second
four beats of data forming a second code word. The spare data
can be multiplexed in and used in place of a failed chip
(chipkill) discovered by the ECC. The ECC is able to detect
and correct a complete DRAM chip failure. When a DRAM
chip is determined to be faulty, a mark is placed and the
chip is spared out. The spare data is steered into the place
of the failed chip and, for 4 bit wide DRAM, supports steering
at 4 bit wide granularity. In this way, the ECC is capable
of handling three chipkill events.
In addition to protecting against failures in the DRAM
itself, the rest of the path to memory is also protected.
The high-speed memory channel to the processor chip
is protected by a CRC (cyclic redundancy check) code with
a retry protocol. Transmissions over the memory channel
are buffered until confirmed, and any transmission that
encounters an error is resent removing the error from the
system, and protecting the memory channel with the same
level of protection as the rest of the system.
Additionally, all the primary data paths through the
chip are protected with a SECDED (single-error correcting,
dual-error detecting) ECC code, while the eDRAM cache
is protected by an ECC code with a fuse-controlled
repair facility.
The POWER8 processor supports an additional RAS
feature called Selective Memory Mirroring (SMM),
originally introduced in the POWER7 processor [3].
SMM enables mirroring of critical data regions such as
hypervisor data. Protected data is mirrored across two
separate memory channels, protecting against a complete
memory channel failure.
I/O subsystem
The I/O subsystem of the POWER8 processor saw a
complete refresh compared to its predecessor designs.
This refresh increased bandwidth, significantly lowered
memory latency, lowered power consumption, and enables
easy adoption by third parties, especially for those that
want to use the coherent coupling features using the
Coherent Accelerator Processor Interface (CAPI) protocol
(see the BOn-chip accelerator[ section).
Starting with the POWER4 processor [12] and z10* [13]
processors, the IBM I/O subsystem was built upon a
proprietary ecosystem of I/O chips [14] attached to IBM’s
proprietary GX bus and later GX+ and GX++ buses to
achieve higher I/O throughput than possible with the
standardized interfaces available at the time. Different
break-out I/O hub chips provided connectivity to PCI-X**,
PCIe devices, and InfiniBand** or Ethernet fabrics.
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
The POWER8 processor is the first generation of POWER
processors that features integrated, industry-standard, PCIe
Gen 3.0 [15] host controllers. PCIe was chosen because
of its maturity, industry-leading bandwidth, and widespread
adoption by third party vendors. The POWER8 processor
chip integrates 3 PCIe host bridges (PHBs) per chip, with a
maximum of 16 lanes per PHB, resulting in a maximum
bidirectional bandwidth of 32 GB/s per PHB. The 12-core
POWER8 processor chip provides a total of 32 PCI Express
lanes. A single POWER8 processor chip thus offers a
maximum bidirectional bandwidth of 64 GB/s.
The move towards an integrated PCIe-based I/O
subsystem allows for more flexibility in adapting to specific
application needs. The POWER7 I/O subsystem [16] was
primarily built around external PCIe I/O hub chips that
split the GX bus bandwidth of up to 20 GB/s into multiple,
but slower x8 PCIe 2.0 [17] buses. In contrast, the POWER8
processor chip now offers the increased bandwidth of
32 GB/s per controller, but also the flexibility to either
couple PCIe devices with very high-bandwidth requirements
directly to the processor chip or to split the bandwidth among
multiple slower devices using external PCIe switches.
Another advantage of moving the PHB onto the processor
chip is the significant reduction in memory-access latency,
providing very low DMA (Direct Memory Access) read
roundtrip times that are crucial for high bandwidth and
low latencies with high-performance network adapter
devices. The on-chip PHBs achieve time-to-first-byte
latencies for DMA reads as low as 200 ns, which is less
than one third of the time-to-first-byte latency feasible
with the POWER7 GX-attached PCIe hub chip.
I/O memory management unit
The I/O memory management unit (IOMMU) provides
enhanced protection and reliability, availability, and
serviceability (RAS) features. The protection mechanisms
allow freezing all traffic to or from a device in an isolation
domain on an error condition, without affecting other
domains. The protection mechanisms also enable recovery
without having to reset the PHB. Therefore multiple devices
can be attached to the same PHB securely without sacrifice
in device reliability and availability.
The IOMMU resided on the GX hub chip in POWER
systems built using the POWER7 processor, and has
been moved onto the POWER8 processor chip. A number
of enhancements were made to the IOMMU to meet the
increased I/O requirements of the POWER8 processor,
and to further exploit the inclusion of the unit in the
processor chip. The IOMMU’s virtualization and protection
capabilities were increased to meet the higher fan out
requirements of the POWER8 processor. Each POWER8
PHB is capable of handling 256 error isolation domains,
which allow devices to continue operation even when
another device sharing the same PHB encounters a failure
W. J. STARKE ET AL. 3:7
and has to be reinitialized. Moreover every POWER8 PHB
can manage 512 virtual address spaces in order to provide
enough resources to fully support and isolate single root
I/O virtualized (SRIOV) [18] PCIe devices.
The address-translation structures in main memory used by
the IOMMU were completely rearranged and optimized to
take full advantage of the new location of the IOMMU, being
now on-chip and thus able to profit of core-like low-latency
memory access. The new structures thus provide the optimal
balance between the number of lookups needed, translation
latency, and area spent on cache or array structures in each
PHB. Also, the IOMMU was extended to support multi-level
translation tables in addition to the traditional single-level
translation scheme used in AIX, in order to better support the
Linux** memory management schemes.
Throughput optimization
The design was optimized to fully utilize the available
link bandwidth, achieving close to the theoretical limits
with more than 95% raw link utilization and roughly 90%
data utilization, even with traffic using the full address
translation and protection capabilities of the IOMMU.
The high sustained link speeds are made possible by
a combination of data and coherence interconnect and
micro-architectural enhancements.
An improved streaming mode exclusively used for I/O
data is implemented in the data interconnect, based on the
streaming mode introduced in the POWER7 processor [3].
It allows data bandwidths of more than 14 GB/s in each
direction for reads and writes, while still maintaining
the strict ordering requirements imposed by the PCIe
Specification. Frequently achieving high bandwidths on
PCIe buses also requires the use of relaxed ordering to reduce
dependencies between reads and writes wherever possible.
The POWER8 processor does not require these ordering
relaxations due to the high bandwidth provided by the
streaming mode. This is critical as allowing relaxed
ordering would create a potential threat to the error
isolation concept of the IOMMU.
In addition, various micro-architectural optimizations
were made within the PHB to enable higher link efficiency.
Address translation prefetching allows early inspection of
PCIe addresses close to the link interface and early fetches
of translation control entries (TCE) [19] in order to hide
most of the lookup latency behind stack queuing and
packet processing. In combination with the low latency
to memory, this allows for more efficient receive buffer
use, and achieves four times higher bandwidth as the prior
generation external I/O hub chip, even with the same-sized
link receive buffers. The PHB further applies prioritization
mechanisms in many areas of the design to optimize the
scheduling and arbitration especially of reads for control
and payload data. It also relaxes ordering requirements
wherever possible by splitting up streams from the
3:8 W. J. STARKE ET AL.
different error isolation domains to prevent stalls, but
without compromising on domain isolation capabilities.
Beside the basic PCIe standard functionality, the PHB
now also supports the use of Translation Layer Packet (TLP)
hint bits on the PCIe link to control the cache injection
mechanism on the coherent interconnect.
SMP interconnect
POWER systems are symmetric multiprocessor (SMP)
systems, providing snooping based cache coherence across
all the cores in the system. On the POWER8 processor
chip, the SMP data and coherence interconnects consist of
on-chip and off-chip interconnects, plus memory-coherence
additions to enable efficient scaling up to 192 cores. The
SMP interconnect is responsible for transferring the current
value of memory locations to and from the processor cores
as well as implementing the coherence protocol and enabling
efficient lock handling.
On-chip SMP interconnect
As shown in Figure 2, the POWER8 processor on-chip
interconnect consists of a multi-ported coherency interface
and a highly distributed data interconnect between the
processor cores each with associated L2 and L3 caches,
the memory subsystem, and the I/O subsystem. As in the
POWER7 processor chip, the on-chip data connect consists
of eight 16 B buses that span the chip horizontally, and
are broken into multiple segments to handle the propagation
delay across the chip, and allows the interconnect to be
pipelined. Four of the buses flow left to right, and four
flow right to left, and the buses operate at up to 2.4 GHz.
Micro-architecture improvements were made to the internal
data interconnect to reduce request latency by resolving
on-chip data-routing decisions using a distributed-arbitration
scheme instead of a centralized data arbiter in previous
generations. The on-chip interconnect also contains an
adaptive-control mechanism to support independently
controlled core frequencies while optimizing
coherence-traffic bandwidth. As core frequencies are adjusted
upward or downward, the coherence-arbitration logic will
throttle up or throttle down the rate at which commands
are issued based on the frequency of the slowest core.
Off-chip SMP interconnect
The off-chip interconnect, shown in Figure 5, is a highly
scalable multi-tiered fully-connected topology redesigned for
the POWER8 processor chip to reduce latency. POWER
systems built on the POWER7 processor required up to
3 hops between chips to get from one chip to another, but
POWER systems built on the POWER8 processor require no
more than 2 hops. By eliminating a chip hop, the POWER
system has been flattened, significantly reducing the latency
between the furthest ends of the SMP (an approximately
25 ns reduction). The first-level system is a single chip.
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015

Figure 5
POWER8 SMP topology consists of up to 4 POWER8 processors
connected in an all-to-all manner to form a 4-chip group, and up to 4
groups with each chip in a group connected to the matching chips in the
other groups.
The second-level system is fully connected with 10 B
single-ended SMP links running at 4.8 Gbps. The POWER8
processor chip has three such links, enabling direct
connection to each of 4 other processor chips, in order to
create a four-chip group. The third-level system connects
each processor chip in a group to its corresponding processor
chip in each other group. Three of the inter-group links are
provided per chip supporting a total of four groups, each
containing four processor chips. A full four-group system of
four chips per group comprises a maximum system of 16
processor chips and a 192-way SMP. The inter-group level
SMP link uses a 22 bit high-speed differential bus running
at 6.4 Gbps.
The reliability, availability, and serviceability (RAS),
and the cost of the SMP links have also been improved in
the POWER8 processor. The SMP links have the ability
to dynamically detect and repair bit lanes without
removing functional lanes from operation. In other words,
SMP-coherence traffic continues to function normally
while a bad lane is repaired. Another major change for
the POWER8 processor was to replace the SMP backplane
with a cabled SMP link connection, significantly reducing
the system cost. System configuration is discussed in
more detail in [2].
Coherence scopes
The data and coherence interconnects described above
provide a large amount of data and command bandwidth.
However, with up to 192 cores and a snooping
memory-coherence protocol, the cores could easily saturate
the links with coherence traffic. The POWER7 architecture
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
had two coherence scopes to limit the coherence traffic
on the SMP links by filtering the traffic that needs to be
broadcast to other chips and groups [3]. The POWER8
architecture extends the POWER7 architecture’s coherence
scopes, adding a third scope.
Coherence filtering
Commands are issued to the coherence interconnect with one
of three scopes specified: chip, group, and system, where the
scopes match the tiers described above. Commands issued
with chip or group scope are not broadcast to the complete
system, but rather only to the local chip or local group
respectively. Since commands issued with chip or group
scope only access a portion of the system, they require
much less command bandwidth and much lower latency to
complete. However, the system must still ensure memory
coherence across the complete system, including the portions
not snooped. The command must be seen by any cache
holding the line. Hardware prediction mechanisms and
software collaboration allows the use of the smallest scope
to maintain coherence, while hardware checks ensure that
scope includes all relevant caches.
Memory-Domain Indicators (MDIs) are included for each
line of data in the system in order to determine if the scope
of the command is adequate to find all caches holding the
requested line.
MDI are assigned classes. Chip-class MDI indicate if the
line has moved off the line’s home chip where the Memory
Controller (MC) owning the line is attached and have been
included in POWER systems since the POWER6 processor
[5]. Group-class MDI are new in the POWER8 processor
and indicate if the line has moved off the line’s home group.
The new group class MDI are kept in a directory structure
called an MCD (memory-coherence directory), and included
on the processor chip. Each MCD is associated with the
full address range of the memory served by that processor
chip’s memory controllers. In the POWER8 processor chip,
the MCD is a coarse-grained directory with each MDI
Bgranule[ representing a 16 MB granule of memory.
The MCD snoops commands issued with group and system
scope. It participates in the address tenure of commands
issued with group scope to indicate if the line specified by
the command has moved off the home group (i.e., when
the MCD MDI is marked Bremote[). At system scope, the
MCD determines during the address tenure if the line is going
to be moved off the group. If the destination of the line is off
the group, the MCD sets the MDI for the line’s granule to
Bremote.[ When the MCD snoops a command at group scope
and indicates that the line is Bremote,[ the master may be
required to retry the operation using system scope, or may be
allowed to complete the operation and be directed to perform
a background kill operation using system scope.
Using both the chip and group MDI, the coherence
protocol may direct a requestor that initially issues a request
W. J. STARKE ET AL. 3:9
with chip scope, to increase its scope to group, and then
to system. Scope-prediction logic within the requestor
is used to determine the initial scope for the operation.
For example, if the line is in the local cache in the IG state,
the predictor can determine that the line was last sent off
the chip and that the request should have group or system
scope. A cache prefetching consecutive lines could use
the scope needed to complete the operation for one line
in the stream for subsequent lines in the stream.
The MC holds chip-class MDI for each 128 B cache
line it holds and can reset the MDI bit to Bhome[ when
a line is removed from all caches in the system. However,
each MCD group-class MDI entry holds the status for
a 16 MB granule, consisting of 128 K lines and requires
a recovery mechanism to reset the MDI. A specialized
coherence-protocol command exists to determine if any
cache holds the line or holds a reservation for the line, and
this command is used to implement the recovery mechanism.
Data routing through topology
Data needs to be routed over the data interconnect to get to its
destination. On the larger SMP systems, there are multiple
routes using inter- and intra-group interconnect buses that
could be used to route the data. Routing normally follows
a preferred shortest route between source and destination,
but may use alternate routing in the case of congestion.
Optimistic non-blocking coherence flow
The POWER8 processor SMP interconnect utilizes a
non-blocking snooping protocol, which has highly scalable,
low-latency characteristics. This has an advantage over
directory-based coherence protocols by decentralizing
coherency and thus vastly reducing latency. The
non-blocking snooping protocol also has an advantage
over message-passing snooping coherence protocols because
it is temporally bounded, i.e. snoopers respond in a fixed
time called Tsnoop. Furthermore, message-passing snooping
protocols rely on queuing structures and communication
bandwidth. These become more constrained as the SMP
system scales to larger n-way systems.
In previous POWER server generations, coherence
operations were evenly divided using time-division
multiplexing in order to ensure that each processor chip’s
coherency bandwidth does not exceed a limit, when all
processor chips are issuing requests. However, limiting
the broadcast rate for worst-case conditions often leaves
significant coherence bandwidth unused. The POWER8 SMP
interconnect introduced the capability to oversubscribe
each processor chip’s allotment of bandwidth in order to
take advantage of this unused bandwidth, without impacting
the base non-blocking operation.
The optimistic coherence flow’s oversubscription uses
the existing POWER coherence-retry protocol. When a
coherence operation is received and the resource required to
3 : 10 W. J. STARKE ET AL.
resolve coherence is unavailable, the protocol allows for the
snooper to retry the operation. Oversubscription provides
the same retry capability but at a chip level scale. This chip
level retry can occur when the coherency operations received
from the incoming SMP links exceeds the chip’s snoop
bandwidth. This implies that the coherence operation was not
broadcast to any downstream processor chips in the SMP
topology. Using oversubscription, a coherence operation
can still complete successfully even if the broadcast did not
reach all chips in the SMP system, so long as the broadcast
was received by the appropriate subset for that request.
Not all requests are handled in the same manner when they
are retried. Speculative commands, such as prefetch, do not
incur as high a penalty if retried, while critical coherence
requests will suffer a much higher penalty. A priority was
added to requests to determine which requests to drop, when
a request must be dropped. The priority of a command can
be specified as low, medium, or high. Low-drop-priority
commands are the first commands to be dropped, while
high-priority commands are the last commands to be
dropped. Furthermore, the interconnect issues requests at
different rates depending on their priority. Low-drop-priority
commands have a higher issue rate than high-drop-priority
commands as they may be more freely dropped. Dropped
requests that are retried, are retried with higher priority.
The priorities and request issue rates allow the interconnect
to manage the number of coherence operations concurrently
in flight, allowing most high priority commands to
succeed regardless of SMP system coherency traffic
while speculative, low-priority commands utilize the
remaining coherence bandwidth, increasing overall
coherence-bandwidth efficiency.
The coherence interconnect issues requests at different rates
based on commands’ scope, priority, and current interconnect
usage. The rates are dynamically controlled in hardware
using feedback from dropped commands, in order to optimize
bandwidth use and minimize dropped requests.
On-chip accelerators
In POWER7+*, IBM introduced on-chip accelerators for
cryptography and active memory expansion (AME) [20]
and provided a true-hardware random-number generator
for cryptographic applications [21]. In POWER8, these
accelerators have been carried forward and improved,
adding new capabilities, such as providing data-prefetching
hints to the memory controller to reduce cache-line-access
latency and improve throughput.
POWER8 introduces the Coherent Accelerator Processor
Interface (CAPI), described in detail elsewhere in this issue
of the IBM Journal of Research and Development [22].
CAPI enables off-chip accelerators to be plugged into
an up to 16-lane PCIe slot and participate in the system
memory-coherence protocol as a peer of other caches
in the system at high bandwidth. Accelerators use effective
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
addresses to reference data structures in memory just like
applications running on the cores. Accelerator designers
are given the freedom to implement their PCIe-card based
designs using field-programmable gate arrays (FPGA)
application-specific integrated circuit (ASIC) chips,
semi-custom integrated circuits, and so forth.
Conclusion
In this paper, we have described the IBM POWER8
cache, interconnect, memory, and input/output subsystems
collectively referred to as the Bnest.[ Systems built from
the POWER8 processor and nest represent a substantial
increase in single thread and throughput computing.
The paper focused on the enhancements made to the nest
to achieve balanced and scalable designs, ranging from
small 12-core single-socket systems, up to large 16-socket,
192-core enterprise rack servers. The local-cache hierarchy
has been improved to accommodate the computational
strength of the core (which has roughly doubled in
performance), with twice the L2 and L3 cache capacity,
twice the bandwidth, more in flight commands, more efficient
locking and translation support, while maintaining similar
access latencies as the POWER7 cache. The memory
subsystem was a major area of focus for the POWER8
processor providing industry-leading bandwidth, while
simultaneously reducing the latency to memory significantly.
The memory subsystem uses the new Centaur chip, including
a new L4 memory-buffer cache and containing the DRAM
scheduler, enabling higher memory-channel efficiency.
The I/O subsystem of the POWER8 processor chip saw
a complete refresh compared to its predecessor designs,
bringing 32 lanes of the PCIe 3.0 onto the processor chip,
significantly increasing bandwidth while lowering DMA
memory latencies considerably. The data and coherence
interconnects were improved to increase end-to-end
bandwidth by increasing on-chip efficiency, increasing
intra-group and inter-group bandwidth, reducing
the maximum number of hops, and improving the
coherence-scope filtering. Finally, the on-chip accelerators
from POWER7 were extended, featuring the inclusion of
the new Coherent Accelerator Processor Interface (CAPI).
*Trademark, service mark, or registered trademark of International
Business Machines Corporation in the United States, other countries,
or both.
**Trademark, service mark, or registered trademark of PCI-SIG,
InfiniBand Trade Association, or Linus Torvalds, Inc., in the United
States, other countries, or both.
References
1. B. Sinharoy, J. A. Van Norstrand, R. J. Eickemeyer, H. Q. Le,
J. Leenstra, D. Q. Nguyen, B. Konigsburg, K. Ward, M. D. Brown,
J. E. Moreira, D. Levitan, S. Tung, D. Hrusecky, J. W. Bishop,
M. Gschwind, M. Boersma, M. Kroener, M. Kaltenbach,
T. Karkhanis, and K. M. Fernsler, BIBM POWER8 processor core
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
microarchitecture,[ IBM J. Res. & Dev., vol. 59, no. 1, paper 2,
pp. 2:1–2:21, 2015.
2. J. Cahill, T. Nguyen, M. Vega, D. Baska, D. Szerdi, H. Pross,
R. Arroyo, H. Nguyen, M. Mueller, D. Henderson, and J. Moreira,
BIBM Power Systems built with the POWER8 architecture and
processors,[ IBM J. Res. & Dev., vol. 59, no. 1, paper 4,
pp. 4:1–4:10, 2015.
3. B. Sinharoy, R. R. Kalla, W. J. Starke, H. Q. Le, R. Cargnoni,
J. A. Van Norstrand, B. Ronchetti, J. Stuecheli, J. Leenstra,
G. L. Guthrie, D. Nguyen, B. Blaner, C. F. Marino, E. Retter,
and P. Williams, BIBM POWER7 multicore server processor,[
IBM J. Res. & Dev., vol. 55, no. 3, paper 1, pp. 1:1–1:29,
May/Jun. 2011.
4. S. Narasimha, P. Chang, C. Ortolland, D. Fried, E. Engbrecht,
K. K. Nummy, P. Parries, T. T. Ando, M. Aquilino, N. Arnold,
R. Bolam, J. Cai, M. Chudzik, B. Cipriany, G. Costrini,
M. Dai, J. Dechene, C. Dewan, B. Engel, M. Gribelyuk,
D. Guo, G. Han, N. Habib, J. Holt, D. Ioannou, B. Jagannathan,
D. Jaeger, J. Johnson, W. Kong, J. Koshy, R. Krishnan,
A. Kumar, M. M. Kumar, J. Lee, X. Li, C. Lin, B. Linder,
S. Lucarini, N. Lustig, P. McLaughlin, K. Onishi, V. Ontalus,
R. Robison, C. Sheraw, M. Stoker, A. Thomas, G. Wang,
R. Wise, L. Zhuang, G. Freeman, J. Gill, E. Maciejewski,
R. Malik, J. Norum, and P. Agnello, B22 nm high-performance
SOI technology featuring dual-embedded stressors, Epi-Plate
High-K deep-trench embedded DRAM and self-aligned
via 15LM BEOL,[ in 2012 IEEE Electron. Devices Meeting
(IEDM), 2012 IEEE Int., Dec. 10–13, 2012, pp. 3.3.1–3.3.4.
5. H. Q. Le, W. J. Starke, J. S. Fields, F. P. O’Connell, D. Q. Nguyen,
B. J. Ronchetti, W. M. Sauer, E. M. Schwarz, and M. T. Vaden,
BIBM POWER6 microarchitecture,[ IBM J. Res. & Dev,
vol. 51, no. 6, pp. 639–662, Nov. 2007.
6. A. Mericas, N. Peleg, L. Pesantez, S. B. Purushotham, P. Oehler,
C. A. Anderson, B. A. King-Smith, M. Anand, B. Rogers,
L. Maurice1, and K. Vu, BIBM POWER8 performance features
and evaluation,[ IBM J. Res. & Dev., vol. 59, no. 1, paper 6,
pp. 6:1–6:10, 2015.
7. H. Q. Le, G. L. Guthrie, D. E. Williams, M. M. Michael,
B. G. Frey, W. J. Starke, C. May, R. Odaira, and T. Nakaike,
BTransactional memory support in the IBM POWER8
processor,[ IBM J. Res. & Dev., vol. 59, no. 1, paper 8,
pp. 8:1–8:14, 2015.
8. B. Sinharoy, R. Swanberg, N. Nayar, B. Mealey, J. Stuecheli,
B. Schiefer, J. Leenstra, J. Jann, P. Oehler, D. Levitan, S. Eisen,
D. Sanner, T. Pflueger, C. Lichtenau, W. E. Hall, and T. Block,
BAdvanced features in IBM POWER8 systems,[ IBM J. Res. &
Dev., vol. 59, no. 1, paper 1, pp. 1:1–1:18, 2015.
9. J. Stuecheli, D. Kaseridis, L. John, D. Daly, and H. C. Hunter,
BCoordinating DRAM and last-level-cache policies with the
virtual write queue,[ IEEE Micro, vol. 31, no. 1, pp. 90–98,
Jan./Feb. 2011.
10. JEDEC Solid State Technology Association, JEDEC Standard:
DDR3 SDRAM, 2010. [Online]. Available: www.jedec.org/
sites/default/files/docs/JESD79-3E.pdf
11. JEDEC Solid State Technology Association, JEDEC Standard:
DDR4 SDRAM, 2012. [Online]. Available: www.jedec.org/
sites/default/files/docs/JESD79-4.pdf
12. J. M. Tendler, J. S. Dodson, J. S. Fields, H. Le, and B. Sinharoy,
BPOWER4 system microarchitecture,[ IBM J. Res. & Dev.,
vol. 46, no. 1, pp. 5–25, Jan. 2002.
13. C.-L. K. Shum, F. Busaba, S. Dao-Trong, G. Gerwig, C. Jacobi,
T. Koehler, E. Pfeffer, B. R. Prasky, J. G. Rell, and A. Tsai,
BDesign and microarchitecture of the IBM System z10
microprocessor,[ IBM J. Res. & Dev., vol. 53, no. 1, pp. 1:1–1:12,
Jan. 2009.
14. E. W. Chencinski, M. A. Check, C. DeCusatis, H. H. Deng,
M. Grassi, T. A. Gregg, M. M. Helms, A. D. Koenig, L. Mohr,
K. Pandey, T. Schlipf, T. Schober, H. Ulrich, and C. R. Walters,
BIBM System z10 I/O subsystem,[ IBM J. Res. & Dev., vol. 53,
no. 1, pp. 6:1–6:13, Jan. 2009.
15. PCI Express Base Specification, Revision 3.0, Nov. 2010. [Online].
Available: http://www.pcisig.com/specifications/pciexpress/base3/
W. J. STARKE ET AL. 3 : 11
16. R. X. Arroyo, R. J. Harrington, S. P. Hartman, and T. Nguyen,
BIBM POWER7 systems,[ IBM J. Res. & Dev., vol. 55, no. 3,
pp. 2:1–2:13, May/Jun. 2011.
17. PCI Express Base Specification, Revision 2.1, Mar. 2009. [Online].
Available: http://www.pcisig.com/members/downloads/
specifications/pciexpress/PCI_Express_Base_r2_1_04Mar09.pdf
18. Single Root I/O Virtualization and Sharing Specification,
Revision 1.1, Jan. 2010. [Online]. Available: http://www.
pcisig.com/specifications/iov/single_root/
19. Logical Partition Security in the IBM eServer pSeries 690, IBM
white paper. [Online]. Available: http://www-1.ibm.com/servers/
eserver/pseries/hardware/whitepapers/lpar_security.html
20. B. Blaner, B. Abali, B. M. Bass, S. Chari, R. Kalla, S. Kunkel,
K. Lauricella, R. Leavens, J. J. Reilly, and P. A. Sandon,
BIBM POWER7+ processor on-chip accelerators for cryptography
and active memory expansion,[ IBM J. Res. & Dev, vol. 57, no. 6,
pp. 3:1–3:16, Nov./Dec. 2013.
21. J. S. Liberty, A. Barrera, D. W. Boerstler, T. B. Chadwick,
S. R. Cottier, H. P. Hofstee, J. A. Rosser, and M. L. Tsai,
BTrue hardware random number generation implemented in the
32-nm SOI POWER7+ processor,[ IBM J. Res. & Dev, vol. 57,
no. 6, pp. 4:1–4:7, Nov./Dec. 2013.
22. J. Stuecheli, B. Blaner, C. R. Johns, and M. S. Siegel, BCAPI:
A Coherent Accelerator Processor Interface,[ IBM J. Res. & Dev.,
vol. 59, no. 1, paper 7, pp. 7:1–7:7, 2015.
Received March 17, 2014; accepted for publication
April 12, 2014
William J. Starke IBM Systems and Technology Group, Austin,
TX 78758 USA (wstarke@us.ibm.com). Mr. Starke joined IBM in
1990 after graduating from Michigan Technological University with
a B.S. degree in computer science. He currently serves as the IBM
Distinguished Engineer and Chief Architect for the Power processor
storage hierarchy, and is responsible for shaping the processor cache
hierarchy, symmetric multi-processor (SMP) interconnect, cache
coherence, memory and I/O controllers, accelerators, and logical system
structures for Power systems. He leads a large engineering team that
spans multiple geographies. Mr. Starke has been employed by IBM
for almost 25 years in several roles, spanning mainframe and Power
systems performance analysis, logic design, and microarchitecture.
Over the past decade, he has served as the storage hierarchy Chief
Architect for POWER6, POWER7, POWER8, and follow-on
design points that are currently in development. Mr. Starke holds
approximately 200 U.S. patents.
Jeff Stuecheli IBM Systems and Technology Group, Austin,
TX 78758 USA (jeffas@us.ibm.com). Dr. Stuecheli is a Senior
Technical Staff Member in the Systems and Technology Group. He
works in the area of server hardware architecture. His most recent work
includes advanced memory architectures, cache coherence, and
accelerator design. He has contributed to the development of numerous
IBM products in the POWER architecture family, most recently the
POWER8 design. He has been appointed an IBM Master Inventor,
authoring about 100 patents. He received B.S., M.S., and Ph.D. degrees
from The University of Texas Austin in electrical engineering.
David M. Daly IBM Research Division, Thomas J. Watson
Research Center, Yorktown Heights, NY 10598 USA (ddaly@ieee.org).
Dr. Daly was a Research Staff Member at the IBM T. J. Watson
Research Center. He received a B.S. degree in computer engineering
from Syracuse University in 1998, and M.S. and Ph.D. degrees
in electrical engineering from the University of Illinois at
Urbana-Champaign in 2001 and 2005, respectively. He subsequently
joined IBM at the IBM T. J. Watson Research Center, where he worked
on next generation POWER microprocessor design, including the
POWER8 microprocessor, from 2008 to 2014. The main focus of his
work has been the design and performance analysis of the memory
subsystem using trace driven, queuing, and spreadsheet analysis.
3 : 12 W. J. STARKE ET AL.
Additional work has focused on system scalability, memory
coherence, and network interconnects, as well as system reliability
and availability. Dr. Daly is a senior member of the Institute of
Electrical and Electronics Engineers (IEEE).
J. Steve Dodson IBM Systems and Technology Group, Austin,
Texas 78758 USA (jsdodson@us.ibm.com). Mr. Dodson is a Senior
Engineer in the POWER microprocessor development organization
at IBM Austin. He received a B.S.E.E. degree from the University
of Kentucky in 1982. He subsequently joined IBM where he has
worked on the development of I/O bridge chips, cache controllers,
and memory controllers for POWER processors from the earliest
POWER systems through the POWER8 processor. Mr. Dodson
is currently a memory stack hardware architect for the POWER8
processor. He is an IBM Master Inventor and coauthor of over
200 issued U.S. patents.
Florian Auernhammer IBM Research - Zurich, Switzerland
(fau@zurich.ibm.com). Dr. Auernhammer is a Research Staff Member
in the Cloud and Computing Infrastructure department at IBM
Research - Zurich. He received an M.S. degree in general engineering
from Ecole Centrale Paris in 2005, and Dipl.-Ing. and Ph.D. degrees
in electrical engineering from the Technical University of Munich
in 2005 and 2011, respectively. He joined IBM at the IBM
Research - Zurich Lab in 2006, where he has worked on I/O host
bridges for InfiniBand and PCI Express, their efficient integration
into coherent fabrics, and I/O virtualization. He is author or coauthor
of six issued patents in addition to several currently pending.
Dr. Auernhammer is a member of the Institute of Electrical and
Electronics Engineers (IEEE).
Patricia M. Sagmeister IBM Research - Zurich, Switzerland
(psa@zurich.ibm.com). Dr. Sagmeister is a Research Staff Member
in the Cloud and Computing Infrastructure department at IBM
Research - Zurich. She received a Dipl.-Inf. degree in computer science
from the University of Passau in 1993, as well as a Ph.D. degree
in computer science from the University of Stuttgart in 2000. In 2013,
she received an M.B.A. degree from Warwick Business School.
She joined IBM in 1999 at IBM Research - Zurich, where she has
worked on various aspects of datacenter optimization. In 2008
and 2009, she had an assignment at the IBM Thomas J. Watson
Research Center working specifically on POWER8 I/O architecture.
She is author or coauthor of several patents and technical papers.
Dr. Sagmeister is a senior member of the Institute of Electrical and
Electronics Engineers (IEEE).
Guy L. Guthrie IBM Systems and Technology Group, Austin,
TX 78758 USA (gguthrie@us.ibm.com). Mr. Guthrie is a Senior
Technical Staff Member on the POWER Processor development team
in the Systems and Technology Group and is an architect for the
IBM POWER8 cache hierarchy, coherence protocol, SMP interconnect,
memory and I/O subsystems. He served in a similar role for POWER4,
POWER5, POWER6, and POWER7 programs as well. Prior to that,
he worked as a hardware development engineer on several PCI
(Peripheral Component Interconnect) Host Bridge designs and
also worked in the IBM Federal Systems Division on a number
of IBM Signal Processor Development programs. He received his
B.S. degree in electrical engineering from Ohio State University
in 1985. Mr. Guthrie is an IBM Master Inventor and holds
184 issued U.S. patents.
Charles F. Marino IBM Systems and Technology Group,
Austin, TX 78758 USA (marinoc@us.ibm.com). Mr. Marino received
his B.S. degree in electrical and computer engineering from
Carnegie-Mellon University. He is a Senior Engineer in the IBM
Systems and Technology Group. In 1984, he joined IBM in Owego,
New York. Mr. Marino is currently the interconnect team lead for
the IBM POWER8 servers.
IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015
 Michael Siegel Systems and Technology Group, Research
Triangle Park, NC 27709 USA (siegelm@us.ibm.com). Mr. Siegel is a
Senior Technical Staff Member in the IBM Systems and Technology
Group. He currently works as the hardware architect of coherent bus
architectures developed for POWER system applications. In 2003,
Mr. Siegel joined the PowerPC* development team to support the
high performance processor roadmap and creating standard products
for both internal and external customers. Roles included memory
controller design lead and coherency bus design lead and architect,
leading to the development of the PowerBus architecture in use in
POWER processor chips starting with POWER7. He supported
multiple projects including IBM POWER7 and POWER8 processors,
and assisting in customer discussions for future game and joint chip
development activity, incorporating new functions into the architecture
as system requirements evolved. While working on the POWER8
PowerBus architecture, he worked as a hardware architect of the
processor side of the Coherent Attached Processor Interface, working
with development teams spanning the processor chip and first
generation coherently attached external coprocessor by specifying
hardware behavior and micro code architecture. Prior to his work
in the IBM Systems and Technology Group, Mr. Siegel worked in
NHD developing the Rainier network processor, the IEEE 802.5
DTR standard, IBM token ring switch products based on the standard.
He started working for IBM in Poughkeepsie, New York, on the
3081 I/O subsystem and the ES/9000 Vector Facility. Mr. Siegel
is an IBM Master inventor and holds over 70 patents issued by
the U.S. Patent Office.

Bart Blaner IBM Systems and Technology Group, Essex

Junction, VT 05452 USA (blaner@us.ibm.com). Mr. Blaner earned
a B.S.E.E. degree from Clarkson University. He is a Senior Technical
Staff Member in the POWER development team of the Systems
and Technology Group. He joined IBM in 1984 and has held a
variety of design and leadership positions in processor and ASIC
development. Recently, he has led accelerator designs in POWER7+
and POWER8 platforms, including the Coherent Accelerator Processor
Proxy design. He is presently focused on the architecture and
implementation of hardware acceleration technologies spanning a
variety of applications for future POWER processors. He is an IBM
Master Inventor, a Senior Member of the Institute of Electrical and
Electronics Engineers (IEEE) and holds more than 30 patents.

IBM J. RES. & DEV. VOL. 59 NO. 1 PAPER 3 JANUARY/FEBRUARY 2015 W. J. STARKE ET AL. 3 : 13

View publication stats