This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
A 3-D CPU-FPGA-DRAM Hybrid Architecture
for Low-Power Computation
Xianmin Chen and Niraj K. Jha, Fellow, IEEE
Abstract The power budget is expected to limit the portion
of the chip that we can power ON at the upcoming technology
nodes. This problem, known as the utilization wall or dark
silicon, is becoming increasingly serious. With the introduction of
3-D integrated circuits (ICs), it is likely to become more severe.
Thus, how to take advantage of the extra transistors, made
available by Moores law and the onset of 3-D ICs, within the
power budget poses a significant challenge to system designers.
To address this challenge, we propose a 3-D hybrid architecture
consisting of a CPU layer with multiple cores, a field-
programmable gate array (FPGA) layer, and a DRAM layer.
The architecture is designed for low power without sacrific-
ing performance. The FPGA layer is capable of supporting
a large number of accelerators. It is placed adjacent to the
CPU layer, with a communication mechanism that allows it
to access CPU data caches directly. This enables fast switches
between these two layers. This architecture reduces the power
and energy significantly, at better or similar performance. This
then alleviates the dark silicon problem by letting us power ON
more components to achieve higher performance. We evaluate
the proposed architecture through a new framework we have
developed. Relative to the out-of-order CPU, the accelerators on
the FPGA layer can reduce function-level power by 6.9 and
energy-delay product (EDP) by 7.2, and application-level power
by 1.9 and EDP by 2.2, while delivering similar performance.
For the entire system, this translates to a 47.5% power reduction
relative to a baseline system that consists of a CPU layer and a
DRAM layer. This also translates to a 72.9% power reduction
relative to an alternative system that consists of a CPU layer,
an L3 cache layer, and a DRAM layer.
Index Terms 3-D integrated circuit (IC), CPU, DRAM,
field-programmable gate array (FPGA), hybrid architecture,
low power.
I. I NTRODUCTION
P OWER consumption is among the top concerns of todays
chip designers. Although Moores law continues to enable
increased transistor count, the power budget constrains the
portion of the chip we can power ON. This problem is known
as the utilization wall or dark silicon [1], [2]. It is expected
to be a major showstopper for high-performance designs at
the upcoming technology nodes. Another recent trend toward
3-D integrated circuits (ICs) aggravates this problem further.
In the 3-D ICs, multiple dies are stacked together in
one package, making more transistors available than predicted
by Moores law. This means designers have more transistors
than they can afford to power ON. Thus, they face the
Manuscript received July 7, 2015; revised September 17, 2015; accepted
September 23, 2015. This work was supported by the National Science
Foundation under Grant CCF-1216457.
The authors are with the Department of Electrical Engineering, Princeton
University, Princeton, NJ 08544 USA (e-mail: xianminc@princeton.edu;
jha@princeton.edu).
Digital Object Identifier 10.1109/TVLSI.2015.2483525
1063-8210 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
challenge of how the extra transistors can be translated to
better performance under the given power budget.
Adding specialized accelerators to aid general-purpose
CPUs has been proven to be an effective method for reducing
power consumption. The power saved by these accelerators
can be used to power extra computational components
to deliver better performance. These accelerators include
coprocessors with special instructions, general-purpose
graphic processing units (GPUs), field-programmable gate
arrays (FPGAs), and application-specific ICs (ASICs).
As opposed to CPUs, they have specialized computation
and control logic, thereby reducing the number of active
transistors and, hence, power consumption. Moreover, the
specialization may also offer a performance advantage over
traditional CPUs. This can be traded for further power
reduction by operating accelerators at a lower frequency.
Among different types of accelerators, ASICs are the
most power-efficient but offer almost no functional flexibility.
By contrast, FPGAs provide higher flexibility and reason-
able power efficiency, making them perfect candidates for
low-power computation. Previous studies have shown the
advantage of FPGAs over CPUs and GPUs in this respect.
For example, Kestur et al. [3] show that using the FPGAs
for basic linear algebra subroutines can achieve 2.7 to
293 better energy efficiency than using GPUs and CPUs.
Chen and Singh [4] show that using the FPGAs, based on high-
level synthesis (HLS), for a document filtering algorithm can
achieve 5.5 and 5.25 better energy efficiency than using
GPUs and CPUs, respectively. Other works also report similar
energy efficiency improvements of FPGAs over GPUs and/or
CPUs [5], [6]. Although these works establish the advantage
of FPGAs in reducing energy consumption, the accelerators
used in them are highly customized for certain applications.
In addition, these FPGA accelerators are only suitable for
large computational jobs, since FPGAs are located off-chip
and switching computation to them is costly.
In this paper, we propose a 3-D hybrid architecture for
low-power computation to forestall the dark silicon problem
while enabling the highest performance possible without hit-
ting the power wall. The architecture is easy to use and meant
for general-purpose computations. It consists of a CPU layer,
an FPGA layer, and a DRAM layer. It takes advantage of
through-silicon via (TSV)-based 3-D IC technology to provide
vast FPGA resources close to CPUs. We use a mechanism to
allow FPGA accelerators to access CPU data caches directly.
This enables fast switches between a CPU and an FPGA
accelerator. The FPGA layer is quite flexible in terms of
how it is used. It could be configured as a large number
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
of small accelerators, a small number of large accelerators,
or something in between. Moreover, the use of a separate
FPGA layer allows it to be fabricated separately using the
state-of-the-art FPGA design. Since the architecture is geared
toward general-purpose computation, we use a state-of-the-art
HLS tool to generate the accelerators directly. This eliminates
the costly manual design process. Note that the proposed
architecture is generic in the sense that it can be extended
to include multiple FPGA layers and multiple DRAM layers.
We have developed an evaluation framework to simulate sys-
tems with such an architecture. Our experimental results show
that, compared with the out-of-order CPU, the accelerators on
the FPGA layer can reduce function-level power and EDP by
6.9 and 7.2, respectively, and application-level power and
EDP by 1.9 and 2.2, respectively, while delivering similar
performance. Compared with a baseline system consisting of
a CPU layer and a DRAM layer, the proposed architecture
can achieve 47.5% power reduction for the entire system.
Compared with an alternative system consisting of a CPU
layer, an L3 cache layer, and a DRAM layer, the proposed
architecture can achieve 72.9% power reduction for the entire
system.
We summarize the contributions of this paper as follows.
1) We propose a new 3-D architecture with the CPU,
FPGA, and DRAM layers for low-power computation.
2) We demonstrate a communication mechanism that
allows the FPGA layer to access CPU data caches,
directly crossing clock domains.
3) We develop a framework to evaluate the proposed archi-
tecture.
4) We evaluate performance, power, and thermal impacts
of the architecture, both at the function and application
levels.
The rest of this paper is organized as follows. Section II
reviews related work on reconfigurable computing and
3-D ICs. Section III presents our proposed architecture.
Section IV describes the framework we have developed to
evaluate the architecture. Section V presents the experimental
results. Finally, the conclusion is drawn in Section VI.
II. R ELATED W ORK
This section reviews related work on the reconfigurable
computing and the 3-D ICs that will help place our work in
the proper context.
A. Reconfigurable Computing
Reconfigurable computing uses reconfigurable fabrics to
perform computations. A reconfigurable computing system
can be built using two methods: 1) using customized
reconfigurable fabrics and 2) using commodity FPGAs.
Several designs, such as Garp, NAPA1000, and Chimaera,
were constructed using the first method [7][11]. Customized
reconfigurable fabrics can usually run at higher frequencies
than commodity FPGAs due to the customization. They
can also be embedded very close to the CPU to enable
fast communication. However, one drawback of this method
is that only a small amount of reconfigurable fabric can
be placed near the CPU due to die area limitations. This
limits the complexity of accelerators implemented on the
reconfigurable fabric. Another drawback is that this method
cannot take advantage of the latest developments in the
FPGA industry. In contrast, systems built using the second
method use the state-of-the-art commodity FPGAs [12][16].
The FPGA is off-chip and connects to the rest of the system
through a bus or other mechanisms. The off-chip FPGA offers
a considerable amount of reconfigurable resources. However,
its drawback is that the communication between the FPGA
and the CPU is costly in terms of both time and energy.
We will show that our proposed architecture can overcome
these drawbacks using 3-D stacking. This not only enables
fast communication but also provides enough FPGA resources
and takes advantage of the state-of-the-art FPGA designs.
Designing FPGA-based accelerators requires hardware
expertise, which is a barrier to wider adoption of reconfig-
urable computing systems. Fortunately, recent developments
in the HLS have significantly lowered this barrier. HLS tools
directly convert functions written in high-level languages to
hardware accelerators [17][22]. Some researchers have also
studied direct compilation of compute unified device architec-
ture (CUDA) or OpenCL (designed for heterogeneous parallel
computing) to hardware [23], [24]. This takes advantage of
the parallelism exposed by both languages and is suitable for
systems with off-chip FPGAs. In this paper, we use a state-
of-the-art HLS tool called Vivado HLS (known as AutoESL
before Xilinxs acquisition) [25]. A third-party report indicates
that it can achieve a quality level similar to that of manual
designs by hardware experts, for several applications [26].
B. 3-D ICs
The 3-D ICs have multiple layers of dies, which are
connected by the TSVs, stacked in a single package. This
makes the so-called More-than-Moore scaling possible. In the
3-D ICs, different layers can be manufactured and tested
separately. This not only increases the yield but also allows
different layers to be optimized and manufactured in different
technologies. Researchers have studied various approaches
for using the 3-D IC architectures to improve performance
and/or reduce power consumption. One popular approach is
to put caches or memories in a layer separate from the CPU
layer [27][32]. Although this approach is easy to implement,
it is confronted with diminishing returns after memory/cache
sizes exceed certain thresholds. Another approach for using
the 3-D ICs is to split computational components, such as
CPU cores, into multiple layers. This can reduce the wire
length compared with the original 2-D designs, thus lead to
performance and power benefits. However, this approach has
two drawbacks: 1) it requires a very large number of TSVs,
which are costly and 2) since computational components
generally consume more power than caches or memories,
stacking multiple layers of computational components may
create thermal problems [33]. Compared with the above two
approaches, as an alternative, adding an FPGA as a sepa-
rate layer has two unique benefits: 1) the FPGA layer adds
computational power and 2) since an FPGA layer consumes
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION
Fig. 1. Proposed 3-D hybrid architecture with a CPU layer, an FPGA layer,
and a DRAM layer.
much less power than a CPU layer, adding it is much less
likely to lead to thermal problems. Because of these benefits,
a recent effort, the FlexTiles platform, makes a move in this
direction [34]. However, it uses a special embedded FPGA,
which does not take advantage of recent FPGA developments.
In this platform, the FPGA accelerators are connected to CPUs
through a network-on-chip, incurring a large communication
overhead. It also uses a dataflow programming model, which
limits its use. As opposed to the FlexTiles platform, our
proposed architecture takes advantage of the state-of-the-art
FPGA and HLS technologies. It is easy to use and suitable for
general-purpose computation. We also conduct experiments to
evaluate its power and thermal impact, which the FlexTiles
work lacks.
III. H YBRID A RCHITECTURE
This section presents our hybrid 3-D architecture.
We first introduce the system architecture and then
present its implementation details, e.g., FPGA acceleration,
instruction set architecture (ISA) extension, CPU-accelerator
communication, and memory access optimization. Finally, we
discuss manufacture, test, cost, and reliability of the proposed
architecture.
A. System Architecture
As shown in Fig. 1, our proposed hybrid 3-D architecture
consists of three layers: 1) CPU; 2) FPGA; and 3) DRAM.
An extended configuration may consist of multiple FPGA and
DRAM layers. For simplicity, we will focus only on the basic
configuration in this paper. The cores located on the CPU layer
run at a high frequency (1 GHz or higher) and generate the
most heat. Therefore, the CPU layer is located closest to
the heat sink to expedite heat dissipation. Adjacent to the
CPU layer is the FPGA layer. It contains accelerators for low-
power computation. These accelerators run at a much lower
frequency (several hundred megahertz) than the CPU cores.
As a result, they generate much less heat and are placed farther
from the heat sink. Above the FPGA layer is the DRAM layer.
It contains systems main memory. It usually generates the
least heat and is, hence, placed farthest from the heat sink.
Adjacency of the CPU and FPGA layers enables fast
communication between the CPU cores and the accelerators
via TSVs that connect the two layers. Each CPU core can
3
have, associated with it, one or multiple accelerators, which
are implemented above the core on the FPGA layer, for the
given application. An accelerator accesses the main memory
via the data cache of the CPU core it belongs to. Such
a design has several advantages: 1) accelerators can use
existing infrastructure, including the CPU memory hierarchy
and memory controllers, which saves hardware resource;
2) caches on the CPU layer can expedite memory accesses and
increase accelerator performance; 3) sharing caches provides
a uniform view of the memory space among CPU cores and
accelerators, simplifying programming; and 4) sharing caches
eliminates the need for copying data, enabling fast switches
between a CPU core and an accelerator.
B. FPGA Accelerator
The accelerators implemented on the FPGA layer are gen-
erated directly from C/C++ code using an HLS tool. The
tool we chose is Vivado HLS, a state-of-the-art tool from
Xilinx [25]. During HLS, the tool maps the input arguments
and return values of C/C++ functions to input and output
signals of the generated accelerator. If an input argument is
a value, it is mapped to an input signal that passes the value
directly. If an input argument is a pointer, it is mapped to a
memory access interface, which consists of the address, data,
and handshaking signals. A return value, if it exists, is mapped
to an output signal, passing the value directly. Besides these
signals, Vivado HLS also adds necessary control signals to the
accelerator, including clock, reset, and handshaking signals.
Inside the accelerator, sequential C/C++ code is mapped
to a hardware structure based on the data and control
flows. Fig. 2(a) and (b) shows an HLS example for an
N N matrix multiplication. Fig. 2(a) shows the original
function in C, which multiplies matrices A and B and stores
the result as matrix C. All matrices are stored in row-major
order. In the function, the two outer loops iterate through
each element of matrix C, and the innermost loop iterates
through the corresponding row of matrix A and column of
matrix B to calculate the matrix C element. Fig. 2(b) shows
the generated accelerator structure. Inputs A, B, and C are
pointers and mapped to three memory access interfaces. Input
argument N is mapped directly to a value-passing signal. In the
generated accelerator, the data path contains four multipliers
to multiply four pairs of inputs and three adders to sum up
the products. It is capable of handling four iterations of the
original functions inner loop. The data path is controlled by
the generated finite-state machine.
During accelerator generation, we use three types
of optimizations supported by Vivado HLS to increase
accelerator performance: 1) pipelining; 2) loop unrolling;
and 3) dataflow. The pipelining optimization splits the data
path into stages by inserting pipeline registers, allowing it to
tackle one batch of inputs per cycle to increase throughput.
The loop unrolling optimization unrolls a loop to allow
multiple iterations of the loop body to be finished in parallel.
This increases the performance at the cost of extra hardware
resources. For instance, in the N N matrix multiplication
example, the innermost loop of the original function is
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

Fig. 2. Example of HLS-generated accelerator. (a) Matrix multiplication source code. (b) Generated accelerator.

TABLE I
A CCELERATOR I NSTRUCTIONS

but may have one or more data channels. With the wrapper
added, an accelerator can be easily connected to the rest of
the system.

Fig. 3. Accelerator interface. C. CPU ISA Extension

Using instructions to directly configure and call accelerators
unrolled four times. The dataflow optimization enables
also allows fast switches between a CPU and an accelerator.
task-level (functions and loops) pipelining by inserting
Therefore, we added two new instructions to the CPU ISA to
buffers, such as first-in first-out (FIFO) or ping-pong buffers.
access the accelerators: 1) WRITE_ACC and 2) RUN_ACC.
All these optimizations can be performed via directives,
WRITE_ACC writes configuration registers in the accelerator.
which do not require any source code change [35].
It has two operands: 1) the value to be written and 2) the
The generated accelerator usually works at a lower
register index. RUN_ACC starts the accelerator. It also
frequency than the CPU core. However, it may still perform
has two operands: the accelerator index and another that
computations faster than the CPU core due to the following
indicates where the execution result (the return value of the
reasons: 1) it can have parallel data paths; 2) it can use cus-
original function) will be stored. Table I summarizes the
tomized pipelines to achieve higher throughput; 3) it can merge
two instructions. In the ALPHA instruction format, each
multiple operations into a single operation in its data path;
instruction can have at most three operands, but only two are
and 4) it can have customized communication mechanisms
used in our design. As we will see, the ISA extension is simple
inside, which is not constrained by the memory bottleneck.
enough to implement, yet support complex accelerators.
To connect an accelerator to a CPU core working in another
A typical call to an accelerator has two steps: 1) the CPU
clock domain, we add a wrapper to the generated accelerator
uses the WRITE_ACC instruction to write accelerator registers
for interclock-domain communication, as shown in Fig. 3.
and 2) it uses the RUN_ACC instruction to start the accelerator
The wrapper has two parts: 1) configuration/status registers
and receive the execution result. During instruction execution,
and 2) interface protocol conversion. The configuration/status
an accelerator may access memory through the CPU data
registers store the inputs and internal states. The interface
cache. Meanwhile, the CPU waits until the accelerator finishes
protocol conversion part converts the generated accelerator
or a translation lookaside buffer (TLB) miss occurs (discussed
interface to an interface consisting of FIFOs to provide a
later). A more sophisticated implementation may support the
simple interface to connect to the rest of the system. Each
simultaneous running of the CPU and accelerator. However,
pair of an inbound FIFO and an outbound FIFO is grouped
we did not choose this implementation, since our main goal
into a channel. Based on the functionality, a channel belongs
is to reduce power.
to one of two possible categories: 1) control channel and
2) data channel. A control channel is responsible for the com-
munication of control information, including the information D. CPU-Accelerator Communication
to set up the accelerator and the execution result. A data An FPGA accelerator can access the CPU data cache
channel is responsible for memory access when the accelerator directly. To enable this sharing, a switch is added to
is running. The outbound FIFO of each data channel conveys enable multiplexing, as shown in Fig. 4. At any given time,
the memory access request, and the inbound FIFO conveys either the CPU core or the accelerator has access to the data
the response. Each accelerator has only one control channel cache. Since the switch only adds a 2-to-1 multiplexer between
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION
Fig. 4. CPU-accelerator communication mechanism.
the CPU kernel and the data TLB (DTLB)/cache, it only incurs
a minor increase in path delay, which we analyze further
in Section V. The CPU and accelerator are connected through
a pair of FIFOs. These FIFOs are used as buffers to alleviate
the speed mismatch, because the CPU core operates at a higher
frequency and the accelerator at a lower one. On the CPU
side, we use an adapter, called CPU-side adapter, to connect
the FIFOs with the data cache switch. On the FPGA side,
another adapter, called accelerator-side adapter, is used to
connect the FIFOs to accelerator channels. Each accelerator
may have more than one channel. Therefore, the accelerator-
side adapter embeds both multiplexing and demultiplexing
functions inside. Because the function of the accelerator-side
adapter is fixed, it can be prefabricated directly into the FPGA,
as in the case of the 3-D network interface [36]. This enables
the accelerator-side adapter to work at the same frequency as
that of the CPU core. As a result, the CPU data cache and both
adapters work at a higher frequency than the accelerator, thus
providing enough bandwidth for the accelerator. This is similar
to the idea of multipumping [37], where the cache operates
at a higher frequency than the FPGA accelerator, allowing
multiple memory accesses per cycle by the accelerator. Note
that our communication mechanism does not require any extra
resources for caches and memory controllers. As a result,
it saves resources compared with the multipumping design.
Six types of packets are used for CPU-accelerator
communication, as shown in Fig. 5. Three go from the CPU
to the accelerator and the other three in the opposite direction.
In each packet, the first two bits indicate the packet type. The
remaining bits provide other information, such as the address.
Details of each type of packet are given as follows.
1) Start (Starting an Accelerator): The index field of
the packet indicates the index of the accelerator (in a
multiaccelerator scenario).
2) Write (Writing an Accelerator Register): The index and
value fields of the packet indicate the register index and
value to be written, respectively.
3) Memory Response (Returning Memory Read Result):
The value field in the packet is the memory read result.
4) Done (Indicating Accelerator Execution is Finished):
The value and size fields in the packet indicate the
5
Fig. 5. Packet format.
value and data size of the return value (if it exists),
respectively. Binary codes 00, 01, 10, and 11 indicate
1-, 2-, 4-, and 8-byte data size, respectively.
5) Memory Read (Reading Memory): The address and size
fields correspond to the address and data size of the read
operation, respectively. Data size encoding is the same
as above.
6) Memory Write (Writing Memory): The address, value,
and size fields of the packet correspond to the address,
value, and data size, respectively. Data size encoding is
the same as above.
Fig. 6(a) shows an example of CPU-accelerator commu-
nication using the packets introduced above. First, the CPU
executes a WRITE_ACC instruction, which sends a Write
packet (packet a) to the accelerator. Then, the CPU executes a
RUN_ACC instruction, sending a Start packet (packet b) to the
accelerator. During accelerator execution, it accesses memory
twice: for a memory read and a memory write. Memory read
leads to a Memory read packet (packet c) from the accelerator
to the CPU and a Memory response packet (packet d) in the
opposite direction as a response. Memory write leads to only
a Memory write packet (packet e) from the accelerator to
the CPU. When the accelerator has finished, it sends a Done
packet (packet f ) to the CPU. In our design, the accelerator
accesses the virtual memory space directly. This means that a
TLB miss may occur during memory accesses. Fig. 6(b) shows
an example of such a scenario where the Memory read packet
(packet i ) triggers a TLB miss exception. In this situation,
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
Fig. 6. CPU-accelerator communication examples. (a) Without TLB miss.
(b) With TLB miss.
the CPU immediately suspends the RUN_ACC instruction and
starts the TLB miss exception handling procedure. At the
same time, the CPU-side adapter stores the packet that triggers
the exception and stops receiving any new packet. When the
CPU is handling an exception, the accelerator keeps running
until it can no longer continue. Once the CPU has finished
exception handling, it re-executes the RUN_ACC function,
which resends a Start packet (packet j ) to the accelerator.
When the CPU-side adapter receives it, it does not forward
the packet to the accelerator. Instead, it deletes the packet and
sends the stored packet (renamed as packet k) to the CPU to
reread the address that caused the TLB miss. At this time,
the memory read succeeds and the accelerator continues its
execution. Since packets j and k only exist between the CPU
and the CPU-side adapter, we have marked them with a
dotted line in Fig. 6(b).
E. Memory Access Optimization
Besides the optimizations performed with directives men-
tioned in Section III-B, we also perform optimizations specif-
ically targeted at speeding up memory access. During HLS,
data channels (memory interfaces) are generated for pointer-
passed arguments in the original function. Since the accelerator
operates at a lower frequency than the CPU core and each data
channel can only access one batch of data per cycle, memory
access may become a performance bottleneck. Note that the
root of this problem is the low-frequency nature of FPGA
accelerators. In our architecture, the accelerator accesses
the CPU data cache directly, which operates at a higher
frequency and provides enough bandwidth. To tap into the
bandwidth, we need to duplicate the data channels to access
the cache data in parallel. Fig. 7(a) shows an example of an
accelerator with duplicated channels for pointer argument A.
As mentioned earlier, the CPU data cache has enough band-
width to feed the duplicated channels. If this bandwidth is still
not sufficient, we can create multiport scratchpad memories
inside the FPGA. They load needed data from the CPU data
Fig. 7. Memory access optimization. (a) Duplicate data channels. (b) Insert
scratchpad memories.
cache at the beginning of accelerator execution, provide these
data to the accelerator and store results during execution,
and store the execution results back to the CPU data cache
at the end. They have multiple ports to deliver sufficient
bandwidth during accelerator execution. Fig. 7(b) shows an
example of a scratchpad memory added for the data channel
for A. Choi et al. [37] and Putnam et al. [38] show that
inserting caches or scratchpad memories is possible with the
help of HLS tools. In our implementation, since we do not
have access to the source code of Vivado HLS, instead of
modifying the HLS tool, we modify the source code of the
accelerated functions to direct the HLS tool to synthesize the
desired scratchpad memories. We use this to mimic direct HLS
support.
F. Manufacture, Test, Cost, and Reliability
In this section, we provide a qualitative analysis of manu-
facture, test, cost, and reliability of our proposed architecture.
First, to manufacture a chip based on our proposed architec-
ture, the die for each layer can be manufactured independently
and then bonded together using face-to-back bonding. The
number of TSVs needed for CPU-accelerator communication
is determined by the data width of the FIFOs. Based on the
packet sizes targeted in Section III-D, the two FIFOs used
by each CPU core are 71- and 132-bit wide, respectively.
For a typical design with eight cores, this translates to
1624 [(71 + 132) 8] TSVs. Together with 608 TSVs
(512 bits for data, 64 bits for address, and 32 bits for
control) needed by the memory controller, the total number
of TSVs needed is 2232. A TSV redundancy scheme may
require a slightly higher number to cope with TSV failures
in manufacture. However, since the increase is minor and
depends on the manufacturing process (e.g., the redundancy
scheme presented in [39] achieves a 95% recovery rate by
adding one redundant TSV per 25-TSV block), we have not
modeled TSV redundancy in this paper.
Second, 3-D ICs pose significant test challenges [40].
To achieve a better yield, the die for each layer needs
to be tested before bonding. In our proposed architecture,
since each layer contains complete functionality, prebond
testing is easier compared with the case when fine-grain
partitioning of functionality is employed across multiple die
layers. In particular, we can add an independent test controller
to each layer for this purpose, and each module on each
layer can be treated as an isolated test island and tested
separately [41].
Third, the cost of a 3-D IC consists of wafer cost, bonding
cost, packaging cost (depends on both pin count and die size),
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION
Fig. 8. Evaluation framework.
and cooling cost [42]. Our proposed architecture incurs little
overhead for the first three costs, given the relatively small
number of TSVs it uses. Since it reduces power consumption
and alleviates hotspots on chip, as shown in Section V-F, it
also reduces the cooling cost.
Finally, in a 3-D IC, the difference in thermal expansion
coefficients between the TSV copper and the surrounding
silicon causes thermal-induced stresses that lead to mechanical
reliability issues. We can use the techniques and tools proposed
in [43] to deal with such issues.
IV. E VALUATION F RAMEWORK
We designed a customized framework to evaluate our
proposed architecture. Fig. 8 shows the structure of the
framework. It consists of four parts: 1) HLS; 2) performance
simulation; 3) power simulation; and 4) thermal simulation.
The HLS part contains Vivado HLS, which converts a
C/C++ function into accelerator hardware at the register-
transfer level. The function to be converted is chosen based
on software profiling results. The generated accelerator
has multiple versions with the same underlying hardware
structure but described in different languages: Verilog
hardware description language (HDL), VHDL, and SystemC.
Then, a script parses the accelerator interface and changes
the benchmark source code to use the generated accelerator
instead of the original function. The new source code is
cross-compiled into an executable binary for performance
simulation.
The performance simulator we use is gem5 [44], an open-
source full-system simulator. Its open-source nature allows
us to add accelerator simulation support. In particular, we
integrate the SystemC version of accelerator code, as well
7
TABLE II
S YSTEM C ONFIGURATION
as wrappers generated by our scripts, into gem5 code and
simulate them together. We choose to use the SystemC version,
since it can be compiled with gem5 source code (written
mainly in C++ and Python). Since the gem5 simulator and the
SystemC simulation engine have two separate event queues,
we employ the method described in [45] to synchronize the
two event queues and enable joint simulation.
After the integration, gem5 yields performance results,
including component-level utilization statistics and wave-
forms for the accelerator (output of the SystemC simulation
engine). For most system components except accelerators,
TSVs, adapters, and DRAM, we use McPAT to estimate
power [46]. McPAT takes the component-level utilization
statistics, together with the system configuration, to estimate
power for each component. As for the accelerator, we first
synthesize, place, and route the accelerators Verilog HDL
code to get its low-level implementation details. Then, we use
them with the dumped waveform to estimate the accelerator
power. This is done using the Xilinx Vivado tool [25]. For
TSVs and adapters, we build power macromodels by adding up
the power consumptions of their internal components, similar
to the way McPAT is built. We use the macromodels, along
with their utilization statistics, to estimate their power. For
DRAM, we use Microns DRAM power calculator to estimate
power (not shown in the figure) [47]. After we have all the
power data, we use HotSpot (with 3-D IC support extension
from [31]) to perform thermal simulation [48]. We have
automated the entire process, shown in Fig. 8, to expedite
experimentation.
1 We assume a 50-cycle memory controller latency (half of the 100-cycle
latency assumed in an off-chip 2-D version), the same as that in [31].
Memory delay is calculated by summing up tRAS and tRP (35 and 13.75 ns,
respectively, for Micron DDR3-1600 synchronous dynamic random-access
memory (SDRAM) [49]), leading to 98 cycles at 2-GHz frequency. In all,
the memory delay is 148 cycles.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
TABLE III
B ENCHMARKS
V. E XPERIMENTAL R ESULTS
This section presents the experimental results. We first
discuss the experimental setup. Then, we show the resource
utilization rate, function-level/application-level performance,
and power of FPGA accelerators. Then, we provide results
for performance and power optimization. Finally, we evaluate
the system-level power and thermal impact of the proposed
architecture.
A. Experimental Setup
To evaluate the proposed architecture, we use an
implementation with an eight-core CPU layer, an FPGA
layer, and a DRAM layer. We refer to it as CPU + Acc
henceforth. To understand how accelerators in the proposed
architecture impact the performance and power, we compare
CPU + Acc with a baseline system (referred to as CPU-only)
consisting of a CPU layer and a DRAM layer, but no FPGA
layer. Since CPU-only has one less layer than CPU + Acc,
we also compare CPU + Acc with another system that
has the same number of layers. The system, referred to as
CPU + L3, consists of a CPU layer, an L3 cache layer,
and a DRAM layer. Table II shows the configurations of
the three systems. They are all assumed to be implemented
in the 28-nm CMOS technology except the DRAM layers,
which are implemented in the 32-nm CMOS technology.
We further assume that memory cells of all caches are
implemented with low-standby-power transistors to reduce
leakage power, while the rest of the system is implemented
with high-performance transistors to improve performance.
Such a feature is supported by the cache-modeling tool,
CACTI [50], which is used inside McPAT.
We assume 1/2/6-m dimensions for TSV diameter/pitch/
depth, which are the minimum ITRS diameter/pitch/depth
guidelines for years 20122014 (they are also within the
predicted range for years 20152018) [51]. We calculate the
TSV capacitance using the method given in [52] for power
estimation. For the CPU layer of CPU + Acc, the estimated
die size is 84.07 mm2 of which TSVs consume 0.02%.
In CPU-only and CPU + L3, the CPU layers have slightly
smaller (0.62%) die sizes than that of CPU + Acc, because
they do not have the CPU-side adapters. For the FPGA layer of
CPU + Acc, we use Xilinx Kintex FPGA (xc7k160tfbg484-3;
implemented in the 28-nm technology). Its die size is
TABLE IV
FPGA R ESOURCE U TILIZATION
8.54 12.05 mm2 [53]. We scale it to the same size as
the L3 cache layer (87.76 mm2 ) of CPU + L3 for a fair
comparison. For the DRAM layers of all three systems, since
the die size information of commercial DRAMs is not easily
accessible, we assume 6F 2 area per memory cell and 56.08%
area efficiency to estimate the die size [54], where F is
the feature size. Following this method, we estimate the
die size of a 1-GB DRAM to be 94.11 mm2 . In all three
systems, the CPU cores run at 2 GHz. In CPU + Acc, the
FPGA accelerators run at 300 MHz, which is the best we
can get on the FPGA. The switch between the CPU kernel
and the DTLB/cache adds 16.5-ps delay to the path delay
(based on McPAT), which amounts to 3.3% of the CPU clock
period. We assume that a margin exists in CPU + Acc to
tolerate this delay increase and allows the CPU cores to run
at 2 GHz. Another method to estimate the delay overhead is
to estimate both the clock period and the switch delay with
fanout-of-4 inverter delay, as in [55]. This method also yields
similar results.
We use the benchmarks listed in Table III to test
CPU-only, CPU + L3, and CPU + Acc. The first two bench-
marks are microbenchmarks: 1) integer and 2) floating-point
512512 matrix multiplications. Both split 512512 matrices
into 16 16 submatrices to increase cache performance. The
next five benchmarks (gsm to bzip2) are from the CHStone
suite plus bzip2 [56]. Since several of these benchmarks
are very short, their main functions are repeated multiple
times using loops to extend their duration in our experiments.
The remaining benchmarks, blackscholes and 264, are from
the PARSEC benchmark suite [57], [58]. Table III includes
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION 9

Fig. 9. Function-level power, energy, execution time, and EDP comparison. (a) Function power. (b) Function energy. (c) Function execution time.
(d) Function EDP.

descriptions of the benchmarks. The function and percentage
columns show one or two of the most time-consuming func-
tions in each benchmark and their percentages of the total
execution time, respectively.
B. Resource Utilization
Table IV shows the percentage of resources used by the
generated accelerators. In the name column, the -1 and -2
suffixes indicate the first and second functions when two func-
tions are accelerated in the same benchmark. These suffixes are
also used in subsequent figures in this section. Four types of
resources are listed in the table: lookup table (LUT), register,
block RAM (BRAM), and DSP blocks. Most accelerators
use more than 10% of the LUTs except 264. For 264,
the accelerated functions, 264_pixel_sad_4_16 16 and
pixel_satd_w h, are small functions and thus consume only
2.3% and 8.2% of the LUTs, respectively.
C. CPU Versus Accelerator: Function-Level Power,
Energy, Performance, and EDP
In this section, we compare the power, energy, performance,
and EDP of the out-of-order CPU core and the FPGA
accelerators when running the same functions. The first two
bars, marked as CPU and Acc, in Fig. 9(a)(d) compare
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

Fig. 10. Function-level power (absolute values).

the power, energy, execution time, and EDP of the CPU

(in CPU-only) and the FPGA accelerators (in CPU + Acc).
All results are normalized to that of the CPU.2 When esti-
mating CPU power and energy, we include data cache/TLB
and instruction cache/TLB inside the core. When estimat-
ing accelerator power and energy, we only include the data
cache (D$) and DTLB, since accelerators only access them.
From Fig. 9(a), we can see that the accelerators consume sig-
nificantly less power than the CPU. This is because the accel-
erators contain dedicated computational structures, whereas
the out-of-order CPU contains components that do not con-
tribute directly to computation, such as components dedicated
to instruction fetching and decoding, branch prediction, and
register renaming. When accelerators are used, the data cache
and DTLB consume a very small amount of power, owing
to the fact that the leakage power of memory cells inside
the data cache is reduced using low-standby-power transistors.
To better understand the impact of using accelerators, we also
show the absolute power values in Fig. 10. On an average, the
Fig. 11. Cache performance. (a) Data cache miss rate. (b) Data cache
accelerators consume only 22.8% of the power consumed by accesses.
the CPU. The power reduction translates to energy reduction
in Fig. 9(b), where the accelerators can be seen to consume To evaluate the benefit of accessing data directly from the
only 20.2% of the energy consumed by the CPU on an CPU data cache, we measure the data cache miss rate of
average. As mentioned earlier, although a CPU core runs at a the CPU and accelerators. The result is shown in Fig. 11(a).
much higher frequency than an accelerator, an accelerator may For most benchmarks, the accelerators have higher miss
still deliver better performance due to its dedicated structure. rates than the CPU. This is because the CPU runs software
This is evident from Fig. 9(c), where, compared with the functions, which have intermediate variables. These variables
CPU, accelerators deliver similar or even better performance usually end up in the data cache and drive the average
for many benchmarks except fmmul, aes, and bzip2. On an miss rate down. By contrast, accelerators only access the
average, the execution time of the accelerators is 82.9% of that cache for pointer-passed arguments. These are inputs from
of the CPU. When combining energy and execution time into outside and less likely to be in the cache. This makes the
one metric, EDP, we see that the accelerators outperform the cache miss rates higher. To show this difference, Fig. 11(b)
CPU on all benchmarks. The average EDP of the accelerators compares the amount of data cache accesses by the CPU and
is 21.5% of that of the CPU. Note that in accelerator power accelerators. On an average, the amount of data cache accesses
consumption calculation, the leakage power of the entire by accelerators is only 23.6% of that of the CPU. In summary,
FPGA is included, which makes the comparison unfair to the all accelerator miss rates are <9%, and the average is 3.6%.
accelerators. When multiple accelerators exist on the FPGA, This means that the CPU data cache provides a very good
this leakage power gets amortized. Thus, for each accelerator, caching for accelerators.
we should only count the portion of power consumed by the
accelerator based on the portion of resources (LUTs) it uses.
In Fig. 9(a)(d), the third bar, labeled +Partial L, shows the D. CPU Versus Accelerator: Application-Level Power,
result after this adjustment. On an average, the power, energy, Energy, Performance, and EDP
and EDP drop to 14.5%, 13.2%, and 13.9% of the CPU, In this section, we evaluate the power, energy, performance,
translating to 6.9, 7.6, and 7.2 reductions, respectively. and EDP of the out-of-order CPU core and the FPGA accel-
The execution time remains the same after the adjustment. erators when running entire benchmark applications on the
proposed architecture. When using accelerators, the CPU is
2 The operating temperature is assumed to be 65 C for power estimation. responsible for the unaccelerated parts of each benchmark.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION 11

Fig. 12. Application-level power, energy, execution time, and EDP comparison. (a) Benchmark power. (b) Benchmark energy. (c) Benchmark execution time.
(d) Benchmark EDP.

Due to this reason, we use CPU + Acc to label results, the
same as the system name. Similarly, we use CPU-only and
CPU + L3 to label the results when running benchmarks on
CPUs of CPU-only and CPU + L3, respectively. Fig. 12(a)(d)
shows the power, energy, execution time, and EDP, respec-
tively, when running each benchmark. On an average, power,
energy, execution time, and EDP of CPU + Acc are 59.7%,
53.5%, 93.8%, and 53.5% of CPU-only, respectively. As in
the case of function-level results, we also include results when
the FPGA leakage power is scaled according to resources used
and show them as the bar labeled +Partial L. in these figures.
On an average, the power, energy, and EDP of CPU + Acc
are now 52.3%, 46.3%, and 46.3% of CPU-only, translating
to 1.9, 2.2, and 2.2 reduction, respectively. As expected,
the power reduction at the application level is less than at the
function level. This is because the accelerators only account
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12
Fig. 13. Benchmark execution time versus frequency.
for part of the execution of each benchmark, and the CPU
is still responsible for the rest. To deal with this issue, we
could use multiple accelerators to cover a larger part of the
execution of each benchmark. This has already been addressed
in our results by accelerating multiple functions for certain
benchmarks.
From Fig. 12(a)(d), we found that CPU + L3 performs
very similarly to CPU-only. The average execution time of
CPU + L3 is 99.8% of the CPU-only. This is because the
L2 cache of CPU + L3 is big enough to accommodate the
working data sets, rendering little advantage from adding
an L3 cache. It confirms the fact that adding caches has
diminishing returns after cache sizes exceed certain thresholds.
As a result, the comparison results between CPU + L3 and
CPU + Acc are very similar to those between CPU-only and
CPU + Acc. On an average, the power, energy, execution time,
and EDP of CPU + Acc (with +Partial L.) is 52.2%, 46.3%,
94%, and 46.3% of CPU + L3, respectively.
E. Performance and Power Optimization
As performance results indicate, CPU + Acc runs slower
than CPU-only for some benchmarks. This may not be allowed
when execution time is a top concern. To solve this problem,
we exploit an automatic selection mechanism to direct the
benchmark to run on the CPU if the accelerator is slower.
This mechanism ensures that the execution time is better than
or the same as that of CPU-only. Note that once an accelerator
is generated, whether it is faster or slower than the CPU
can be determined by simulations beforehand. For situations
where execution speed is data-dependent, a dynamic selection
method can be used, which chooses the CPU or accelerator
dynamically, epoch by epoch, using the history data. In our
experiments, we assume that the time budget for a benchmark
is the same as its execution time on CPU-only. We further
assume that the execution time using the CPU or accelerators
can be predetermined by simulations.
With the power budget assumed above, for benchmarks on
which CPU + Acc runs faster, we also apply a dynamic
voltage and frequency scaling (DVFS) to the CPU core to
further reduce its power. To show that this is applicable,
Fig. 13 shows the benchmark execution time versus the CPU
frequency when using CPU + Acc. The values are normalized
to CPU-only. For six out of the nine benchmarks, even when
we lower the CPU frequency, CPU + Acc can still deliver
better performance than CPU-only. This means we can use
DVFS to achieve more power reduction for these benchmarks.
To incorporate DVFS into power estimation, we scale McPAT
IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
Fig. 14. Comparison of system power and maximum temperature. (a) System
power. (b) Maximum temperature.
results based on voltage and frequency obtained using the
method given in [59]. We assume that the frequency changes
from 2 to 1.5 GHz in 0.1-GHz step, and Vdd changes linearly
with frequency in this range. The bar labeled +Auto. Sel. in
Fig. 12(a)(d) shows the results of using automatic selection
and DVFS. In Fig. 12(a), we can see that the average power
increases to 58.9% of CPU-only, from 52.3% before. This is
because the CPU is chosen to run some benchmarks due to
the execution time constraint. Similarly, the average energy
per benchmark in Fig. 12(b) increases to 55.5% of CPU-only,
from 46.3% before. In Fig. 12(c), all the execution times can
be seen to be equal to or shorter than those of CPU-only. This
ensures no adverse impact on execution time. On an average,
the execution time is reduced to 86.7% of CPU-only, from
93.8% before. From Fig. 12(d), we see that the average EDP
increases to 53.9% of CPU-only, from 46.3% before.
F. System Power and Thermal Impact Evaluation
In this section, we evaluate the system power and thermal
impact of the proposed architecture. The system power
consists of power of all the components in the system,
including all caches and main memory. To fully take
advantage of the FPGA resources, we evaluate a scenario
where multiple homogeneous accelerators exist on the FPGA
layer of CPU + Acc. We use multithread settings for the
benchmarks with multithread support (blackscholes and
264), and duplicate single-thread version for those without
this support. A CPU core and an accelerator (two accelerators
if two functions are accelerated) are allocated to run each
thread. Note that the number of threads is limited by the
FPGA resource utilization shown in Table IV and the number
of CPU cores. We keep increasing the number of threads until
we exhaust at least one type of FPGA resource or reach the
maximum number of CPU cores. The final thread/accelerator
numbers are given in the last column of Table IV. Fig. 14(a)
shows the comparison of system power of CPU-only,
CPU + L3, and CPU + Acc. We found that CPU + Acc
consumes much less power than CPU-only. On an average,
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHEN AND JHA: 3-D CPU-FPGA-DRAM HYBRID ARCHITECTURE FOR LOW-POWER COMPUTATION
CPU + Acc reduces power consumption by 47.5% relative
to CPU-only. By contrast, CPU + L3 consumes much higher
power than CPU-only and CPU + Acc. This is because the L3
cache of CPU + L3 consumes considerable amount of power,
even though we choose to use low-standby-power transistors
for memory cells in the L3 cache. On an average, CPU + Acc
reduces power consumption by 72.9% relative to CPU + L3.
The differences of power consumption lead to differences
in maxium temperatures, or hotspots, on the chip. Fig. 14(b)
shows comparisons of maximum temperatures. We can see that
the maximum temperature of CPU + Acc is much lower than
those of both CPU-only and CPU + L3. On an average, the
maximum temperature of CPU + Acc is 49.7 C, 6.4 C lower
than that of CPU-only (56.1 C), and 9.5 C lower than that of
CPU + L3 (59.2 C). This indicates that CPU + Acc does not
aggravate the thermal problem, which is a major problem in
the 3-D ICs. On the contrary, it helps to alleviate the thermal
problem.
VI. C ONCLUSION
In this paper, we proposed a 3-D CPU-FPGA-DRAM hybrid
architecture for low-power computation. The architecture fea-
tures an FPGA layer between the CPU and the DRAM layers.
The FPGA layer implements accelerators that work with CPU
cores to achieve low-power computation. This helps address
the dark silicon problem. We also take advantage of HLS
to make the proposed architecture easy to use and suitable
for general-purpose computation. To evaluate the architecture,
we created an evaluation framework and conducted various
experiments. At the function level, experimental results show
that the accelerators on the FPGA layer can reduce power and
EDP by 6.9 and 7.2, respectively, compared with an out-
of-order CPU while delivering similar performance. At the
application level, the accelerators on the FPGA layer can
reduce power and EDP by 1.9 and 2.2, respectively. For
the entire system, this translates to 47.5% power reduction
compared with the baseline system without the FPGA layer
and 72.9% power reduction compared with an alternative
system which has the FPGA layer replaced with an L3 cache
layer. These results demonstrate that the proposed architecture
is effective in achieving the power reduction goal.
R EFERENCES
[1] G. Venkatesh et al., Conservation cores: Reducing the energy of mature
computations, in Proc. 15th Ed. ASPLOS Archit. Support Int. Conf.
Support Program. Lang. Oper. Syst., 2010, pp. 205218.
[2] H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam, and
D. Burger, Dark silicon and the end of multicore scaling, in Proc.
38th Annu. Int. Symp. Comput. Archit., 2011, pp. 365376.
[3] S. Kestur, J. D. Davis, and O. Williams, BLAS comparison on FPGA,
CPU and GPU, in Proc. IEEE Comput. Soc. Annu. Symp. VLSI,
Jul. 2010, pp. 288293.
[4] D. Chen and D. Singh, Using OpenCL to evaluate the efficiency of
CPUs, GPUs and FPGAs for information filtering, in Proc. 22nd Int.
Conf. Field Program. Logic Appl., Aug. 2012, pp. 512.
[5] E. S. Chung, P. A. Milder, J. C. Hoe, and K. Mai, Single-chip
heterogeneous computing: Does the future include custom logic,
FPGAs, and GPGPUs? in Proc. 43rd Annu. IEEE/ACM Int. Symp.
Microarchitecture, Dec. 2010, pp. 225236.
[6] B. Betkaoui, D. B. Thomas, and W. Luk, Comparing performance and
energy efficiency of FPGAs and GPUs for high productivity computing,
in Proc. Int. Conf. Field-Program. Technol., Dec. 2010, pp. 94101.
13
[7] J. R. Hauser and J. Wawrzynek, Garp: A MIPS processor with
a reconfigurable coprocessor, in Proc. 5th Annu. IEEE Symp.
Field-Program. Custom Comput. Mach., Apr. 1997, pp. 1221.
[8] C. R. Rupp et al., The NAPA adaptive processing architecture, in Proc.
IEEE Symp. FPGAs Custom Comput. Mach., Apr. 1998, pp. 2837.
[9] S. C. Goldstein et al., PipeRench: A coprocessor for streaming
multimedia acceleration, in Proc. 26th Int. Symp. Comput. Archit.,
May 1999, pp. 2839.
[10] Z. A. Ye, A. Moshovos, S. Hauck, and P. Banerjee, CHIMAERA:
A high-performance architecture with a tightly-coupled reconfigurable
functional unit, in Proc. 27th Int. Symp. Comput. Archit., Jun. 2000,
pp. 225235.
[11] V. Govindaraju, C.-H. Ho, and K. Sankaralingam, Dynamically spe-
cialized datapaths for energy efficient computing, in Proc. IEEE 17th
Int. Symp. High Perform. Comput. Archit., Feb. 2011, pp. 503514.
[12] P. Bertin, D. Roncin, and J. Vuillemin, Introduction to programmable
active memories, Paris Res. Lab., Paris, France, Tech. Rep., pp. 18,
Jun. 1989.
[13] P. M. Athanas and H. F. Silverman, Processor reconfiguration through
instruction-set metamorphosis, Computer, vol. 26, no. 3, pp. 1118,
Mar. 1993.
[14] Cray, Inc. (2004). Cray XD1 Supercomputer. [Online]. Available:
http://www.cray.com
[15] C. Chang, J. Wawrzynek, and R. W. Brodersen, BEE2: A high-end
reconfigurable computing system, IEEE Des. Test Comput., vol. 22,
no. 2, pp. 114125, Mar./Apr. 2005.
[16] M. Lavasani, H. Angepat, and D. Chiou, An FPGA-based in-line
accelerator for Memcached, Comput. Archit. Lett., vol. 13, no. 2,
pp. 5760, Jul./Dec. 2014.
[17] J. L. Tripp, M. B. Gokhale, and K. D. Peterson, Trident: From
high-level language to hardware circuitry, Computer, vol. 40, no. 3,
pp. 2837, Mar. 2007.
[18] A. Putnam, D. Bennett, E. Dellinger, J. Mason, P. Sundararajan, and
S. Eggers, CHiMPS: A C-level compilation flow for hybrid CPU-FPGA
architectures, in Proc. Int. Conf. Field Program. Logic Appl., Sep. 2008,
pp. 173178.
[19] J. Cong, B. Liu, S. Neuendorffer, J. Noguera, K. Vissers, and Z. Zhang,
High-level synthesis for FPGAs: From prototyping to deployment,
IEEE Trans. Comput.-Aided Design Integr. Circuits Syst., vol. 30, no. 4,
pp. 473491, Apr. 2011.
[20] A. Canis et al., LegUp: An open-source high-level synthesis tool
for FPGA-based processor/accelerator systems, ACM Trans. Embedded
Comput. Syst., vol. 13, no. 2, Sep. 2013, Art. ID 24.
[21] Calypto Design Systems. Catapult. [Online]. Available: http://calypto.
com/en/products/catapult/overview, accessed Mar. 2015.
[22] Cadence, Inc. C-to-Silicon Compiler. [Online]. Available: http://www.
cadence.com/products/sd/silicon_compiler, accessed Mar. 2015.
[23] A. Papakonstantinou, K. Gururaj, J. A. Stratton, D. Chen, J. Cong, and
W.-M. W. Hwu, FCUDA: Enabling efficient compilation of CUDA
kernels onto FPGAs, in Proc. IEEE 7th Symp. Appl. Specific Processors,
Jul. 2009, pp. 3542.
[24] Altera Corporation. (Nov. 2012). Implementing FPGA Design With
the OpenCL Standard. [Online]. Available: http://www.altera.com/
literature/wp/wp-01173-opencl.pdf
[25] Xilinx, Inc. (2014). Vivado Design Suite System Edition
(Version 2014.1). [Online]. Available: http://www.xilinx.com
[26] Berkeley Design Technology, Inc. (2010). An Independent Evaluation of:
The AutoESL AutoPilot High-Level Synthesis Tool. [Online]. Available:
http://www.bdti.com/MyBDTI/pubs/AutoPilot.pdf
[27] C. C. Liu, I. Ganusov, M. Burtscher, and S. Tiwari, Bridging the
processor-memory performance gap with 3D IC technology, IEEE Des.
Test Comput., vol. 22, no. 6, pp. 556564, Nov./Dec. 2005.
[28] G. H. Loh, 3D-stacked memory architectures for multi-core proces-
sors, in Proc. 35th Int. Symp. Comput. Archit., Jun. 2008, pp. 453464.
[29] D. H. Woo, N. H. Seong, D. L. Lewis, and H.-H. S. Lee, An optimized
3D-stacked memory architecture by exploiting excessive, high-density
TSV bandwidth, in Proc. IEEE 16th Int. Symp. High Perform. Comput.
Archit., Jan. 2010, pp. 112.
[30] J. Zhao, X. Dong, and Y. Xie, An energy-efficient 3D CMP design
with fine-grained voltage scaling, in Proc. IEEE Design, Autom., Test
Europe Conf., Mar. 2011, pp. 14.
[31] J. Meng, K. Kawakami, and A. K. Coskun, Optimizing energy effi-
ciency of 3-D multicore systems with stacked DRAM under power and
thermal constraints, in Proc. 49th ACM/EDAC/IEEE Design Autom.
Conf., Jun. 2012, pp. 648655.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
14
[32] G. Sun, H. Yang, and Y. Xie, Performance/thermal-aware design of
3D-stacked L2 caches for CMPs, ACM Trans. Design Autom. Electron.
Syst., vol. 17, no. 2, Apr. 2012, Art. ID 13.
[33] B. Black et al., Die stacking (3D) microarchitecture, in Proc. 39th
Annu. IEEE/ACM Int. Symp. Microarchitecture, Dec. 2006, pp. 469479.
[34] R. Brillu, S. Pillement, A. Abdellah, F. Lemonnier, and P. Millet,
FlexTiles: A globally homogeneous but locally heterogeneous many-
core architecture, in Proc. 6th Workshop Rapid Simul. Perform. Eval.,
Methods Tools, 2014, Art. ID 3.
[35] Xilinx, Inc. (May 2014). Vivado Design Suite User Guide: High-Level
Synthesis. [Online]. Available: http://www.xilinx.com
[36] F. Lemonnier et al., Towards future adaptive multiprocessor systems-
on-chip: An innovative approach for flexible architectures, in Proc. Int.
Conf. Embedded Comput. Syst., Jul. 2012, pp. 228235.
[37] J. Choi, K. Nam, A. Canis, J. Anderson, S. Brown, and T. Czajkowski,
Impact of cache architecture and interface on performance and
area of FPGA-based processor/parallel-accelerator systems, in Proc.
IEEE 20th Annu. Int. Symp. Field-Program. Custom Comput. Mach.,
Apr./May 2012, pp. 1724.
[38] A. Putnam et al., Performance and power of cache-based reconfigurable
computing, in Proc. 36th Annu. Int. Symp. Comput. Archit., 2009,
pp. 395405.
[39] A.-C. Hsieh and T. Hwang, TSV redundancy: Architecture and design
issues in 3-D IC, IEEE Trans. Very Large Scale Integr. (VLSI) Syst.,
vol. 20, no. 4, pp. 711722, Apr. 2012.
[40] H.-H. S. Lee and K. Chakrabarty, Test challenges for 3D integrated
circuits, IEEE Des. Test, vol. 26, no. 5, pp. 2635, Sep. 2009.
[41] D. L. Lewis and H.-H. S. Lee, A scan-island based design enabling
pre-bond testability in die-stacked microprocessors, in Proc. IEEE Int.
Test Conf., Oct. 2007, pp. 18.
[42] X. Dong, J. Zhao, and Y. Xie, Fabrication cost analysis and cost-
aware design space exploration for 3-D ICs, IEEE Trans. Comput.-
Aided Design Integr. Circuits Syst., vol. 29, no. 12, pp. 19591972,
Dec. 2010.
[43] D. Z. Pan et al., Design for manufacturability and reliability for
TSV-based 3D ICs, in Proc. IEEE 17th Asia South Pacific Design
Autom. Conf., Jan./Feb. 2012, pp. 750755.
[44] N. Binkert et al., The gem5 simulator, ACM SIGARCH Comput. Archit.
News, vol. 39, no. 2, pp. 17, Aug. 2011.
[45] M. Yu, J. Song, F. Fu, S. Sun, and B. Liu, A fast timing-accurate
MPSoC HW/SW co-simulation platform based on a novel synchroniza-
tion scheme, in Proc. Int. MultiConf. Eng. Comput. Sci., Mar. 2010,
pp. 15.
[46] S. Li, J. H. Ahn, R. D. Strong, J. B. Brockman, D. M. Tullsen, and
N. P. Jouppi, McPAT: An integrated power, area, and timing modeling
framework for multicore and manycore architectures, in Proc. 42nd
Annu. IEEE/ACM Int. Symp. Microarchitecture, Dec. 2009, pp. 469480.
[47] Micron Technology, Inc. DDR3 SDRAM System-Power Calculator.
[Online]. Available: http://www.micron.com, accessed Mar. 2015.
[48] K. Skadron, M. R. Stan, W. Huang, S. Velusamy, K. Sankaranarayanan,
and D. Tarjan, Temperature-aware microarchitecture, in Proc. 30th
Annu. Int. Symp. Comput. Archit., Jun. 2003, pp. 213.
[49] Micron Technology, Inc. 1GB DDR3 SDRAM Datasheet. [Online].
Available: http://www.micron.com, accessed Mar. 2015
[50] N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, Architecting
efficient interconnects for large caches with CACTI 6.0, IEEE Micro,
vol. 28, no. 1, pp. 6979, Jan./Feb. 2008.
[51] International Roadmap Committee. (2012). International Technology
Roadmap for Semiconductors (2012 Update). [Online]. Available:
http://www.itrs.net
[52] D. H. Kim, S. Mukhopadhyay, and S. K. Lim, Fast and accurate
analytical modeling of through-silicon-via capacitive coupling, IEEE
Trans. Compon., Packag., Manuf. Technol., vol. 1, no. 2, pp. 168180,
Feb. 2011.
[53] Xilinx, Inc. (Mar. 2014). 7 Series FPGAs Packaging and Pinout (v1.11).
[Online]. Available: http://www.xilinx.com
[54] Semiconductor Industries Association. (2008). International Technology
Roadmap for Semiconductors (2008 Update). [Online]. Available:
http://http://www.itrs.net
[55] Z. Chishti and T. N. Vijaykumar, Optimal power/performance pipeline
depth for SMT in scaled technologies, IEEE Trans. Comput., vol. 57,
no. 1, pp. 6981, Jan. 2008.
[56] Y. Hara, H. Tomiyama, S. Honda, and H. Takada, Proposal and quan-
titative analysis of the CHStone benchmark program suite for practical
C-based high-level synthesis, J. Inf. Process., vol. 17, pp. 242254,
Oct. 2009.
IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
[57] C. Bienia, Benchmarking modern multiprocessors, Ph.D. dissertation,
Dept. Comput. Sci., Princeton Univ., Princeton, NJ, USA, Jan. 2011.
[58] M. Gebhart, J. Hestness, E. Fatehi, P. Gratz, and S. W. Keckler, Running
PARSEC 2.1 on M5, Dept. Comput. Sci., Univ. Texas Austin, Austin,
TX, USA, Tech. Rep. TR-09-32, Oct. 2009.
[59] R. Miftakhutdinov, E. Ebrahimi, and Y. N. Patt, Predicting performance
impact of DVFS for realistic memory systems, in Proc. 45th Annu.
IEEE/ACM Int. Symp. Microarchitecture, Dec. 2012, pp. 155165.
Xianmin Chen received the B.S. degree in informa-
tion engineering from Shanghai Jiao Tong Univer-
sity, Shanghai, China, in 2007, and the M.S. degrees
from Shanghai Jiao Tong University and the Georgia
Institute of Technology, Atlanta, GA, USA, in 2010.
He is currently pursuing the Ph.D. degree in com-
puter engineering with Princeton University, Prince-
ton, NJ, USA. His current research interests include
FinFET-based system design, reconfigurable com-
puting, 3-D integrated circuit, and network-on-chip.
Niraj K. Jha (S85M85SM93F98) received
the B.Tech. degree in electronics and electrical
communication engineering from IIT Kharagpur,
Kharagpur, India, in 1981, the M.S. degree in
electrical engineering from SUNY at Stony Brook,
Stony Brook, NY, USA, in 1982, and the
Ph.D. degree in electrical engineering from the
University of Illinois at UrbanaChampaign,
Champaign, IL, USA, in 1985.
He is currently a Professor of Electrical
Engineering with Princeton University, Princeton,
NJ, USA. He has co-authored and co-edited five books entitled Testing and
Reliable Design of CMOS Circuits (Kluwer, 1990), High-Level Power Analysis
and Optimization (Kluwer, 1998), Testing of Digital Systems (Cambridge
University Press, 2003), Switching and Finite Automata TheoryThird Edition
(Cambridge University Press, 2009), and Nanoelectronic Circuit Design
(Springer, 2010). He has authored 15 book chapters. He has authored or
co-authored over 400 technical papers. He holds 14 U.S. patents. His current
research interests include FinFETs, low power hardware/software design,
computer-aided design of integrated circuits and systems, digital system
testing, quantum computing, and secure computing.
Prof. Jha is a fellow of the Association for Computing Machinery.
He received the Distinguished Alumnus Award from IIT Kharagpur. He
is a recipient of the AT&T Foundation Award and the NEC Preceptorship
Award for research excellence, the NCR Award for teaching excellence, and
the Princeton University Graduate Mentoring Award. A paper of his was
selected for The Best of ICCAD: A collection of the best IEEE International
Conference on Computer-Aided Design papers of the past 20 years, two
papers by the IEEE Micro Magazine as one of the top picks from the 2005 and
2007 Computer Architecture Conferences, and two others as being among the
most influential papers of the last ten years at the IEEE Design Automation
and Test in Europe Conference. He has received nine other best paper awards
and six best paper award nominations. He has served as the Editor-in-Chief
of the IEEE T RANSACTIONS ON VLSI S YSTEMS and an Associate Editor
of the IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMS I AND II, the
IEEE T RANSACTIONS ON VLSI S YSTEMS , the IEEE T RANSACTIONS
ON C OMPUTER -A IDED D ESIGN , and the Journal of Electronic Testing:
Theory and Applications. He serves as an Associate Editor of the IEEE
T RANSACTIONS ON C OMPUTERS , the Journal of Low Power Electronics,
and the Journal of Nanotechnology. He served as the Program Chairman
of the 1992 Workshop on Fault-Tolerant Parallel and Distributed Systems,
the 2004 International Conference on Embedded and Ubiquitous Computing,
and the 2010 International Conference on VLSI Design. He has served as the
Director of the Center for Embedded System-on-a-Chip Design funded by
the New Jersey Commission on Science and Technology. He has served on
the program committees of more than 150 conferences and workshops. He has
given several keynote speeches in the area of nanoelectronic design and test.