IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 30, NO. 12, DECEMBER 2011 1883
Energy-Efficient Multiobjective Thermal Control
for Liquid-Cooled 3-D Stacked Architectures
Mohamed M. Sabry, Student Member, IEEE, Ayse K. Coskun, Member, IEEE, David Atienza, Member, IEEE,
Tajana
ˇ
Simuni
´
c Rosing, Member, IEEE, and Thomas Brunschwiler, Member, IEEE
Abstract—3-D stacked systems reduce communication delay in
multiprocessor system-on-chips (MPSoCs) and enable heteroge-
neous integration of cores, memories, sensors, and RF devices.
However, vertical integration of layers exacerbates temperature-
induced problems such as reliability degradation. Liquid cooling
is a highly efficient solution to overcome the accelerated thermal
problems in 3-D architectures; however, it brings new challenges
in modeling and run-time management for such 3-D MPSoCs
with multitier liquid cooling. This paper proposes a novel design-
time/run-time thermal management strategy. The design-time
phase involves a rigorous thermal impact analysis of various
thermal control variables. We then utilize this analysis to design
a run-time fuzzy controller for improving energy efficiency in 3-D
MPSoCs through liquid cooling management and dynamic volt-
age and frequency scaling (DVFS). The fuzzy controller adjusts
the liquid flow rate dynamically to match the cooling demand
of the chip for preventing overcooling and for maintaining a
stable thermal profile. The DVFS decisions increase chip-level
energy savings and help balance the temperature across the
system. Our controller is used in conjunction with temperature-
aware load balancing and dynamic power management strategies.
Experimental results on 2-tier and 4-tier 3-D MPSoCs show
that our strategy prevents the system from exceeding the given
threshold temperature. At the same time, we reduce cooling
energy by up to 63% and system-level energy by up to 21%
in comparison to statically setting a flow rate setting to handle
worst-case temperatures.
Index Terms—3-D integration, liquid cooling, multiprocessor
SoC (MPSoC), thermal management.
I. Introduction
3-D
INTEGRATION is a recently proposed design
method to overcome the limitations with respect
Manuscript received May 19, 2011; revised July 19, 2011; accepted July 21,
2011. Date of current version November 18, 2011. This work was supported
in part by the PRO3D EU FP7-ICT-248776 Project, and in part by the Nano-
Tera.ch RTD Project CMOSAIC (ref. 123618), which is financed by the
Swiss Confederation and scientifically evaluated by Swiss National Science
Foundation. This paper was recommended by Associate Editor Y. Xie.
M. M. Sabry and D. Atienza are with the Embedded Systems Laboratory,
École Polytechnique F
´
ed
´
erale de Lausanne, Lausanne 1015, Switzerland (e-
mail: mohamed.sabry@epfl.ch; david.atienza@epfl.ch).
A. K. Coskun is with the Department of Electrical and Computer Engineer-
ing, Boston University, Boston, MA 02215 USA (e-mail: acoskun@bu.edu).
T. S. Rosing is with the Department of Computer Science and Engineering,
University of California at San Diego, San Diego, CA 92093 USA (e-mail:
tajana@ucsd.edu).
T. Brunschwiler is with IBM Research GmbH, Zurich Research Laboratory,
Zurich 8803, Switzerland (e-mail: tbr@zurich.ibm.com).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TCAD.2011.2164540
to the delay, bandwidth, and power consumption of the in-
terconnects in large multicore chips, while reducing the chip
footprint and improving the fabrication yield. However, high
temperatures resulting from higher thermal resistivity [17],
[28] are among the main challenges for designing 3-D multi-
processor system-on-chips (MPSoCs). In addition, it is more
difficult to remove the heat from 3-D MPSoCs as cores may be
located at different tiers and have significantly different heat-
ing/cooling rates in comparison to conventional 2-D chips [9].
3-D MPSoCs are also prone to larger thermal variations, which
have adverse effects on system reliability, performance, and
cooling costs [11].
A number of thermal management techniques have been
proposed for controlling temperature on 2-D (single-tier)
MPSoCs. Dynamic voltage and frequency scaling (DVFS)
and thread migration/scheduling based on thermal feedback
are examples of such techniques [14]. Recent research has
extended 2-D management techniques for workload scheduling
and DVFS-based thermal management in 3-D MPSoCs [9],
[44], [45]. However, as power densities, number of cores, and
number of tiers increase, extremely high temperature values
appear in 3-D stacks [44], resulting in severe restrictions in
high-performance 3-D MPSoC design.
Interlayer liquid cooling is an attractive solution to address
the high temperatures in 3-D chips, due to the higher heat
removal capability of liquids in comparison to air [7], [10].
However, we need to integrate liquid cooling management with
task scheduling and DVFS to maximize the energy efficiency
and reliability of high-performance 3-D MPSoCs. In addition,
previous work has shown that as workload dynamics change
at run-time, choosing the flow rate setting dynamically to meet
the cooling demand saves significant energy [10].
Combining various control knobs in a single low-overhead
optimum controller is a highly challenging task, as the control
parameters differ in their time constants, performance/energy
overheads, and benefits. For example, DVFS has an overhead
in the order of tens to hundreds of microseconds, while
flow rate changes may take hundreds of milliseconds. Hence,
simultaneously utilizing such distinct control knobs at run-
time requires a thorough understanding of their impact and
interactions.
In this paper, we propose a combined design-time/run-time
thermal management strategy for 3-D MPSoCs to interpret
current system state (temperature, power, and workload) with a
degree of uncertainty and flexibility. In particular, we advance
0278-0070/$26.00
c
2011 IEEE
1884 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 30, NO. 12, DECEMBER 2011
the state-of-the-art on dynamic thermal management (DTM)
for 3-D MPSoCs in the following directions.
1) We perform a thorough design-time thermal impact
analysis of various DTM methods (i.e., flow rate control,
DVFS, and task scheduling/migration) in 3-D MPSoCs
with interlayer liquid cooling to identify a set of optimal
decisions in achieving energy-efficient thermal manage-
ment with minimal performance degradation.
2) We utilize this study to design a fuzzy controller that
extends previous work [30] by including a complete
stability and implementation complexity analysis. The
controller is able to trigger appropriate flow rate and
DVFS adjustments based on the system’s temperature,
workload requirements, and spatial location of the com-
putational units and caches.
3) We analyze the benefits of the state-of-the-art tempera-
ture-aware job scheduling methods [9], [44] in our DTM
scheme for 3-D MPSoC with liquid cooling. We propose
a novel job scheduler which takes the physical locations
of the units in the 3-D stack into consideration to
stabilize the temperature on the die and to improve
cooling efficiency without affecting performance.
4) We provide extensive experimental evaluation on 2-
tier and 4-tier 3-D MPSoCs by comparing our thermal
management approach with respect to state-of-the-art
DTM techniques [10], [30], [44] for a large number of
metrics, such as peak temperatures, thermal gradients,
energy efficiency, and performance degradation. These
experiments show that our strategy completely removes
thermal hot spots in the 3-D stacks, while saving cooling
energy by up to 63% and 45%, and system-level energy
by up to 21% and 18%, in comparison to using a static
worst-case flow rate setting and in comparison to using
a look-up table-based control [10], respectively.
5) We analyze how core and cache temperatures in different
locations of the 3-D MPSoC are affected by the liquid
flow to reduce the pumping power and the thermal
gradients in the 3-D stack.
The rest of this paper starts with an overview of the prior
work in Section II. Then, Section III describes our developed
thermal model for liquid-cooled 3-D MPSoCs. Next, we de-
scribe the proposed design-time/run-time thermal management
strategy in Section IV. In Section V, we explore the thermal
impact analysis of various thermal control knobs on the 3-
D MPSoCs temperature. Section VI describes our new fuzzy
controller for DTM in 3-D MPSoCs with liquid cooling, and
we present the experimental results in Section VII. Finally,
Section VIII summarizes the main conclusions of this paper.
II. Related Work
A. Accurate and Compact Thermal Modeling of 2-D/3-D ICs
Accurate thermal modeling is critical in system design
and evaluation. HotSpot [33] is a R-C network-based ther-
mal model that calculates transient temperature response for
a given power trace. To reduce the potentially long ther-
mal simulation time in HotSpot for large MPSoCs, recent
work proposes a thermal emulation framework using field-
programmable gate arrays [2]. Latest versions of HotSpot
include 3-D modeling, and methods to extend HotSpot for
liquid-cooled 3-D MPSoCs are available [8]. 3-D interlayer
cooling emulator (3D-ICE) [34] is a new thermal modeling
tool specifically designed for transient thermal analysis of 3-D
stacks, including interlayer liquid cooling modeling. The au-
thors showed that their modeling and simulation framework
can be extended to account for different cavity structures, such
as pin-fins [35]. Feng et al. [15] introduced a thermal simu-
lation framework of 3-D stacks where graphical processing
units are used for accelerating temperature calculation. All
these methods to model and speedup 3-D MPSoC thermal
simulations are complementary to our paper.
B. Interlayer Liquid Cooling
The use of convection in microchannels to cool down high
power density chips has been an active area of research
since the initial work by Tuckerman and Pease [41]. Their
liquid cooling system can remove 1000 W/cm
2
; however, the
volumetric flow rate and the pressure drop are too large
for practical applications in 3-D integrated circuits (ICs).
Recent work shows that back-side liquid cold plates, i.e.,
staggered microchannel and distributed return jet plates, can
handle up to 400 W/cm
2
in single-chip applications [6]. The
heat removal capability of interlayer heat-transfer with pin-fin
inline structures for 3-D chips has been also investigated [7],
[18]. At a chip size of 1 cm
2
and maximal difference between
junction and liquid cooling temperatures of 60 K, the heat-
removal performance is more than 200 W/cm
2
at interconnect
pitches larger than 50 μm. However, research in liquid cooling
has typically developed thermal packaging solutions rather
than utilizing active cooling in DTM, which is the focus of
our paper.
C. DTM of 3-D MPSoCs
Prior work on thermal management for 3-D MPSoCs mainly
addresses design-time optimization, such as thermally aware
floorplanning [16], integrating thermal via planning in the
3-D floorplanning process [22], and joint optimization that
targets temperature, power interconnect, and signal wires [20].
A tradeoff study in recent work compares thermal behavior
and interconnect congestion for two 3-D MPSoC cooling
technologies: inter-tier liquid cooling and thermal through-
silicon-vias (TSVs) [20]. This paper shows that inter-tier liquid
cooling has superior cooling abilities, but induces limitations
for TSVs and increases cost with respect to using thermal
TSVs only.
Recent work considers DTM for 3-D MPSoCs. Zhu et al.
[45] evaluated several policies for task migration and DVFS by
exploring thermal profiles of adjacent processing cores on
the same vertical column (interlayer adjacent) or within the
same layer (intralayer). Zhou et al. [44] integrated a thermally
aware task scheduler with DVFS on a 2-tier system with
eight cores. A recent paper proposed a temperature-aware
scheduling method specifically designed for air-cooled (AC) 3-
D systems [9]. Their method considered thermal heterogeneity
among the 3-D MPSoC layers; however, it does not study
interlayer cooling. Prior work on DTM in AC 3-D systems
SABRY et al.: ENERGY-EFFICIENT MULTIOBJECTIVE THERMAL CONTROL FOR LIQUID-COOLED 3-D STACKED ARCHITECTURES 1885
demonstrated very high temperatures (85–120 °C), motivating
the search for alternative energy-efficient cooling techniques
beyond conventional methods.
Prior liquid cooling work [8] evaluated existing thermal
management policies on a 3-D MPSoC with a fixed-flow
rate value, and the benefits of using a policy to increment/
decrement the flow rate based on temperature measurements.
Our recent paper [10] considered the energy efficiency of 3-
D MPSoCs with variable flow rate adjustment and thermally
aware load balancing, without utilizing DVFS for increased
energy savings. Recently, Qian et al. [29] explored the use of
a cyber-physical approach to manage the temperature of 3-D
MPSoCs with inter-tier liquid cooling. They construct their
control mechanism with software-based thermal estimation
and prediction. They use a nonuniform liquid flow in different
microchannels, to meet the cooling demands of different
modules.
This paper brings several major contributions over prior
work. First, we study the thermal impact of various thermal
management methods (flow rate control, DVFS, and task
scheduling/migration) in 3-D MPSoCs to identify a set of
optimal decisions in achieving energy-efficient thermal man-
agement with minimal performance degradation. Second, we
utilize this study in designing a fuzzy controller that generates
the appropriate control decisions, as well as discussing the
controller’s use in conjunction with a novel job scheduler to
further improve the cooling efficiency without affecting perfor-
mance. Finally, we demonstrate how design-time optimization
can help improve energy efficiency in 3-D systems with liquid
cooling.
III. Thermal Modeling Framework for
Liquid-Cooled 3-D MPSoCs
Modeling the temperature dynamics of liquid-cooled
3-D MPSoC architectures consists of forming the grid-level
thermal R-C network model of the whole stack, modeling
the TSVs and the microchannels [34], [45], and modeling the
impact of the pump and coolant flow rate.
Fig. 1 shows the 3-D MPSoCs designs we target in this
paper. These 3-D MPSoCs consist of two or more stacked
tiers (i.e., tiers A and B include cores, L2 caches, crossbar,
and other units for memory control and buffering), and
microchannels are built in between the vertical layers for
liquid flow (tier C). In this paper, we assume face-to-back
bonding, and we assume forced convective interlayer cooling
with single-phase fluids, in particular water [7]. TSVs are
located and etched within the microchannel walls, and
uniformly distributed in TSV bundles as in layout C in Fig. 1.
Thus, the channel wall dimensions take the TSV features
and spacing requirements into account. We use uniformly
distributed microchannels, and equivalent fluid flow rate
through each channel in the same layer. Although variation
of the fluid flow due to nonuniform heat flux [18] can exist,
we examine the impact of this flow through simulations and
show that variations do not exceed 2% for single-phase flows
(see Section V). Finally, the liquid flow rate provided by the
pump can be dynamically altered at run-time [6], [7].
Fig. 1. Layouts of the modeled 3-D MPSoCs using Sun T1 SPARC cores.
Fig. 2. Cross-section of the (a) 3-D layers and the (b) resistive network [10].
A. Grid-Level Thermal Model for Liquid-Cooled 3-D MPSoCs
Similar to thermal modeling in 2-D chips [1], [33], 3-D
thermal modeling is performed using a discrete thermal model
that forms the R-C circuit for given grid dimensions. In this
paper, we utilize 3D-ICE [34] to perform the transient thermal
simulation of 3-D MPSoCs with interlayer liquid cooling.
To model the heterogeneous characteristics of the interlayer
material including run-time variable flow rate in microchan-
nels, we introduce the ability to change the resistivity value
of the microchannels cells, based on the liquid flow rate, at
run-time. However, the thermal resistance of the channel walls
(with the TSVs) are fixed to a specific value.
We compute the local junction temperature of 3-D MPSoCs
with liquid cooling using the resistive network of Fig. 2
for each layer, where heat is dissipated to both opposing
vertical directions (up and down) from the heat sources. The
top and bottom layers’ temperatures are denoted by T
S1
and
T
S2
, respectively. Then, the thermal resistance of the back-
end-of-line (BEOL) layers (R
BEOL
), the silicon slab thermal
resistance (R
slab
), and the convective thermal resistance (R
conv
)
are combined to model the 3-D stack, while the heat flux
values (˙q) represent the heat sources. We compute the junction
temperature (T
j
= T
S1
in Fig. 2) by assuming isothermal chan-
nel walls (i.e., the top and bottom parts of the microchannel
have the same temperature).
Table I lists all the parameters used in our thermal model,
which are based on experimentally validated liquid cooling
technology [7]. Note that the provided flow rate (
˙
V ) range is
for the interlayer cavity, and it is equally divided between all
the microchannels.
1886 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 30, NO. 12, DECEMBER 2011
TABLE I
Parameters for Computing (1)
Parameter Definition
Value
R
th−BEOL
Thermal resistance (3)
of wiring levels
t
B
BEOL thickness (Fig. 2) 12μm
k
BEOL
Conductivity of wiring levels 2.25 W/(m·K)
R
th−heat
Effective thermal resistance (5)
A
heater
Heater area Area of grid cell
c
p
Coolant heat capacity 4183 J/(kg·K)
ρ Coolant density 998 kg/m
3
˙
V Volumetric flow rate per cavity 0.01–0.0323 l/min
h Heat-transfer coefficient 371323 W/(m
2
·K)
w
c
Channel width (Fig. 2) 50 μm
w
w
Wall width (Fig. 2)
100 μm
t
c
Channel thickness (Fig. 2) 100 μm
t
s
Silicon slab thickness (Fig. 2) 50 μm
p
Channel pitch (Fig. 2) 150 μm
The junction temperature change (shown in Fig. 2) is a sum
of the following three components: 1) the thermal gradient due
to conduction (T
cond
); 2) the change in coolant temperature,
which increases linearly with position along the channel due
to heat absorption (T
heat
); and 3) the convective (T
conv
)
portion, which is independent of the flow rate with fully
developed hydrodynamic and thermal boundary layers that
have been reached [7]. Hence, the total temperature rise on
the junction (T
j
) can be computed as follows:
T
j
= T
cond
+ T
heat
+ T
conv
. (1)
The thermal gradient due to heat conduction through BEOL
layer (T
cond
) can be computed using the definitions and
values of t
B
and k
BEOL
in Table I as follows:
T
cond
= R
th−BEOL
·˙q
1
(2)
R
th−BEOL
=
t
B
k
BEOL
. (3)
The temperature change between two adjacent cells due to
heat absorption (T
heat
) is computed as follows:
T
heat
=(˙q
1
+ ˙q
2
) · R
th−heat
(4)
R
th−heat
=
A
heater
c
p
· ρ ·
˙
V
. (5)
Finally, (6) and (7) show how to calculate the temperature
change due to convection (T
conv
). The heat-transfer coef-
ficient (h) is dependent on the hydraulic diameter, Nusselt
number, and conductivity of the fluid [7]. As the effective
heat-transfer coefficient (h
eff
) is not affected by flow rate
changes with developed boundary layers, we compute this
parameter prior to simulation and use it during the experiments
(Section VII). Fig. 2 demonstrates w
c
, t
c
, and p, and Table I
shows their definitions and values.
T
conv
=(˙q
1
+ ˙q
2
) · h
eff
(6)
h
eff
= h
2 · (w
c
+ t
c
)
p
. (7)
In our target 3-D MPSoCs, we use copper TSVs with 50 μm
diameter and 150 μm pitch. These particular dimensions have
been used in 3-D ICs with interlayer cooling in previous
work [3], [19]. Considering the dimensions and pitch require-
ments of microchannels and TSVs, there are 66 microchannels
in each cavity (tier C), and each cavity is located between
every two silicon tiers (A or B). Thus, there are 66 and
TABLE II
Parameters Definition Used in Relating Flow Rate to the
Applied Pressure Difference
Parameter
Definition
ε
Cavity porosity (0.33)
κ
Cavity permeability (7.17E-11 m
2
)
μ Dynamic viscosity (1E-3 Pascal· s at 300 K)
P Pressure difference between the inlet and outlet ports (1 bar)
L Channel length
198 microchannels in the 2-tier and 4-tier 3-D MPSoCs,
respectively. Based on previous work [3], [8], we assign a
uniform TSV density for the interlayer material.
B. Modeling the Pump and Liquid Flow Rate
All the microchannels are connected to a pump to receive
the coolant. The pump injects the fluid at a certain pressure dif-
ference required by the system. In accordance with prior work
on liquid cooling technology validation for 3-D stacks [7], we
fix the maximum applied pressure on the stack to 1 bar. The
flow velocity within the microchannels is governed by:
ν
bulk
=
ν
darcy
ε
(8)
ν
darcy
=
κ
μ
·∇P (9)
∇P =
P
L
(10)
where ν
bulk
is the actual velocity in the channel, and the
other symbols are defined in Table II. These equations show
that the fluid velocity is dependent on the applied pressure
difference and on the fluid temperature via the dynamic
viscosity. In particular, if the fluid temperature increases
the dynamic viscosity decreases, and the fluid moves at a
higher speed. However, in our experiments, we have observed
that the change in the fluid temperature with nonuniform
heat fluxes does not exceed 10°, which in turn negligibly
affects the dynamic viscosity. Therefore, we assume that
the fluid velocity remains constant at a constant pressure
gradient.
In a high-performance computing cluster, several chips are
included in a set of racks and a central pump must be used
for several 3-D stacks for reducing the cost [27]. Hence, a
centrifugal pump EMB MHIE [31] is responsible for the fluid
injection to a cluster of nodes via a pumping network. This
pump has the capability of producing large discharge rates
at small pressure heads. To enable different flow rates for
each stack, the cooling infrastructure includes valves in the
network. We assume normally closed valves (NCVs) provided
by the Festo group [32]. NCVs use external power to reduce
the pressure drop and to increase the flow rate. Fig. 3 shows
the pump and valve power consumption for three flow rate
settings of a single stack deployed in a cluster with 60
similar 3-D computing stacks, as proposed in the Aquasar
data-center design [27]. The maximum energy required to
inject the fluid to all stacks is approximately 180 W, which
is a significant overhead to the whole system as this value is
similar to the energy consumed by a 4-tier 3-D chip. Thus,
the energy consumed in the liquid cooling subsystem should
be minimized.
SABRY et al.: ENERGY-EFFICIENT MULTIOBJECTIVE THERMAL CONTROL FOR LIQUID-COOLED 3-D STACKED ARCHITECTURES 1887
Fig. 3. Power consumption and flow rates of the cooling infrastructure per
one stack in a 60 3-D chips cluster [27].
Fig. 4. Schematic diagram of the combined design-time/run-time thermal
management scheme.
IV. Combined Design-Time/Run-Time Thermal
Management Strategy
Designing an efficient multiobjective run-time thermal man-
agement technique for liquid-cooled 3-D systems requires
the study of the target system at design-time. Fig. 4 shows
a schematic diagram of our combined design-time/run-time
thermal management scheme. It starts at design-time (see Sec-
tion V) with the identification of the different control knobs,
along with the definition of the ranges of values these knobs
can have. Hence, our choices on the controlled parameters are
highly flexible, unlike various assumptions taken in previous
works [10], [43]. After the identification of control knobs,
we perform a superposition-based thermal impact analysis of
each knob to quantify its individual thermal impact on the
whole system and to develop a suitable multiobjective run-
time thermal scheme (Section VI). In particular, we identify
the following thermal control parameters to be explored for
DTM in 3-D MPSoCs.
A. Dynamic Voltage and Frequency Scaling
This technique has been used for 2-D/3-D MPSoCs DTM.
DVFS has a fast response time (μs range, 100–200 μs) with
respect to the other DTM techniques, but it can also imply a
significant performance overhead [11].
B. Task Scheduling and/or Migration
Job scheduling is an effective tool for reducing and balanc-
ing MPSoC temperature [9], [44], [45]. This technique has
a lower control decision frequency (fewer control actions per
unit time) with respect to DVFS, as it relies on higher-level
OS-based decisions, which are taken on the order of tens of
milliseconds.
C. Variable Fluid Flow Rate
Interlayer liquid cooling plays a major role in DTM of 3-
D stacked MPSoCs [10], [23]. Applying variable flow rates
changes the interlayer thermal resistance and creates the
effect of having intermediate heat sinks in between tiers in
3-D architectures [6], [7] without performance degradation.
However, interlayer liquid cooling has a large response time
as it relies on mechanical changes of the pumping network.
After performing the design-time thermal analysis, we de-
rive a set of rules that are used in run-time thermal man-
agement. We use fuzzy logic to derive the run-time control
actions. Although the same approach can be adapted to use
different cyber-physical controllers, we opt to use rule-based
fuzzy controller instead of other linear multi-input multi-
output (MIMO) controllers [13] as with this technique, we
are able to achieve effective control with a straightforward,
low-complexity, and flexible implementation. Various low-
complexity techniques can be used for deriving the fuzzy
rules, such as offline analytical analysis and online learning
mechanisms [37], and fuzzy control can be implemented at the
software-level with low overhead, as we show in Section VII.
Moreover, fuzzy control operates efficiently at run-time with
inputs that have a degree of uncertainty in describing the
system state [26], which is the case in 3-D MPSoCs where
various inputs can be affected by a number of conditions
(e.g., ambient temperature changes, unexpected workloads,
temperature sensors inaccuracy, and stack degradation).
Overall, using fuzzy control for 3-D MPSoCs with interlayer
liquid cooling with time-varying liquid flow rate is a low-
cost yet effective approach to find an optimal solution [42],
in comparison to other MIMO control techniques for linear
time-varying systems.
V. Thermal Response Analysis in Liquid-Cooled
3-D MPSoCs
In this section, we analyze the thermal response and impact
of each of identified thermal control knobs (Section IV) on 3-
D MPSoC designs with respect to the worst case and typical
operating conditions. In the analysis of DVFS and varying flow
rate, we model an infinite thread input with full utilization.
The threads are executed on a variable number of active cores
in our 3-D test-bed, which is shown in Fig. 5(c). Our 5-
tier 3-D test chip prototype (presented in [7] and [34]) is
shown in Figs. 5(a) and (b). The 3-D MPSoC we use for the
analysis consists of two tiers, where each tier contains four
main hot spot sources modeling high performance processors.
The remaining area contains background heaters playing the
role of caches, interconnects, and other blocks. We have
experimentally validated the transient thermal response of our
test-bed using 3-D test chip stacks [Figs. 5(a) and (b)] with a
die area of 1 cm
2
. In this test-bed, the hot spot sources occupy
an area of 5 mm×2 mm and dissipate 250 W/cm
2
, while the
residual area dissipates 50 W/cm
2
.
A. Variant Liquid Flow Rate
We first examine the effect of changing the liquid flow
rate when the maximum temperature of the 3-D chip exceeds
a certain threshold. This experiment involves the dynamic
variation of the pumping flow rate (i.e., between 0.1–0.2 l/min)
[7] to observe the variation of the core temperature with
respect to the change in flow rate. We use five different flow
rate values: {0.1, 0.125, 0.15, 0.175, 0.2} l/min. Fig. 6 shows
