ANALOG IMPLEMENTATION OF THE SOFT-MAX FUNCTION
Rodolfo Zunino and Paolo Gastaldo
DIBE - Genoa University - Italy - Email: { zunino, gastaldo }@dibe.unige.it
ABSTRACT
The paper describes the analog implementation of the
S o M a x function by using CMOS circuitry. First, an
optimization-based strategy is presented, which fits
technology requirements to the desired accuracy in the
SoftMax-mapping. Then circuit solutions are described to
both the approximation of the exp(x) function and the
normalizing ratio. These optimized design aspects improve
overall effectiveness, reduce VLSI complexity by
exploiting inherent parallelism, and ultimately limit overall
power consumption.
1. INTRODUCTION
Soft-Max (SM) and Winner-Take-All (WTA) modules
111 provide the natural support for ranking and selection,
which are basic elements in modem systems for nonlinear
signal processing. SM is a usual operation in the modeling
of continuous phenomena [2-31, whereas WTA mainly fits
digital, discrete quantities and processes. Theory recently
proved the validity of these structures as universal function
approximators [4].
Hardware SM processing seems almost unexplored in
the literature. The approach proposed in [5] uses the
subthreshold operation of MOS transistors.
Approximating axp(x) [6] and accurate normalizing ratio
[7] are crucial aspects of SM VLSI design. Simplistic
approaches, plugging together exponential and ratio
circuits, prove ineffective in terms of mapping accuracy
and power consumption.
This paper approaches hardware SM processing as an
integrated problem; overall functional requirements drive
the design of individual submodules. All transistors in the
current-mode CMOS circuit operate in strong inversion.
The basic strategy approximates the exp(x) hnction by
a 2nd-degree polynomial to limit circuit complexity.
Imposing the SoftMax characteristics on the circuit
transfer function poses a quadratic-cost problem, within
the constraints of ranking consistency and discrimination
accuracy. Thus, the eventual MOS sizing results from an
optimization process, leaving the designer with a degree of
0-7803-7448-7/02/$17.00 02002 IEEE 11- 117
freedom to comply with technology specifications. The
design methodology adopts a modular approach, and VLSI
simulations confirm the satisfactory level of accuracy of
the circuit. Architectural features limit power consumption
in spite of the square-law MOST operation.
2. CIRCUIT DESIGN
The SoftMax function of a vector, Y, of n input scalar
variables: Y = {y,,...,yn}, is a vector, S, of n values
computed as
j=l
The (gain) parameter y rules the sharpness of the SM
mapping. Small gain values lead to a "smooth" SM
operation, which asymptotically supports a uniform
distribution: y=O 3 si(Y)=l/nVi. Large values of y
make the SM function tend to a truly WTA behaviour:
y- 3 q(Y) = 1, ify,= max{ yjs Y,j=l, ...n }, and s4Y) =
0 VYk # Yi.
The ability to tune the transfer function by means of
one quantity justifies the interest in SM for signal
processing (e.g., interfacing analog and digital
representations.) For practical purposes, in the following
we shall write -vi = xi and 0 I xi I XM Vi.
SM circuitry design is made complicated by two issues:
first, the dynamics of the exp(x) function shrinks the valid
input range; secondly, the normalizing ratio in (1) may
hinder accurate analog solutions. Our approach limits the
approximation of the exp(x) function to a 2nd-degree
polynomial, thus allowing an acceptable tradeoff between
representation accuracy and circuit complexity.
(. i
Therefore, the exp x) function is replaced with
f2(p)(x) = c . + @ -+ x2 . As the ratio in (1) leaves c as
a degree of freedom, the parameter set to be optimized is
P={ a,p}. The SM approximationis given by:
j=1
 Fig. 1 - Modular architecture of the SoftMax circuit
Consistent design looks for the set P that approximatesthe
SM mapping by minimizing the quadratic cost:
(3)
The above expression averages the (Euclidean) distortion
between the actual and approximating output vectors over
the probability distribution of input vectors X. The crucial
advantage of the analytical cost ( 3 ) over a direct fitting of
to exp(x) is that the approximation problem is set in terms
of global SM mapping behaviour.
Two constraints shape the cost function so as to
preserve consistency with the SM mapping: first, f2(p)(x)
must be monotonically increasing in the range [0, x ~ ] :
Resolution-accuracy requirements in (2) set the second
constraint, which imposes that, for a critical input vector
= (xM,O,...,O ) , the largest output be lower-bounded by:
where q is a design parameter trimming the SM-mapping
quality. Such a parameter relates to the smallest gain value
for which the eventual circuit will still discriminate a
critical input vector consistently. A few simple derivations
from (5) yield the second constraint on P :
Thus, the design process minimizes the quadratic cost c(P)
within the linear constraints (4) and (6). An especially
profitable result of the optimization process is that
coefficient p invariably nullifies, which eliminates
amplifiers from the design of fi'p)(x). Moreover, the bias
I1 - 118
Fig.2 - The squaring circuit for &)(xi).
coeficent a usually lies in very narrow ranges and
simplifies the supporting circuitry; in fact, in most cases, it
can be dropped altogether.
3. MODULAR CIRCUIT IMPLEMENTATION
The analog VLSI implementation involves current-
mode inputs and voltage-mode outputs. An effort to
simplifjl the basic elements making up the overall schema
justifies such a choice. A modular approach is adopted for
the circuit-architecture strategy (Fig. 1). Each i-th element
in SM (2) is supported by a specific pair of cells, which
operate locally and compute f y ) ( x i ) and the final ratio,
s?)(X). Increasing the dimension of the SM vector (2)
simply requires that an additional pair of such cells be
included. The presence of a valid output is signaled by the
TRIGGER logic level.
The only section propagating global information is the
"denom" module, which handles the normalizing
denominator, iz. The fact that it requires a minimal wiring
is a benefit resulting from the: modular architecture, which
greatly facilitates a VLSI implementation in the layout
design.
3.1 Implementation of the expo approximation
Thanks to the optimization result p=O, the function
approximating exp(x) becomes fi(p)(x)= a + cx2, which
can be easily implemented by joining a bias source to a
current-squaring circuit. The circuit adopted for the
current-squaring task (Fig.:?) exploits the translinear
principle in MOS-transistor loops, and is described in [ 8 ] .
The transfer function of this subcircuit is given by:
(7)
This schema has been selected for its accurate
 Fig.3 - The denominator cell and two (numerator) output cells.
performance. The normalizing current, Iz, is directly
available by summing, in a common node, a copy of the
output current from each stage.
3.2 Implementation of the normalizing ratio
The novel circuit that supports the normalizing ratios in
(2) exploits two features to sharply reduce the schema
complexity: 1) all quantities are positive; 2) all outputs
share a common denominator value. The modular
multiple-divider circuit treats the numerator(s) and the
denominator separately in specific subcircuits. Each
numerator module simply includes a switched capacitor,
whereas the single denominator module combines a
capacitor with an inverter stage.
Figure 3 displays the schemata of two output cells and
of the denominator cell; the voltages across the capacitors
ci and Cd are initially null. The denominator current, iz,
feeds Cd whose voltage increases approximately linearly
(over a limited time interval). The switches M,, remain
“ON’ as long as vz keeps smaller than the inverter’s
threshold voltage, VL+ This enables $xi) to charge Ci
linearly as well. The circuit status changes as soon as vz
exceeds VLH; then each M,, cuts off the current flow that
was charging Ci. The switching time, T, is given by
T=- cd ‘LH
(8)
KdlZ
where Kd is the static gain of the denominator current-
driving stage. Disregarding the switching transient at the
inverter stage, the charge stored in Ci is given by
Civi( T ) = Ki T $xi). By using (8), one obtains:
K.C, if (xi)
V i (T)= IVLH- (9)
K d ci I,
The voltage on each capacitor, Ci, is proportional to the
ratio of the associate current, $xi), to the denominator;
since I z = c , i f(xi),expression (9) implements (2). The
I circuit was simulated using HSPICE at level 13; the
availability of a valid result in vi(T) is flagged by the
I1 - 119
switching of the TRIGGER logic level, which is
represented by the inverter status. The reset of the overall
circuit status is obtained by activating the switches that
ground all capacitors. As output voltages v(sJ are read out
on each “numerator” side, the ratios sharing the common
denominator are worked out in parallel. Likewise, the
largest current flows in the unique “denominator” module.
Cascode mirror stages provide high impedance levels
toward storage capacitors. The designer may take
advantage of the SM property $ x i ) d z and rescale current
amplifications in (9) by sizing the MOS aspect ratios to
impose dynamic-range constraints on output-voltage
levels. The overall ratio performance is sensititive to
mismatches rather than the actual parameters of the
individual components.
4. EXPERIMENTAL RESULTS
The optimization-based design methodology was tested
on different SM configurations, that is, for a range of SM
dimensions: n E [3,15]. In all cases, the resulting circuit
performances turned out to be satisfactory in terms of
mapping accuracy. This section presents a sample of the
obtained results by illustrating the complete design process
for the case n = 5 . In our experiments, the input current
bound was always set to xM = 4 PA. The final VLSI
implementation was supported by 0 . 8 ~ AMS technology
with Vcc=SV.
A random-search algorithm [9] optimized the cost (3)
with the parameter set P={a=O.198,p=O}. One verifies the
consistency of this result: the a value makes the supported
function retum a uniform distribution si@)= Iln when
E O . The result p=O removes from the final schema any
term proportional to the input current. The remaining
degree of freedom, c, allows one to size the current-
squaring stage so as to control power consumption. For
n=5, this process leads to the settings given in Table I. The
maximum power consumption amounted to 690pW.
 Table I.
Circuit parameters (n=5)
1 Subcircuit I
I I Device I Setting I
MI I W/L = 16/4
sauaring
I Y
M, I W/L= 10/10
Cell Ib 3pA
Numerator Msw WIL = 212
cell c: n 5 nF
WIL = 6/12
Denominator
Inverter nMOS W/L = 20/2
Inverter MOS W/L = 10130
0.01 F
Fig.4 - Sample plot of SM output voltages.
Table 11.
Sample experimental results
I Input TrueSM ApproxSM HWSM achieved at the cost of an event-driven normalization
schema, which involves an asynchronous operation. The
X ( M ~ si(* .y!’)(~) V(si)No
circuit schema obviously imposes the classical tradeoff
0.40 0.02 0.02 0.040 between speed and power. Therefore, promising lines of
4.00 0.84 0.78 0.775
0.50 0.03 0.02 0.045 research associated with the design methodology consist in
1.01 0.04 0.06 0.080 testing different current-squaring circuits to reduce power
1S O 0.07 0.12 0.139 consumption.
0.10 0.02 0.01 0.040
4.00 0.93 0.95 0.919 5. REFEEWNCES
0.10 0.02 0.01 0.040
0.10 0.02 0.01 0.040 [ 11 Liu Shi-Chii “A Winner-Take-All circuit with
0.10 0.02 0.01 0.040 controlable Soft Max property” Adv. in Neur. In$ Proc.
4.00 0.20 0.20 0.230
NIPS-13,2000, pp.717-723
4.00 0.20 0.20 0.230 [2] Gold S, Rangarajan A, “Soflmax to softassign: neural
4.00 0.20 0.20 0.230 network algorithms for combinatorial optimization” J.
4.00 0.20 0.20 0.230 Artif: Neural Netwoks, 1995, vol. 2, No. 4, pp. 381-399.
4.00 0.20 0.20 0.230
[3] Cox DR, Millar HD The theory of stochastic process,
Methuen, 1965.
Figure 4 exemplifies a plot of output voltages resulting [4] Maas W “Neural computation with Winner-Take-All
from post-layout simulations of an example input X=(2.0, as the only nonlinear operation” Adv. in Neur. In$ Proc.
4.0, 0.1, 0.0, 0.0) (currents in pA). However, the SoftMax NIPS-13,2000, pp. 293-299
operation requires one to test the circuit performance over [ 5 ] Baysoy C, Kinney LL “Subthreshold MOS fizzy
a wide range of input vector configurations. To this end, a madmin neuron circuits” Proc. World Congress Neural
large set of experimental runs for different settings were Networks, 1994, vol. 2, pp. 500-505.
carried out that confirmed the approach effectiveness. [6] Chang C-C, Liu S-I “Pseudo-exponential function for
Table I1 presents some sample results and compares, for MOSFETs in saturation” IEEE Trans. Circuits and Sys. 11,
each input dimension, the true SM values with the NOV2000, ~01.47,N0.11, pp.1318-1321
theoretical approximations predicted by (2) and with [7] Laopulos TL, Karybakas CA “A simple analog
hardware results (scaled by an invariant factor Vo=l.6V). division scheme” IEEE Trans. Instrum. and Meas., Aug
Empirical evidence proves the satisfactory and stable 1991, ~01.40,No.4, pp.779-782
performance of the circuit in the widest range of operation. [8] Wiegerink RJ Analysis and synthesis of MOS
translinear circuits, Kluwer Academic Publishers, 1993
4. CONCLUSIONS [9] Baba N “A new approach for finding the global
minimum of error finction of neural networks” Neural
The optimization-based strategy yields profitable Networks, V01.2, 1989, pp.367-373.
results concerning both SM-representation accuracy and
VLSI-realization efficiency. These advantages are

I1 - 120