Sunteți pe pagina 1din 8

ARTICLE IN PRESS

Neurocomputing 69 (2006) 1860–1867


www.elsevier.com/locate/neucom

Mixed-signal neuron-synapse implementation for large-scale


neural network
Il Song Han
Department of Electronic and Electrical Engineering, University of Sheffield, Mappin Street, Sheffield, South Yorkshire S1 3JD, UK
Received 28 February 2005; received in revised form 7 August 2005; accepted 22 November 2005
Available online 14 June 2006

Abstract

This paper describes a mixed-signal neural networks VLSI for low power and asynchronous operation. The voltage-controlled
transconductance produces the synaptic function of multiplication and summation of synaptic currents for neuron, by compensating the
non-linearity of MOSFET resistance in the triode region.
The flexible configuration of synapse accommodates the spike-based neural networks, inspired by the biological plausibility and low
power requirement. The neuron with a combination of synapses demonstrates asynchronous spikes of integration-and-firing with a
refractory period. The speed of individual synapse is simulated up to 300 Mega operations/s with the power consumption of less than
33 mW, using 0.18 mm CMOS.
r 2006 Elsevier B.V. All rights reserved.

Keywords: Analogue-mixed VLSI neural network; Pulse/spike-based neural computation; Asynchronous operation; MOSFET resistance; Voltage-
controlled linear resistance

1. Introduction cognition [19]. Hence, the current demand in large capacity


or biological performance drives analogue-mixed neural
VLSI neural networks have been continuously developed networks VLSI of pulse/spike-based operation, because of
either in digital or analogue, as both methods have its advantages in large-scale implementation and low
different advantages. The advantage of analogue VLSI is power consumption in comparison to the digital. There
low power consumption or larger networks integration, exists an issue of accuracy improvement in analogue-mixed
though digital VLSI has advantages of design flexibility or implementation, with technology advancement towards
leading-edge technology. Issues in an analogue neural 0.18 or 0.13 mm. An example of analogue-mixed VLSI is in
network VLSI can be the accuracy problem, or the Fig. 1, which was developed for real-time packet control
complexity, in comparison with the digital. In some special [9]. Though the circuit exhibits the accuracy, it demands
applications, the analogue utilised or developed better its the complex supply voltages which limit the level of
non-ideal characteristic or complex design [11,15]. integration under low supply voltages of up-to-date VLSI
Recently, biologically inspired neural networks, i.e. technology.
spike-based operation is widely investigated in various This paper introduces the new development in analogue-
areas, from robots to regenerative medicine [3,5,6,16,21]. mixed neural networks VLSI with advantages of up-to-date
The pulse or spike based implementation of neural advanced technology and fully asynchronous spike operation.
networks has advantages of VLSI neural networks, which
is suitable for the large-scale real-time or embedded 2. Analogue-mixed neural network synapses for low power
requirement [4,8]. In the brain science, a large scale and and asynchronous operation
general neural network VLSI has been also expected for
The current–voltage (I–V) relationship of MOSFET in
Tel.: +44 (0)114 222 5810; fax: +44 114 222 5146. the triode region can be adopted to implement electronic
E-mail address: i.s.han@sheffield.ac.uk. synapses, as it provides the multiplication function.

0925-2312/$ - see front matter r 2006 Elsevier B.V. All rights reserved.
doi:10.1016/j.neucom.2005.11.013
ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1861

(a) (b)

Fig. 1. (a) Electronic synapse circuit with compensated linear MOSFET resistance in the triode region, and (b) its measured linearity of output synapse
current vs. weight voltage (710 mA with 70.5 V).

(a) (b)

Fig. 2. (a) New synapse circuit by voltage-controlled linear resistance of two MOSFETs in the triode region, and (b) its representation as differential input
and multiplying transconductor.

The equation of interest is that the drain-source current new MOSFET resistance-based analogue multiplier is
IDS for a MOSFET in the linear or triode region: shown in Fig. 2(a). For an efficient analogue multi-
plication, two terms, VT?VDS and V2DS/2, from Eq. (1)
I DS ¼ afðV GS  V T ÞV DS  V 2DS =2g. (1) should be eliminated from the output. Based on Eq. (1) for
the MOSFET in the triode region, the currents of transistor
Here, a is the MOSFET process parameter including the M1 and transistor M2 are
geometry, VGS, VDS, VT, the transistor gate-source, drain-
source and threshold voltage, respectively. To achieve a I M1 ¼ afðV 2  V T ÞV 1  V 21 =2g, (2)
linear voltage (VGS)-to-voltage (VDS) multiplier, the
balanced circuit with an operational amplifier can be used I M2 ¼ afðV 2þ  V T ÞV 1  V 21 =2g. (3)
to remove the second order term of V2DS/2 in Eq. (1).
Though various methods for suppressing or controlling the Here, V2+ and V2 produce one input of two variables
nonlinearity have been developed, there are limitations in for the multiplication and voltages of V2+ and V2 keep
solving the nonlinear problems. The synapse circuit of Fig. transistors M1 and M2 in the triode region, i.e. both M1
1 is an effective method to compensate such nonlinearity, and M2 are operated under the condition of
but it still has the drawback of demanding bipolar supply VGSVT4VDS. V1 is the other input of multiplication,
voltages to remove the second-order term in Eq. (1). which represents the drain-source voltage of M1 and M2.
The synapse circuit proposed in this paper makes also a The source voltages of both M1 and M2 remain in
simple use of the MOSFET in the triode region. It realises common as M3 or M4 acts as a diode. With the help of
the high speed and a small size analogue multiplier without current mirrors (M3–M8 and M4–M7–M5–M6), the
an amplifier or dual supply voltages. The synapse circuit of synaptic output current is the difference of IM1 and IM2
ARTICLE IN PRESS
1862 I.S. Han / Neurocomputing 69 (2006) 1860–1867

in Eqs. of (2) and (3): simulation. The power supply voltage is 3.3 V and both
accurate operation and low power consumption are design
I OUT ¼ I M1  I M2 objectives. The linearity of synapse circuit is shown in
¼ aðV 2þ  V 2 ÞV 1 Fig. 3(a) and demonstrates the behaviour of multiplier in
¼ a V WEIGHT V 1 , ð4Þ Eq. (4). One of inputs to M1 and M2 is applied with
sinusoidal signal with DC offset, while the other is applied
where VWEIGHT is the difference of V2+ and V2. One of with the same DC offset. The output current illustrates the
V2+ and V2 can be a DC reference voltage while the other analogue multiplication with amplitude modulation by
one is a synaptic weight plus DC reference. From Eq. (4), neural input of triangular signal, though the neural input
the synaptic multiplication of synapse weight (VWEIGHT) signal of binary state is enough for general purpose pulse/
and effective neural input (V1) is achieved by two spike neural networks. The functional description of
MOSFETs operated in the triode region and pairs of synapse in Fig. 2(b) presents the general output character-
current mirrors. istic of Fig. 3(a), as a synapse can be used as an
The summation of post-synaptic current is attained by element for complex neural signal processing. The transis-
common-output connection of synapses as each synapse tor sizes used for the simulated output in Fig. 3(a) are
can contribute individual synaptic output current. Both of (W/L)M1 ¼ (W/L)M1 ¼ 0.4m/0.6m, (W/L)M3 ¼ (W/L)M4 ¼
MOSFET M6 and M8 act as current source by either (W/L)M7 ¼ (W/L)M8 ¼ 1.6m/0.4m, and (W/L)M5 ¼
sourcing or sinking the synaptic output current, and the (W/L)M6 ¼ 0.4m/0.4m.
summation of post-synaptic current is computed by a The speed of more than 300 Mega connection per second
integration capacitor in each neuron. inputs is simulated as in Fig. 3(b). The flexibility in power
The new synapse of Fig. 2(a) designed using 0.18 mm consumption can increase the operation speed further, as it
standard CMOS technology and evaluated by HSPICE accelerates charging or discharging rate of output current

Fig. 3. (a) New synapse circuit’s characteristics as a multiplier, a neural input voltage (triangular wave) and synaptic weight voltage (sinusoidal wave) for
inputs and modulated current for synaptic output current, and (b) transient characteristics of pulses/spikes operation over 300 Mega connections per
second for synapse cell.
ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1863

faster. The overall power is less than 33 mW per activated


synapse cell and the new circuit is suitable for large scale
VLSI neural network implementation. As observed in the
circuit of Fig. 2(a), there is no current or power
consumption without any active neural input. Without
neural input, V1, IM1 and IM2 become null and there flows
no current in M1–M3 or M2–M4. Also, there flows no
current in M5–M7 or M6–M8, as the source current of
current mirror is null. Therefore, the synapse cell does not
consume any power if there is no activity. The operational
principle in Eq. (4) does not require any timing character-
istics of neural inputs, and the synapse circuit of Fig. 2(a)
operates either asynchronously or synchronously, free from
any synchronous constraints. The power consumption is
dependant on the activity of neural input or pulse/spike
firing rates of neural networks.

Fig. 5. Measured characteristics of the test circuit in Fig. 4, where two


3. The experimentation of a proposed multiplier circuit with 0.5VPP sinusoidal voltage sources are applied as synapse weight and neural
input. The output waveform is measured at the output node of the circuit
conventional CMOS devices
in Fig. 4.

The basic operation characteristics of new synapse


4. Analogue-mixed and bio-inspired neurons for VLSI
circuit in Fig. 2(a) is tested and measured with the discrete
implementation
CMOS transistor circuit in Fig. 4. The experimentation
circuit is implemented without any additional elements,
There are different requirements for the VLSI imple-
other than M1A and M2A. M1A and M2A are added to
mentation of neurons depending on applications, i.e.
improve the design constraints from the pre-determined
asynchronous pulse/spike or synchronous ones. Recent
discrete transistors. The NMOS transistors of ALD1116
developments of biologically inspired neural networks or
and PMOS transistors of ALD1117 are used to measure
neuromorphic solutions are largely based on asynchronous
the characteristics of the circuit in Fig. 4. For conductive
operation, while certain applications like ‘real-time packet
elements of (M1, M1A) and (M2, M2A) in the triode
switch controller’ are based on synchronous operation [9].
region, PMOS transistors are used to implement the larger
Either asynchronous or synchronous neuron is available to
dynamic range by simulating lower transconductance.
the synapse of Fig. 2(a) for mixed-signal neuron-synapse
The measured characteristics in Fig. 5 demonstrated the
implementation of large-scale neural network.
circuit in Fig. 2(a) or Fig. 4 as the synapse circuit or
multiplier circuit, where the synapse weight was applied
4.1. Synchronous neuron
with the 0.5VPP of the sinusoidal waveform and the neural
input was applied with 0.5VPP of the sinusoidal waveform
For analogue or analogue-mixed neural network opera-
with the offset voltage of 1.1 V. The measured character-
tion, the neural state is represented as the voltage at a
istics in Fig. 5 shows the 2 QAM function as in Fig. 3(a),
neuron capacitor connected to networked synapse outputs
though the test circuit was based on only two fixed
of Fig. 2(a). Every synapse produces the current by
transistors instead of three designed ones.
multiplication of weight value and input value based on
Eq. (4), which results a voltage of neural state by the
integration of synaptic current in a neuron capacitor.
The circuit in Fig. 6(a) shows the block diagram of a
neuron with the linear ramp function and sigmoid-like
continuality. At each sampling with CLK1 ‘high’ in
Fig. 6(a), the summation of the synaptic computation as
a buffered input voltage is fed to a capacitor for copying
the neural state into the neuron. The CLK2 signal operates
on that sampled voltage, which is then transformed into the
new voltage level for a sigmoid-like function enabled by
two MOSFETs in the diode configuration. Two MOSFET
Fig. 4. The experimentation circuit of Fig. 2(a). PMOS transistors M1,
as in Fig. 6(a) are for defining the dynamic range of neural
M1A, M2, M2A, M5 and M6 are ALD1117, and NMOS transistors M3, states, where one is for the upper bound conversion and the
M4, M7, and M8 are ALD1116. other is for the lower bound conversion. With the reduced
ARTICLE IN PRESS
1864 I.S. Han / Neurocomputing 69 (2006) 1860–1867

(a) (b)

Fig. 6. (a) Analogue-mixed neuron processing circuit and (b) its transfer characteristics.

IDISCHARGE

(a) (b)

Fig. 7. (a) Pulse/spike based synchronous neuron, and (b) its chip photograph of a mixed-signal neuron-synapse VLSI.

gate-to-source voltage during the operation, the sub-


threshold conduction delivers the overall conversion close GK
Cmembrane Gleak GNa
to a continuous sigmoid function as shown in the transfer
curve of Fig. 6(b). The transformed voltage as a neural Eleak ENa EK
state is applied to the following part of neuron circuit while
CLK3 is ‘high’.
In order to provide the pulse/spike output, the dischar-
ging current is subtracted from the neural state of a Fig. 8. An electrical equivalent circuit of a neuron.
sampling capacitor Cs when the comparator CON output
of Fig. 7(a) turns the switch MOSFET M2 ‘on’. The output
pulse from a comparator is generated when the voltage at capacitor Cs of holding neural state occupies larger portion
the sampling capacitor is higher than the provided of neuron area. The synchronised operation is required for
reference level (REF). An AND gate in Fig. 7(a) encodes real-time synchronous applications like a high-speed
the pulse/spike output with system reference clock CLK telecommunication switch controller.
and PWM output of comparator CON. A MOSFET
switch M1 controls the synchronous operation of neuron 4.2. Asynchronous neuron
output. The neuron circuit does not always include the
transformation block in Fig. 6(a), because many applica- The biologically motivated neuron has been targeted as a
tions demand only binary neuron output. neuron model for its advantages of integrating various
As a neuron circuit in Fig. 7(a) is not based on any applications [1,2,12,13,14,18]. The feature of asynchronous
amplifying circuit but only a comparator, diodes, or firing or spike dynamics emerges as a key aspect, because
switches, the total operation speed of neural networks such electrical signals seem to be essential to the neural
does not degrade the overall performance based on information processing in biological unit. The Hodgkin–
analogue-digital mixed synapse operation of Fig. 2(a). Huxley (H–H) formalism is widely adopted for its biophy-
The complexity of neuron introduces unlikely any issue in sical characterization and dynamics. An electrical equivalent
synapse-neuron integration as illustrated in Fig. 7(b) of circuit model of Fig. 8 is known as an empirical model by
neuron-synapse chip photograph, where the sampling the H–H formalism, which describes quantitatively the
ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1865

KA +

INTEGRATOR
LOWPASS
+ SIGMOID
FILTER
X +

MEMBRANE
X + POTETNTAL LOWPASS
FILTER X
EL
GL
ENA
+
Fig. 9. A H–H based neuron block diagram based on functions of synapse in Fig. 2.

dynamics of the voltage-dependent conductance. The


asynchronous neural spikes inspire all sorts of neuro-
morphic implementation, although most of particular
recognition tasks do not exhibit any major advantages
based on H–H formalism. Asynchronous neuron spikes or
pulses are even considered as a key element in high level
cognition [20]. Hence, asynchronous dynamics of the H–H
formalism is adopted as a reference model for asynchronous
neuron compatible to the synapse of Fig. 2(a).
The voltage–current relationship of Eq. (4) can imple- Fig. 10. (a) Simulation of action potential, and (b) an experimental action
ment the voltage-dependant conductance employed in potential from Ref [10].
Fig. 7. An empirical mathematical formalism models
dynamics of each conductance element as
G ion ¼ Gion max x,
implement a asynchronous silicon neuron in Fig. 11.
dx=dt ¼ aðb  xÞ, A further simplification is introduced by an approximation
iion ¼ Gion ðV m  E ion Þ, ð5Þ of sigmoidal function and a reduction of one differentiated
conductance stage. A differential amplifier with buffer
where b is sigmoidal function of the membrane potential. stages (Diff amp in Fig. 11) acts as a sigmoidal transfer
Vm is a membrane potential and the overall dynamic function for the steady-state activation variable, where a
modelled by an action potential and related ionic reference voltage is introduced for the half-activation
conductance. potential. Two synapses of SYNAPSE2 and SYNAPSE3
The block diagram of asynchronous neuron in Fig. 9 is in Fig. 11(a) realise the approximation of two voltage-
inspired by controlled conductance from H–H model controlled conductances for the smaller chip area without
and sum-multiplier of spike-based synapse in Fig. 2. The major drawbacks.
differential equation in Eq. (5) is implemented by first- A low-pass filter is implemented by one synapse of
order low-pass filter, which induces a delayed response. SYNAPSE1 and a capacitor CL, as a synapse of Fig. 2 also
The neuron of Fig. 9 models three components of ionic represents an operational transconductance amplifier from
conductance, where KA, ENA, and EL represent EK, ENA, Eq. (4). A transconoductance amplifier acts as a resistor
Eleak respectively and GL as Gleak in Fig. 8. The capacitor of or an equivalent conductance of the RC low-pass filter,
Cmembrane is modelled as an integrator for its functional where the resistance R can be programmed by a control
behaviour. voltage (SETbias in Fig. 11). The control voltage (SETbias)
The MATLAB simulation of H–H neuron block corresponding to VWEIGHT of Eq. (4) can be used to
diagram in Fig. 9 shows the result in Fig. 10(a), control the bandwidth of a low-pass filter, otherwise
which exhibits matching behaviour to H–H model in remaining as a fixed one. The linearity of low pass filter
Fig. 10(b). is 45 dB in harmonic distortion and tuneable up to 50%
The H–H formalism inspired neuron, which is based on in bandwidth by a control voltage, from the simulated
the conductance controlled model of Fig. 9 is used to result of the circuit in Fig. 2(a).
ARTICLE IN PRESS
1866 I.S. Han / Neurocomputing 69 (2006) 1860–1867

Eleak

SETleak Ebias

Cm

amp
Diff
Membrane
potential Vref
CL
SETbias

parator
Com-
Vthres

(a) (b)

Fig. 11. (a) Asynchronous spike firing neuron by three synapses of Fig. 2, inspired by H–H model, and (b) asynchronous behaviour of a neuron circuit
with synaptic spike currents as inputs: (from top to down) neuron capacitor’s potential as a membrane potential, synaptic current spikes as input, and
firing pulses with the refractory period.

The neuron of Fig. 11(a) shows a membrane dynamic networks VLSI with small power consumption and no need
behaviour consistent to the simulated result in Fig. 10(a). for a synchronous operation.
Instead of single stimulus as a single firing in Fig. 10, the
current spike stream is applied to the simulation of References
asynchronous neuron as in Fig. 11(b). The 2 Mega spike
stimulus per second is simulated as in Fig. 11(b), though [1] T. Asai, et al., A subthreshold MOS neuron circuit based on the
elements of synapse circuit for a neuron can operate in Volterra system, IEEE Trans. Neural Networks 14 (5) (2003)
1308–1312.
much higher speed as in Fig. 3. The HSPICE simulation [2] C. Bartolozzi, G. Indiveri, A neuromorphic selective attention
result demonstrates the asynchronous behaviour of inte- architecture with dynamic synapses and integrate-and fire neurons,
gration-and-firing with a refractory period. There are in: Proceedings of BICS, Sterling, BIS2.2, 2004.
advantages of asynchronous operation, removal of refer- [3] G. Bugmann, Biologically plausible neural computation, Biosystems
ence clocks, and low voltage operation compared to 40 (1997) 11–19.
[4] C. Christodoulou, G. Bugmann, T. Clarkson, A spiking neuron
previous pulse-based analogue-mixed neural networks model: applications and learning, Neural Networks 15 (2002)
VLSI [7]. Another advantage of H–H formalism inspired 891–908.
silicon neuron in Fig. 11(a) is its fast operation speed in [5] R. Eckmiller, O. Baruth, D. Neumann, Neural information proces-
Fig. 11(b), which may demand substantial computing sing efforts to restore vision in the blind, in: Proceedings of ICONIP,
Lecture Notes in Computer Science, No. 3316, Springer, Berlin, 2004,
otherwise [17].
pp. 10–18.
[6] D. Floreano, C. Mattiussi, Evolution of spiking neural controllers for
5. Conclusion autonomous vision-based robots, ER, Lecture Notes in Computer
Science, No. 2217, Springer, Berlin, 2001, pp. 38–61.
The new spike-based synapse circuit and neuron circuit [7] I.S. Han, Neural network VLSI implementation and its applications,
demonstrate the feasibility of large scale, low power and Korea Telecom J. 2 (1) (1997) 12–20.
[8] I.S. Han, Mixed-signal neuron-synapse implementation for large
asynchronous neuron-synapse VLSI implementation, for scale neural network, in: Proceedings of BICS Sterling, NC4.2, 2004.
its simpler circuit, speed and biological plausibility based [9] I.S. Han, R. Webb, Neural network Switch controller with analogue-
on the controlled conductance. It is flexible to adapt its digital mixed neural network VLSI, in: Proceedings of the EANN’97,
characteristics of speed, power, accuracy on demand, as 1997, pp. 299–302.
[10] M. Hausser, The Hodgkin–Huxley theory of action potential, Nat.
those can be tailored by controlling the operating synapse
Neurosci. Suppl. 3 (2000) 1165.
current and parallel wired-OR expansion. For an example, [11] Y. Horio, K. Aihara, O. Yamamoto, Neuron-synapse IC chip-set for
the higher operation speed, or the fast settlement is large-scale chaotic neural networks, IEEE Trans. Neural Networks
observed with increased operational current levels in a 14 (5) (2003) 1393–1404.
synapse element. The overall power consumption can be [12] E.M. Izhikevich, Which model to use for cortical spiking neurons?
less in practice, as individual synapse only demands the IEEE Trans. Neural Networks 15 (5) (2004) 1063–1070.
[13] B. Linares-Barranco, et al., A CMOS implementation of FizHugh–
power when there is an active neural input signal. Nagumo neuron model, IEEE JSSC 26 (7) (1991) 956–965.
The core of proposed electronic synapse and neuron are [14] S. Matinoia, et al., Cultured neurons coupled to micoelectrode
estimated, respectively, as within the size of 2 mm  2 mm arrays: circuit models, simulations and experimental data, IEEE
and 6 mm  6 mm in 0.18 mm CMOS VLSI technology, Trans. Biomed. Eng. 51 (5) (2004) 859–864.
though the overall neuron area is finalised by the desired [15] M. Milev, M. Hristov, Analog implementation of ANN with inherent
quadratic nonlinearity of the synapses, IEEE Trans. Neural Net-
average firing frequency. The asynchronous spike based works 14 (5) (2003) 1187–1200.
neural networks in advanced CMOS VLSI technology [16] T. Pearce, et al., Silicon-based neuromorphic olfactory pathway
provides the advantage of analogue-digital mixed neural implementation, in: Proceedings of BICS, Sterling, BIS 6.3, 2004.
ARTICLE IN PRESS
I.S. Han / Neurocomputing 69 (2006) 1860–1867 1867

[17] U. Seiffert, Artificial neural networks on massively parallel computer developed the large scale pulse-based neural network VLSI. Since 2002,
hardware, Neurocomputing 57 (2004) 135–150. he has been with the department of Electronic and Electrical Engineering,
[18] M. Simoni, et al., A multiconductance silicon neuron with University of Sheffield, Sheffield, UK.
biologically matched dynamics, IEEE Trans. Biomed. Eng. 51 (2) His research interests include analogue-mixed CMOS VLSI systems for
(2004) 342–354. neural networks and telecommunication applications.
[19] J. Taylor, Paying attention to consciousness, Progr. Neurobiol. 71
(2003) 305–335.
[20] J. Taylor, Private communication, 2004.
[21] N. Taylor, M Hartley, J. Taylor, The learning of insertions by the
Cerebellum, in: Proceedings of BICS, Sterling, CNS 2.2, 2004.

Il Song Han received the B.Sc. degree in


Electronic Engineering from Seoul National
University in 1979, the M.Sc. degree and the
Ph.D. degree in Electrical Engineering from
Korea Advanced Institute of Science and Tech-
nology, Korea, in 1981 and 1984, respectively. He
received the MBA degree from Cranfield Uni-
versity, Cranfield, UK, in 2000.
He worked for 14 years as a researcher in
telecommunication R&D industry, where he