Edited by: Ad Aertsen, Albert Ludwigs University, Germany
Reviewed by: Bruce Graham, University of Stirling, UK; Guenther Palm, Universit of Ulm, Germany
*Correspondence: Weiliang Chen, Computational Neuroscience Unit, Okinawa Institute of Science and Technology, Seaside House, 7542, Onna, Onna-Son, Kunigami, Okinawa 904-0411, Japan. e-mail:
This is an open-access article subject to a non-exclusive license between the authors and Frontiers Media SA, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and other Frontiers conditions are complied with.
The problem we address in this paper is that of finding effective and parsimonious patterns of connectivity in sparse associative memories. This problem must be addressed in real neuronal systems, so that results in artificial systems could throw light on real systems. We show that there are efficient patterns of connectivity and that these patterns are effective in models with either spiking or non-spiking neurons. This suggests that there may be some underlying general principles governing good connectivity in such networks. We also show that the clustering of the network, measured by Clustering Coefficient, has a strong negative linear correlation to the performance of associative memory. This result is important since a purely static measure of network connectivity appears to determine an important dynamic property of the network.
Network models for associative memories store the information to be retrieved in the values of the synaptic weights. Weighted summation of their synaptic inputs then allows the neurons to transform any input pattern into an associated output pattern. Network models for associative memories come in two flavors. In pure feedforward models, like the one-layer perceptron, a static input pattern in the afferent fibers is in one step transformed, by weighted summation, into a static pattern of activity of the neurons in the output layer. For instance Purkinje cells in the cerebellum have been proposed to recognize in this way patterns of activity in the mossy fibers (Marr,
These questions were solved analytically for the so-called Hopfield network (Hopfield,
In the present paper, building on our previous work (Davey and Adams,
For each model investigated, a collection of
With this configuration, two extreme cases are widely known and commonly studied. The first case is a completely local network, or lattice, whose nodes are connected to those nodes that are closest to it. An example of a local network is the cellular neural network (CNN), where units are connected locally in 2-D (Brucoli et al.,
Our first connection strategy is adapted from the method proposed by Watts and Strogatz (
Another connection strategy investigated by us exhibits Gaussian connectivity, where each unit has
The biological context of this connectivity comes from the mammalian cortex, which is thought to have a similar connectivity between individual neurons (Hellwig,
The modularity of mammalian cortex, commonly referred to as the hypothesis of cortical columns (Mountcastle,
Another modular network we have investigated is called the Gaussian-Uniform Modular Network. In such a network, the intra-modular connections within each module have a Gaussian-distributed connectivity with a SD of σ
The best connection strategy for an associative memory network may be dependent on details of the neuron model. For instance, the wiring length of a connection may have no significant impact on simple threshold neuron models, but is important in a spiking model simulating geometry-dependent time delays. To reveal the intrinsic principles that may govern network construction, this paper investigates connection strategies in various associative memory networks with either the traditional threshold unit models or the more biologically realistic, leaky integrate-and-fire spiking neuron models.
Each model needs to be trained before any measure of performance can be obtained. Canonical associative memory models with threshold units, for example the Hopfield net, commonly use a one-shot Hebbian learning rule. This, however, does not perform well if the networks are sparse and non-symmetric (Davey et al.,
A set of random, bipolar, or binary vectors is presented as training patterns, where the probability of any bit on the pattern being on (+1), referred to as the
The value
For threshold unit models we use the standard bipolar +1/−1 representation. For the spiking networks we use 1/0 binary patterns, as they can be easily mapped onto the presence or absence of spikes.
Threshold unit models and integrate-and-fire spiking neuron models require different update dynamics. For threshold unit models, we use the standard asynchronous dynamics: units output +1 if their net input is positive and −1 if negative. As the connectivity is not symmetrical there is no guarantee that the network will converge to a fixed point, but, in practice these networks normally exhibit straightforward dynamics (Davey and Adams,
Due to the intrinsic model complexity, associative memory models with spiking units require careful tuning so that the performance is comparable to threshold unit models. We use a leaky integrate-and-fire neuron model which includes synaptic integration, conduction delays, and external current charges. The membrane potential
A spike that arrives at a synapse triggers a current; the density of this current (in
where
where
The change of membrane potential is defined by
In this equation, −(
In networks trained with a learning threshold
External currents are injected into the network in order to trigger the first spikes in the simulation. Each current injection transforms a static binary pattern to a set of current densities. Given an input pattern, unit
The associative memory performance of the threshold unit network is measured by its effective capacity (EC; Calcraft,
The EC of a particular network is determined as follows:
The EC of the network is therefore the highest pattern loading for which a 60% corrupted pattern has, after convergence, a mean similarity of 95%, or greater with its original value.
The EC measure needs modification to suit the spiking model. We adopt the concept of
where
The memory retrieval measure was designed for sparse patterns where the chance overlap is important. In our case the patterns are unbiased so the chance overlap is close to 0. The memory retrieval
In recent years connectivity measures from Graph Theory have been used in the investigation of both biological and artificial neural networks. We use two of these measures to quantify our connectivity strategies.
A common measure of network connectivity is Mean Path Length. The Mean Path Length of a network
where
The Mean Path Length was originally used to define the “small-world” phenomenon found in social science (Milgram,
A fully connected network has the shortest and unique Mean Path Length of 1, whilst in sparse networks the measure varies for different types of connectivity (see Tables in Appendix). A locally connected network has a high Mean Path Length, since each unit is only connected to its nearest neighbors and it is difficult to reach distal units. On the other hand, completely random networks usually have short Mean Path Lengths. Intermediate cases, for example the Watts–Strogatz small-world network, have Mean Path Lengths similar to completely random networks, but significantly lower than the ones of lattices.
Although commonly used as a connectivity measure of neural networks, our investigation reveals that the Mean Path Length is insensitive to connectivity changes if the network is far from local. Another measure, named the
The definition of
Then we define
which denotes the fraction of all possible edges of
The Clustering Coefficient of a fully connected network is 1, since each node is connected to all others directly. Locally connected sparse networks have high Clustering Coefficients whilst in a completely random networks it is usually low. Interestingly, the so-called small-world networks usually have high Clustering Coefficients, similar to local networks, but short Mean Path Lengths, like completely random networks. Such types of connectivity (short Mean Path Length, high Clustering Coefficient) are also observed in natural networks, for instance, the mammalian cortex.
All networks in our study have 5000 units. The first set of results are for the non-spiking model with 250 incoming connections per unit, that is,
As pattern correction is a global computation intuitively one might expect networks with shorter mean path lengths to perform better than those with longer mean path lengths. Our results, shown in Figure
Once again intuition suggests that performance could be related to clustering. In a highly clustered network global computation could be difficult: with information staying within clustered subnetworks and not passing through the whole network. This is confirmed by our results in Figure
We now examine how well our results generalize to the more complicated integrate-and-fire model. Because the EC evaluation of this network takes much longer to compute than for the threshold unit model, we reduced the number of incoming connections per unit,
Figure
In nature, the construction of associative memory networks is restricted by resource, thus a connection strategy that optimizes both wiring cost and performance would be preferable. We define the wiring cost of two connected nodes in the network simply as the distance between them, and average it over all connected nodes in the network. Note that this is a quite different measure to mean path length, which measures steps along the connection graph. This measure is also different from the connection delay used in the spiking network models, where the cube-root of
Figure
Our study of memory performance in networks with varying connection strategies extends previous work on the effect of connectivity on network dynamics (Stewart,
The main finding of the present study is that for networks with a fixed number
These results were obtained with random uncorrelated patterns, hence without taking into account the statistics of a natural environment. The patterns were stationary (no sequences) and so was largely the dynamics of the network (no oscillations or temporal patterns). Our study was conducted on a ring of neurons, but the results can be extended to 2-D networks (Calcraft et al.,
In the network of spiking neurons, the resulting feedback excitation and inhibition forced the neurons into persistent UP or DOWN states. The network lacked, however, dynamical behavior like gain control, synchronization, or rhythm generation, for which separate populations of excitatory and inhibitory neurons would be required (Sommer and Wennekers,
Clustering (the occurrence of connections between the targets of the same neuron) may make some connections functionally redundant or induce loops. Cycling through loops may hamper convergence to an attractor. In addition, Zhang and Chen (
An unexpected finding was that for a given wiring cost, Gaussian-connected networks performed better than small-world networks. Although very-long-range connections speed up the convergence to an attractor during memory retrieval (Calcraft et al.,
The connectivity of the brain, and more particularly of neocortex, is sparse, but certainly not random (see Laughlin and Sejnowski,
Within a column, Song et al. (
We studied the capacity of memory storage and retrieval in networks of bipolar and spiking neurons, and compared different connection strategies and connection metrics. The single metric best predicting memory performance, for all strategies, was the clustering coefficient, with performance being highest when clustering was low. In large networks, the best connection strategy was a local Gaussian probability function of distance, both in terms of avoiding clustering and of minimizing the cost of wiring.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
W–S small-world network. | ||||
---|---|---|---|---|
Rewiring | MPL | CC | WC | EC |
0 | 10.540 | 0.745 | 63 | 63.3 |
0.1 | 2.030 | 0.481 | 187 | 79.3 |
0.2 | 1.959 | 0.320 | 311 | 88.2 |
0.3 | 1.951 | 0.218 | 432 | 94.7 |
0.4 | 1.950 | 0.152 | 553 | 99.7 |
0.5 | 1.950 | 0.108 | 672 | 103.5 |
0.6 | 1.950 | 0.080 | 790 | 105.9 |
0.7 | 1.950 | 0.063 | 902 | 106.9 |
0.8 | 1.950 | 0.054 | 1022 | 107.4 |
0.9 | 1.950 | 0.050 | 1132 | 107.5 |
1 | 1.950 | 0.050 | 1250 | 107.8 |
σ | MPL | CC | WC | EC |
---|---|---|---|---|
0.4 | 2.632 | 0.501 | 91 | 76.2 |
0.6 | 2.626 | 0.378 | 117 | 86.8 |
0.8 | 2.580 | 0.299 | 147 | 91.6 |
1 | 2.522 | 0.247 | 179 | 94.8 |
2 | 2.218 | 0.131 | 341 | 101.6 |
3 | 1.989 | 0.089 | 504 | 104.8 |
4 | 1.952 | 0.069 | 656 | 106.6 |
6 | 1.950 | 0.054 | 869 | 107.4 |
8 | 1.9502 | 0.052 | 979 | 107.4 |
10 | 1.9502 | 0.051 | 1043 | 107.4 |
Rewiring | MPL | CC | WC | EC |
---|---|---|---|---|
0 | undefined | 1.000 | 167 | 39 |
0.1 | 2.035 | 0.638 | 284 | 62.8 |
0.2 | 1.959 | 0.418 | 400 | 83.8 |
0.3 | 1.951 | 0.279 | 515 | 91.8 |
0.4 | 1.950 | 0.189 | 627 | 97.6 |
0.5 | 1.950 | 0.129 | 736 | 102.4 |
0.6 | 1.950 | 0.091 | 843 | 105 |
0.7 | 1.950 | 0.067 | 948 | 106.4 |
0.8 | 1.950 | 0.055 | 1051 | 106.8 |
0.9 | 1.950 | 0.050 | 1151 | 107.2 |
1 | 1.950 | 0.050 | 1250 | 107.2 |
σ | MPL | CC | WC | EC |
---|---|---|---|---|
0.4 | 1.950 | 0.154 | 595 | 98.8 |
0.6 | 1.950 | 0.135 | 607 | 101.2 |
0.8 | 1.950 | 0.124 | 616 | 102.8 |
1 | 1.950 | 0.119 | 623 | 103 |
2 | 1.950 | 0.117 | 636 | 103 |
4 | 1.950 | 0.117 | 641 | 103.2 |
8 | 1.950 | 0.117 | 643 | 103.6 |
10 | 1.950 | 0.116 | 644 | 103.6 |
0.4 | 1.957 | 0.287 | 353 | 91.2 |
0.6 | 1.957 | 0.254 | 369 | 93.4 |
0.8 | 1.957 | 0.245 | 379 | 94.2 |
1 | 1.957 | 0.242 | 385 | 94.4 |
2 | 1.957 | 0.242 | 396 | 94.4 |
4 | 1.957 | 0.242 | 400 | 94.6 |
8 | 1.957 | 0.242 | 402 | 94.8 |
10 | 1.957 | 0.242 | 402 | 95 |
0.4 | 2.019 | 0.398 | 234 | 82.8 |
0.6 | 2.020 | 0.361 | 250 | 86 |
0.8 | 2.020 | 0.353 | 260 | 87 |
1 | 2.020 | 0.351 | 265 | 87 |
2 | 2.020 | 0.350 | 275 | 87 |
4 | 2.020 | 0.350 | 280 | 87 |
8 | 2.020 | 0.350 | 281 | 87.2 |
10 | 2.020 | 0.350 | 282 | 87.6 |
W-S small-world network. | ||||||
---|---|---|---|---|---|---|
Rewiring | MPL | CC | WC | EC | EC_Fixed | EC_Cuberoot |
0 | 25.495 | 0.737 | 25 | 22 | 17 | 19.4 |
0.1 | 2.629 | 0.484 | 138 | 28.6 | 27 | 28.6 |
0.2 | 2.438 | 0.313 | 264 | 35.4 | 34.2 | 31.8 |
0.3 | 2.317 | 0.205 | 390 | 39 | 40 | 35 |
0.4 | 2.239 | 0.133 | 513 | 41.2 | 44.4 | 37.6 |
0.5 | 2.187 | 0.085 | 638 | 42.2 | 48 | 40 |
0.6 | 2.153 | 0.054 | 761 | 44 | 50.8 | 43.2 |
0.7 | 2.132 | 0.035 | 884 | 44.6 | 52.4 | 45.2 |
0.8 | 2.117 | 0.024 | 1006 | 44.4 | 54.4 | 48 |
0.9 | 2.111 | 0.020 | 1129 | 44 | 54.8 | 50 |
1 | 2.110 | 0.020 | 1252 | 44 | 55.2 | 51.6 |
σ | MPL | CC | WC | EC | EC_Fixed | EC_Cuberoot |
---|---|---|---|---|---|---|
0.4 | 2.889 | 0.484 | 52 | 28 | 24.2 | 27.4 |
0.6 | 3.012 | 0.371 | 53 | 32.6 | 29 | 31 |
0.8 | 3.102 | 0.295 | 61 | 36.2 | 32.4 | 34.8 |
1 | 3.142 | 0.244 | 72 | 38.4 | 34.8 | 36.8 |
2 | 3.074 | 0.130 | 131 | 41.8 | 44.4 | 44.8 |
3 | 2.865 | 0.089 | 192 | 42 | 46.8 | 46.8 |
4 | 2.642 | 0.067 | 254 | 42.6 | 50 | 49.6 |
6 | 2.396 | 0.045 | 377 | 43.8 | 51.6 | 50 |
8 | 2.274 | 0.034 | 500 | 44 | 54 | 51.6 |
10 | 2.193 | 0.028 | 616 | 44.2 | 54.8 | 51.6 |
Rewiring | MPL | CC | WC | EC | EC_Fixed | EC_Cuberoot |
---|---|---|---|---|---|---|
0 | undefined | 1.000 | 34 | 9.2 | 6.2 | 8.6 |
0.1 | 2.671 | 0.651 | 146 | 18.8 | 19.2 | 23.2 |
0.2 | 2.468 | 0.418 | 271 | 31 | 30 | 30 |
0.3 | 2.339 | 0.271 | 394 | 37.2 | 37 | 33.4 |
0.4 | 2.256 | 0.173 | 518 | 40.1 | 42.8 | 36.6 |
0.5 | 2.200 | 0.108 | 642 | 42 | 47.2 | 39.4 |
0.6 | 2.164 | 0.066 | 765 | 44 | 49.4 | 42.8 |
0.7 | 2.139 | 0.040 | 886 | 44 | 51.8 | 46 |
0.8 | 2.124 | 0.026 | 1009 | 44.2 | 52.2 | 48 |
0.9 | 2.116 | 0.021 | 1130 | 44 | 53.8 | 50.4 |
1 | 2.116 | 0.020 | 1250 | 44 | 55 | 51.4 |