Source author record

Haiping Huang

Haiping Huang appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

cond-mat.stat-mech cond-mat.dis-nn Machine Learning Neurons and Cognition Information Theory math.IT Biological Physics eess.SP Neural and Evolutionary Computing nlin.CD physics.soc-ph

Catalog footprint

What is connected

24works

11topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2022arXiv

Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

Binary perceptron is a fundamental model of supervised learning for the non-convex optimization, which is a root of the popular deep learning. Binary perceptron is able to achieve a classification of random high-dimensional data by computing the marginal probabilities of binary synapses. The relationship between the algorithmic instability and the equilibrium analysis of the model remains elusive. Here, we establish the relationship by showing that the instability condition around the algorithmic fixed point is identical to the instability for breaking the replica symmetric saddle point solution of the free energy function. Therefore, our analysis would hopefully provide insights towards other learning systems in bridging the gap between non-convex learning dynamics and statistical mechanics properties of more complex neural networks.

preprint2020arXiv

Classification and Recognition of Encrypted EEG Data Neural Network

With the rapid development of Machine Learning technology applied in electroencephalography (EEG) signals, Brain-Computer Interface (BCI) has emerged as a novel and convenient human-computer interaction for smart home, intelligent medical and other Internet of Things (IoT) scenarios. However, security issues such as sensitive information disclosure and unauthorized operations have not received sufficient concerns. There are still some defects with the existing solutions to encrypted EEG data such as low accuracy, high time complexity or slow processing speed. For this reason, a classification and recognition method of encrypted EEG data based on neural network is proposed, which adopts Paillier encryption algorithm to encrypt EEG data and meanwhile resolves the problem of floating point operations. In addition, it improves traditional feed-forward neural network (FNN) by using the approximate function instead of activation function and realizes multi-classification of encrypted EEG data. Extensive experiments are conducted to explore the effect of several metrics (such as the hidden neuron size and the learning rate updated by improved simulated annealing algorithm) on the recognition results. Followed by security and time cost analysis, the proposed model and approach are validated and evaluated on public EEG datasets provided by PhysioNet, BCI Competition IV and EPILEPSIAE. The experimental results show that our proposal has the satisfactory accuracy, efficiency and feasibility compared with other solutions.

preprint2020arXiv

Statistical physics of unsupervised learning with prior knowledge in neural networks

Integrating sensory inputs with prior beliefs from past experiences in unsupervised learning is a common and fundamental characteristic of brain or artificial neural computation. However, a quantitative role of prior knowledge in unsupervised learning remains unclear, prohibiting a scientific understanding of unsupervised learning. Here, we propose a statistical physics model of unsupervised learning with prior knowledge, revealing that the sensory inputs drive a series of continuous phase transitions related to spontaneous intrinsic-symmetry breaking. The intrinsic symmetry includes both reverse symmetry and permutation symmetry, commonly observed in most artificial neural networks. Compared to the prior-free scenario, the prior reduces more strongly the minimal data size triggering the reverse symmetry breaking transition, and moreover, the prior merges, rather than separates, permutation symmetry breaking phases. We claim that the prior can be learned from data samples, which in physics corresponds to a two-parameter Nishimori constraint. This work thus reveals mechanisms about the influence of the prior on unsupervised learning.

preprint2020arXiv

Variational mean-field theory for training restricted Boltzmann machines with binary synapses

Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses and neural activity together is still lacking. The computational obstacle originates from the discrete nature of synapses, and complex interactions among these three essential elements of learning. Here, we propose a variational mean-field theory in which the distribution of synaptic weights is considered. The unsupervised learning can then be decomposed into two intertwined steps: a maximization step is carried out as a gradient ascent of the lower-bound on the data log-likelihood, in which the synaptic weight distribution is determined by updating variational parameters, and an expectation step is carried out as a message passing procedure on an equivalent or dual neural network whose parameter is specified by the variational parameters of the weight distribution. Therefore, our framework provides insights on how data (or sensory inputs), synapses and neural activities interact with each other to achieve the goal of extracting statistical regularities in sensory inputs. This variational framework is verified in restricted Boltzmann machines with planted synaptic weights and handwritten-digits learning.

preprint2020arXiv

Weakly-correlated synapses promote dimension reduction in deep neural networks

By controlling synaptic and neural correlations, deep learning has achieved empirical successes in improving classification performances. How synaptic correlations affect neural correlations to produce disentangled hidden representations remains elusive. Here we propose a simplified model of dimension reduction, taking into account pairwise correlations among synapses, to reveal the mechanism underlying how the synaptic correlations affect dimension reduction. Our theory determines the synaptic-correlation scaling form requiring only mathematical self-consistency, for both binary and continuous synapses. The theory also predicts that weakly-correlated synapses encourage dimension reduction compared to their orthogonal counterparts. In addition, these synapses slow down the decorrelation process along the network depth. These two computational roles are explained by the proposed mean-field equation. The theoretical predictions are in excellent agreement with numerical simulations, and the key features are also captured by a deep learning with Hebbian rules.

preprint2016arXiv

Clustering of neural codewords revealed by a first-order phase transition

A network of neurons in the central nervous system collectively represents information by its spiking activity states. Typically observed states, i.e., codewords, occupy only a limited portion of the state space due to constraints imposed by network interactions. Geometrical organization of codewords in the state space, critical for neural information processing, is poorly understood due to its high dimensionality. Here, we explore the organization of neural codewords using retinal data by computing the entropy of codewords as a function of Hamming distance from a particular reference codeword. Specifically, we report that the retinal codewords in the state space are divided into multiple distinct clusters separated by entropy-gaps, and that this structure is shared with well-known associative memory networks in a recallable phase. Our analysis also elucidates a special nature of the all-silent state. The all-silent state is surrounded by the densest cluster of codewords and located within a reachable distance from most codewords. This codeword-space structure quantitatively predicts typical deviation of a state-trajectory from its initial state. Altogether, our findings reveal a non-trivial heterogeneous structure of the codeword-space that shapes information representation in a biological network.

preprint2016arXiv

Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition

Unsupervised neural network learning extracts hidden features from unlabeled training data. This is used as a pretraining step for further supervised learning in deep networks. Hence, understanding unsupervised learning is of fundamental importance. Here, we study the unsupervised learning from a finite number of data, based on the restricted Boltzmann machine learning. Our study inspires an efficient message passing algorithm to infer the hidden feature, and estimate the entropy of candidate features consistent with the data. Our analysis reveals that the learning requires only a few data if the feature is salient and extensively many if the feature is weak. Moreover, the entropy of candidate features monotonically decreases with data size and becomes negative (i.e., entropy crisis) before the message passing becomes unstable, suggesting a discontinuous phase transition. In terms of convergence time of the message passing algorithm, the unsupervised learning exhibits an easy-hard-easy phenomenon as the training data size increases. All these properties are reproduced in an approximate Hopfield model, with an exception that the entropy crisis is absent, and only continuous phase transition is observed. This key difference is also confirmed in a handwritten digits dataset. This study deepens our understanding of unsupervised learning from a finite number of data, and may provide insights into its role in training deep networks.

preprint2015arXiv

Advanced Mean Field Theory of Restricted Boltzmann Machine

Learning in restricted Boltzmann machine is typically hard due to the computation of gradients of log-likelihood function. To describe the network state statistics of the restricted Boltzmann machine, we develop an advanced mean field theory based on the Bethe approximation. Our theory provides an efficient message passing based method that evaluates not only the partition function (free energy) but also its gradients without requiring statistical sampling. The results are compared with those obtained by the computationally expensive sampling based method.

preprint2015arXiv

Effects of hidden nodes on network structure inference

Effects of hidden nodes on inference quality of observed network structure are explored based on a disordered Ising model with hidden nodes. We first study analytically small systems consisting of a few nodes, and find that the magnitude of the effective coupling grows as the coupling strength from the hidden common input nodes increases, while the field strength of the input node has opposite effects. Insights gained from analytic results of small systems are confirmed in numerical simulations of large systems. We also find that the inference quality deteriorates as the number of hidden nodes increases. Furthermore, increasing field variance of hidden nodes improves the inference quality of the effective couplings, but worsens the quality for the effective fields. In addition, attenuated coupling strengths involved in at least one hidden node lead to high quality of coupling inference.

preprint2015arXiv

Structure of attractors in randomly connected networks

The deterministic dynamics of randomly connected neural networks are studied, where a state of binary neurons evolves according to a discreet-time synchronous update rule. We give a theoretical support that the overlap of systems' states between the current and a previous time develops in time according to a Markovian stochastic process in large networks. This Markovian process predicts how often a network revisits one of previously visited states, depending on the system size. The state concentration probability, i.e., the probability that two distinct states co-evolve to the same state, is utilized to analytically derive various characteristics that quantify attractors' structure. The analytical predictions about the total number of attractors, the typical cycle length, and the number of states belonging to all attractive cycles match well with numerical simulations for relatively large system sizes.

preprint2014arXiv

Code optimization, frozen glassy phase and improved decoding algorithms for low-density parity-check codes

The statistical physics properties of low-density parity-check codes for the binary symmetric channel are investigated as a spin glass problem with multi-spin interactions and quenched random fields by the cavity method. By evaluating the entropy function at the Nishimori temperature, we find that irregular constructions with heterogeneous degree distribution of check (bit) nodes have higher decoding thresholds compared to regular counterparts with homogeneous degree distribution. We also show that the instability of the mean-field calculation takes place only after the entropy crisis, suggesting the presence of a frozen glassy phase at low temperatures. When no prior knowledge of channel noise is assumed (searching for the ground state), we find that a reinforced strategy on normal belief propagation will boost the decoding threshold to a higher value than the normal belief propagation. This value is close to the dynamical transition where all local search heuristics fail to identify the true message (codeword or the ferromagnetic state). After the dynamical transition, the number of metastable states with larger energy density (than the ferromagnetic state) becomes exponentially numerous. When the noise level of the transmission channel approaches the static transition point, there starts to exist exponentially numerous codewords sharing the identical ferromagnetic energy.

preprint2014arXiv

Dynamics of asymmetric kinetic Ising systems revisited

The dynamics of an asymmetric kinetic Ising model is studied. Two schemes for improving the existing mean-field description are proposed. In the first scheme, we derive the formulas for instantaneous magnetization, equal-time correlation, and time-delayed correlation, considering the correlation between different local fields. To derive the time-delayed correlation, we emphasize that the small correlation assumption adopted in previous work [M. Mézard and J. Sakellariou, J. Stat. Mech., L07001 (2011)] is in fact not required. To confirm the inference efficiency of our method, we perform extensive simulations on single instances with either temporally constant external driving fields or sinusoidal external fields. In the second scheme, we develop an improved mean-field theory for instantaneous magnetization prediction utilizing the notion of the cavity system in conjunction with a perturbative expansion approach. Its efficiency is numerically confirmed by comparison with the existing mean-field theory when partially asymmetric couplings are present.

preprint2014arXiv

Origin of the computational hardness for learning with binary synapses

Supervised learning in a binary perceptron is able to classify an extensive number of random patterns by a proper assignment of binary synaptic weights. However, to find such assignments in practice, is quite a nontrivial task. The relation between the weight space structure and the algorithmic hardness has not yet been fully understood. To this end, we analytically derive the Franz-Parisi potential for the binary preceptron problem, by starting from an equilibrium solution of weights and exploring the weight space structure around it. Our result reveals the geometrical organization of the weight space\textemdash the weight space is composed of isolated solutions, rather than clusters of exponentially many close-by solutions. The point-like clusters far apart from each other in the weight space explain the previously observed glassy behavior of stochastic local search heuristics.

preprint2014arXiv

The network source location problem: ground state energy, entropy and effects of freezing

Ground state entropy of the network source location problem is evaluated at both the replica symmetric level and one-step replica symmetry breaking level using the entropic cavity method. The regime that is a focus of this study, is closely related to the vertex cover problem with randomly quenched covered nodes. The resulting entropic message passing inspired decimation and reinforcement algorithms are used to identify the optimal location of sources in single instances of transportation networks. The conventional belief propagation without taking the entropic effect into account is also compared. We find that in the glassy phase the entropic message passing inspired decimation yields a lower ground state energy compared to the belief propagation without taking the entropic effect. Using the extremal optimization algorithm, we study the ground state energy and the fraction of frozen hubs, and extend the algorithm to collect statistics of the entropy. The theoretical results are compared with the extremal optimization results.

preprint2013arXiv

Adaptive Thouless-Anderson-Palmer approach to inverse Ising problems with quenched random fields

The adaptive Thouless-Anderson-Palmer equation is derived for inverse Ising problems in the presence of quenched random fields. We test the proposed scheme on Sherrington-Kirkpatrick, Hopfield, and random orthogonal models and find that the adaptive Thouless-Anderson-Palmer approach allows surprisingly accurate inference of quenched random fields whose distribution can be either Gaussian or bimodal, compared with other existing mean-field methods.

preprint2013arXiv

Entropy landscape of solutions in the binary perceptron problem

The statistical picture of the solution space for a binary perceptron is studied. The binary perceptron learns a random classification of input random patterns by a set of binary synaptic weights. The learning of this network is difficult especially when the pattern (constraint) density is close to the capacity, which is supposed to be intimately related to the structure of the solution space. The geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solution space. We evaluate the entropy at the annealed level as well as replica symmetric level and the mean field result is confirmed by the numerical simulations on single instances using the proposed message passing algorithms. From the first landscape (a random configuration as a reference), we see clearly how the solution space shrinks as more constraints are added. From the second landscape of solution-pairs, we deduce the coexistence of clustering and freezing in the solution space.

preprint2013arXiv

Sparse Hopfield network reconstruction with $\ell_{1}$ regularization

We propose an efficient strategy to infer sparse Hopfield network based on magnetizations and pairwise correlations measured through Glauber samplings. This strategy incorporates the $\ell_{1}$ regularization into the Bethe approximation by a quadratic approximation to the log-likelihood, and is able to further reduce the inference error of the Bethe approximation without the regularization. The optimal regularization parameter is observed to be of the order of $M^{-ν}$ where $M$ is the number of independent samples. The value of the scaling exponent depends on the performance measure. $ν\simeq0.5001$ for root mean squared error measure while $ν\simeq0.2743$ for misclassification rate measure. The efficiency of this strategy is demonstrated for the sparse Hopfield model, but the method is generally applicable to other diluted mean field models. In particular, it is simple in implementation without heavy computational cost.

preprint2012arXiv

Counting solutions from finite samplings

We formulate the solution counting problem within the framework of inverse Ising problem and use fast belief propagation equations to estimate the entropy whose value provides an estimate on the true one. We test this idea on both diluted models (random 2-SAT and 3-SAT problems) and fully-connected model (binary perceptron), and show that when the constraint density is small, this estimate can be very close to the true value. The information stored by the salamander retina under the natural movie stimuli can also be estimated and our result is consistent with that obtained by Monte Carlo method. Of particular significance is sizes of other metastable states for this real neuronal network are predicted.

preprint2011arXiv

Combined local search strategy for learning in networks of binary synapses

Learning in networks of binary synapses is known to be an NP-complete problem. A combined stochastic local search strategy in the synaptic weight space is constructed to further improve the learning performance of a single random walker. We apply two correlated random walkers guided by their Hamming distance and associated energy costs (the number of unlearned patterns) to learn a same large set of patterns. Each walker first learns a small part of the whole pattern set (partially different for both walkers but with the same amount of patterns) and then both walkers explore their respective weight spaces cooperatively to find a solution to classify the whole pattern set correctly. The desired solutions locate at the common parts of weight spaces explored by these two walkers. The efficiency of this combined strategy is supported by our extensive numerical simulations and the typical Hamming distance as well as energy cost is estimated by an annealed computation.

preprint2011arXiv

State sampling dependence of the Hopfield network inference

The fully connected Hopfield network is inferred based on observed magnetizations and pairwise correlations. We present the system in the glassy phase with low temperature and high memory load. We find that the inference error is very sensitive to the form of state sampling. When a single state is sampled to compute magnetizations and correlations, the inference error is almost indistinguishable irrespective of the sampled state. However, the error can be greatly reduced if the data is collected with state transitions. Our result holds for different disorder samples and accounts for the previously observed large fluctuations of inference error at low temperatures.

preprint2010arXiv

Learning by random walks in the weight space of the Ising perceptron

Several variants of a stochastic local search process for constructing the synaptic weights of an Ising perceptron are studied. In this process, binary patterns are sequentially presented to the Ising perceptron and are then learned as the synaptic weight configuration is modified through a chain of single- or double-weight flips within the compatible weight configuration space of the earlier learned patterns. This process is able to reach a storage capacity of $α\approx 0.63$ for pattern length N = 101 and $α\approx 0.41$ for N = 1001. If in addition a relearning process is exploited, the learning performance is further improved to a storage capacity of $α\approx 0.80$ for N = 101 and $α\approx 0.42$ for N=1001. We found that, for a given learning task, the solutions constructed by the random walk learning process are separated by a typical Hamming distance, which decreases with the constraint density $α$ of the learning task; at a fixed value of $α$, the width of the Hamming distance distributions decreases with $N$.

preprint2010arXiv

Message passing algorithms for the Hopfield network reconstruction: threshold behavior and limitation

The Hopfield network is reconstructed as an inverse Ising problem by passing messages. The applied susceptibility propagation algorithm is shown to improve significantly on other mean-field-type methods and extends well into the low temperature region. However, this iterative algorithm is limited by the nature of the supplied data. Its performance deteriorates as the data becomes highly magnetized, and this method finally fails in the presence of the frozen type data where at least two of its magnetizations are equal to one in absolute value. On the other hand, a threshold behavior is observed for the susceptibility propagation algorithm and the transition from good reconstruction to poor one becomes sharper as the network size increases.

preprint2009arXiv

Cavity approach to the Sourlas code system

The statistical physics properties of regular and irregular Sourlas codes are investigated in this paper by the cavity method. At finite temperatures, the free energy density of these coding systems is derived and compared with the result obtained by the replica method. In the zero temperature limit, the Shannon's bound is recovered in the case of infinite-body interactions while the code rate is still finite. However, the decoding performance as obtained by the replica theory has not considered the zero-temperature entropic effect. The cavity approach is able to consider the ground-state entropy. It leads to a set of evanescent cavity fields propagation equations which further improve the decoding performance, as confirmed by our numerical simulations on single instances. For the irregular Sourlas code, we find that it takes the trade-off between good dynamical property and high performance of decoding. In agreement with the results found from the algorithmic point of view, the decoding exhibits a first order phase transition as occurs in the regular code system with three-body interactions. The cavity approach for the Sourlas code system can be extended to consider first-step replica-symmetry-breaking.

preprint2009arXiv

Reconstructing the Hopfield network as an inverse Ising problem

We test four fast mean field type algorithms on Hopfield networks as an inverse Ising problem. The equilibrium behavior of Hopfield networks is simulated through Glauber dynamics. In the low temperature regime, the simulated annealing technique is adopted. Although performances of these network reconstruction algorithms on the simulated network of spiking neurons are extensively studied recently, the analysis of Hopfield networks is lacking so far. For the Hopfield network, we found that, in the retrieval phase favored when the network wants to memory one of stored patterns, all the reconstruction algorithms fail to extract interactions within a desired accuracy, and the same failure occurs in the spin glass phase where spurious minima show up, while in the paramagnetic phase, albeit unfavored during the retrieval dynamics, the algorithms work well to reconstruct the network itself. This implies that, as a inverse problem, the paramagnetic phase is conversely useful for reconstructing the network while the retrieval phase loses all the information about interactions in the network except for the case where only one pattern is stored. The performances of algorithms are studied with respect to the system size, memory load and temperature, sample-to-sample fluctuations are also considered.

Haiping Huang

What is connected

Connect this record

See the researcher in context

Building this map preview

24 published item(s)

Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

Classification and Recognition of Encrypted EEG Data Neural Network

Statistical physics of unsupervised learning with prior knowledge in neural networks

Variational mean-field theory for training restricted Boltzmann machines with binary synapses

Weakly-correlated synapses promote dimension reduction in deep neural networks

Clustering of neural codewords revealed by a first-order phase transition

Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition

Advanced Mean Field Theory of Restricted Boltzmann Machine

Effects of hidden nodes on network structure inference

Structure of attractors in randomly connected networks

Code optimization, frozen glassy phase and improved decoding algorithms for low-density parity-check codes

Dynamics of asymmetric kinetic Ising systems revisited

Origin of the computational hardness for learning with binary synapses

The network source location problem: ground state energy, entropy and effects of freezing

Adaptive Thouless-Anderson-Palmer approach to inverse Ising problems with quenched random fields

Entropy landscape of solutions in the binary perceptron problem

Sparse Hopfield network reconstruction with $\ell_{1}$ regularization

Counting solutions from finite samplings

Combined local search strategy for learning in networks of binary synapses

State sampling dependence of the Hopfield network inference

Learning by random walks in the weight space of the Ising perceptron

Message passing algorithms for the Hopfield network reconstruction: threshold behavior and limitation

Cavity approach to the Sourlas code system

Reconstructing the Hopfield network as an inverse Ising problem