What it means by learning in a neural network: easing the knot

What constitutes learning in humans and machines remains puzzling despite the unprecedented growth witnessed in both. Starting with a Perceptron and, in subsequent interrogation of multilayer perceptron (MLP), this paper revisits learning and searches for new learning signatures in artificial neural networks (ANN). Precisely, we consider the transport of the initial weight distribution to its final form while optimally balancing information entropy and the statistical complexity. Projecting the learning dynamics on the complexity-entropy plane helps identify optimal information organization internally within the network. As observed, training neural networks guided by complexity-entropy trade-off improves model performance and reproducibility. In continuation, we further assess depth dependence and information flow using entropy differences, the KL-divergence of the weight distributions between successive layers, and mutual information (MI) between the input, hidden layers, and output. Our findings indicate dependency between learning rate, neural network depth, and mutual information. Insights obtained so far in our ongoing analysis are relevant to applications ranging from explainable AI to understanding brain function.