In this paper relation between complex systems, information theory and the simplest models of neural networks are elucidated. Two different types of complex systems are distinguished, new complexity measure based on the graph theory defined, hierarchy of the correlation matrices introduced and connection with the correlation matrix memories and other types of neural models explained.