The continuously increasing interest in intelligent autonomous systems underlines the need for new developments in cyber-physical systems that can learn, adapt, and reason. Towards this direction, we will formally analyze the properties of learning for inference, verification, and control of general systems, when time and computational resources are limited, and robustness and interpretability are prioritized. We will focus on the notion of progressive learning: an adaptive process that hierarchically approximates the solution of an optimal decision-making problem given real-time observations of a system and its environment. We will introduce the Online Deterministic Annealing (ODA) approach as a gradient-free stochastic optimization method to construct a learning model that progressively increases its complexity as needed, through an intuitive bifurcation phenomenon. We will study the properties of robustness and interpretability, and the importance of being able to control the performance-complexity trade-off in real time. Finally, we will discuss how these properties can be incorporated in the development of system identification and verification algorithms with a wide range of applications, from robotics and multi-agent systems to automatic theorem proving and software debugging.