Technical Foundations of Cat.IA
Cat.IA draws upon several complementary mathematical and computational frameworks. The appropriate framework depends on the structure of the problem: prediction, detection, uncertainty quantification, sequential decision-making or generative modelling. This page introduces the main technical foundations underlying our approach. For recent mathematical developments and research-oriented discussions, see our
AI & Math Blog.
The current technology behind machine learning (ML) and artificial intelligence (AI) relies on the use of advanced algorithms and computational power for handling very large datasets to train models, based on very large neural networks, with up to trillions of hyperparameters, to enable intelligent behavior.
The empirical astonishing performance of these massive neural networks is not fortuitous, but has strong mathematical foundations: these large learning systems verify two very useful and powerful properties, demonstrated by Belkin and its co-authors and exposed in the
AI & Math Blog.
The set of learning systems used in Cat.IA-AI.Innovations draws upon three classes of learning models:
- The supervised, semi-supervised, unsupervised, and self-supervised learning models. These models leverage various levels of labeled data to learn patterns and make predictions.
- The reinforcement learning (RL) models. RL models excel in decision-making tasks, learning through trial and error and optimizing their actions based on rewards. They are backed by a robust mathematical framework, providing strong guarantees on their performance.
- The generative large language models (LLMs), which are increasingly powerful tools for text generation. LLM have now reached a level of sophistication where they can assist in proving advanced mathematical theorems, a milestone exemplified by Terence Tao’s (2026) works on Bernstein’s inequality for functions of exponential type, or the Navier–Stokes Millennium Prize problem, see the section Solving Mathematical Problems in the
companion page of the AI & Math Blog.
Supervised, unsupervised and self-supervised learning
Supervised Learning
Supervised learning is the simplest learning method.
We have a dataset $ \mathcal{D} = \{(x_1, y_1), (x_2, y_2), \ldots, (x_N, y_N)\} $, which consists of a collection of $N$ pairs of inputs, $x_i$ or features, and outputs $ y_i$, written in matrix form as $ \mathcal{D} =(X,Y)$. The output $Y$ might be discrete, continuous, a set of labels in the case of the classification problem, etc.
The goal of supervised learning is to approximate the function $f$ that maps $X$ to $Y$, i.e., $Y=f(X)$, by learning this function from the dataset $\mathcal{D}$.
The estimation of the function $f$ is obtained by minimizing a
loss function:
$$
\min_{f \in \mathcal{H}} \mathcal{L}(Y,X;\theta).
$$
where $\mathcal{H}$ is the space of candidate functions, $\theta \in \Theta$ the set of hyperparameters to be estimated.
The function $f$ is approximated by using neural networks, which are universal approximators.
The hyperparameters are adjusted on a training set, the model is then tested and validated on different datasets.
Further details on the minimization of the loss function, and the recent advances in this field for very large systems are given in the
AI & Math Blog.
Unsupervised & Semi-supervised Learning
However, labels are either costly to obtain, or at least partially available. Furthermore, errors in the labels affect the result of supervised learning
and require either to use specific corrective tools (Northcutt et al., 2021), or to design alternative methods for estimating the model without depending on labels.
Unsupervised learning takes into account the internal structure/organisation of the data, and does not need the set of labels. This approach is similar to the non-parametric approach in statistics which 'let the data speak for themselves'.
Semi-supervised learning is the intermediate case between supervised and unsupervised learning: the learning procedure uses first the labelled data while leveraging a larger amount of unlabeled data, which strongly enhance the accuracy of the model.
Self-supervised Learning
In self-supervised learning, the model generates its own supervisory signals from the underlying structure/organisation of the data. The model learns meaningful representation or features of the data using e.g., pretext tasks, contrastive learning, and redundancy reduction (Zbontar et al., 2021).
Reinforcement learning
Reinforcement learning (RL) is a fundamental and specific learning method.
DeepSeek-R1's efficiency, a hybrid model combining RL and LLMs, will likely accelerate the adoption of RL methods, see also Xu
et al. (2025) and the survey by Zhang
et al. (2025).
In the standard supervised, unsupervised, self-supervised frameworks, the learner receives the data and learns a model from them.
In the RL framework, the learner is more active as he/she is an
agent interacting in an unsupervised manner with its
environment through a sequence of
actions. In response to the action just taken, the agent receives two types of information through a standard feedback mechanism:
- Its current state in the environment,
- A real-valued immediate reward for its actions. In more advanced frameworks, future or long-term reward are provided. The trade-off between immmediate and future reward is controled by the discount factor $\gamma \in (0,1)$.
The agent takes into account this feedback for its future actions.
The objective of the agent is to maximize its reward and thus to select the best course of actions, the
policy, for maximizing the cumulative reward over time. If the environment is known, this is a
planning problem, whilst if the environment is unknow, this is a
learning problem: the agent learns a sequence of decisions, i.e., a policy.
In the latter case, the agent balances the trade-off between the exploration, i.e., obtaining more information on the environmemt and rewards, and the exploitation of the information already collected for optimizing the rewards.
Markov Decision Process
The standard framework for formalizing the reinforcement learning is the Markov decision process (MDP), introduced by Bellman (1957), and defined by $M=(\mathcal{S,A,P},r,\gamma )$ where $\mathcal{S,A}$ are respectively the state space and the action space, which might be infinite, $r : \mathcal{S\times A} \rightarrow [0,1]$ is the reward function, i.e., $r(s,a)$ is the immediate reward upon choosing the action $a \in \mathcal{A}$ while in the state $s \in \mathcal{S}.$
The probability transition kernel is defined as $\mathcal{P}: \mathcal{S \times A} \rightarrow \Delta(S)$, where $\mathcal{P}(s^{'}|s,a)$ is the probability of transiting from state $s$ to state $s^{'}$ when $a$ is selected; $\Delta(S)$ is the probability simplex over $\mathcal{S}$. In the generative case, the probability transition kernel $\mathcal{P}$ is empirically estimated from the data.
The model is Markovian since the transition and reward probabilities depend only on the current state $s$ and not the entire history of states and actions. In the standard tabular case, the cardinalities $|\mathcal{S}|$ and $|\mathcal{A}|$, i.e., the number of states and actions respectively, are finite.
Optimal Policy
A deterministic policy $\pi$ is a mapping $\pi: \mathcal{S}\rightarrow \mathcal{A}$.
The objective of the agent is to find the optimal policy $\pi^\star$ maximizing the value function $V^\pi(s)$, which is the expected discounted total reward starting from the initial state $s^0=s$ , i.e.,
\begin{eqnarray*}
\pi^\star &:= &\arg \max_\pi V^\pi(s),\\
V^\pi(s)& := &E \Big[ \sum_{t=0}^\infty \gamma^t r(s^t, a^t)\big | s^0 = s \Big ].
\end{eqnarray*}
The corresponding action value function $Q^\pi: \mathcal{S\times A} \rightarrow \mathbb{R}$ of a policy $\pi$ is defined
as
$$
Q^\pi(s,a) := E \Big [ \sum_{t=0}^\infty \gamma^t r(s^t, a^t) \big | (s^0, a^0)=(s, a) \Big ],
$$
$\forall (s,a) \in \mathcal{S \times A}$. There exists an optimal policy $\pi^\star$ that simultaneoulsy maximizes $V^\pi(s) \; \forall s \in \mathcal{S}$, and $Q^\pi(s,a) \; \forall (s,a) \in \mathcal {S \times A}$.
Policy gradient (PG) methods offer an alternative approach to estimating optimal policies compared to methods based on action-value functions $Q^\pi(s, a)$.
PG methods assume that the policies $\pi_\theta$ are parameterized by a vector $\theta \in \Theta$, and search and find the optimal policy over a class of parameterized policies by maximizing a differentiable performance measure $J(\theta)$ with gradient ascent methods. PG methods often exhibit better convergence properties compared to argmax-based algorithms.
However, the convergence behavior within the class of PG methods varies significantly. For instance, while the convergence time of softmax-based PG methods can be exponential in some scenarios, natural policy gradient (NPG), the iteration complexity of which is dimension-free, appears to be more favorable in terms of convergence speed.
The determination of the optimal policy $\pi^\star_\theta$ by these PG methods and the recent advances in this field are presented in the
AI & Math Blog.
Transfer and Off-Policy Learning
Reinforcement Learning (RL) systems frequently exhibit limited generalization capabilities, often failing to transfer learned policies effectively to novel, unseen environments or tasks. Addressing this limitation, Transfer Learning (TL) has emerged as a promising paradigm that facilitates the reuse of knowledge acquired from one task to enhance learning efficiency in another. As emphasized by Taylor and Stone (2009), TL can substantially alleviate the learning burden by enabling the transfer of relevant representations, policies, or experiences across related domains. The most direct approach involves reusing solutions derived from tasks previously mastered by the agent; however, numerous studies have also explored the incorporation of external sources of knowledge, such as human demonstrations or guidance from other agents, to further improve adaptability and performance, see Da Silva
et al. (2017) and Zhu
et al. (2023) among others. Recent developments of TL and generalization properties of RL methods are surveyed in the
AI & Math Blog.
Flexibility and Reliability of the MDP Framework
This MDP framework is flexible enough to deal with several configurations, e.g., off-policy RL where data logged from a different policy are used in the exploration step, but rigorous enough as it allows to assess the reliability of the results, e.g., we obtain confidence intervals for the rewards. Furthermore, this setup allows to provides guarantees on the reliability of the algorithms used for estimating the optimal value function $V^\star := V^{\pi^\star}$,
or optimal action value function $Q^\star := Q^{\pi^\star}$.
The MDP framework is particularly valuable for analyzing the sample complexity of reinforcement learning algorithms. This means we can study how efficiently these algorithms learn from potentially limited data.
New results on the sample complexity and the minimum sample size for achieving the minimax sample complexity are given in the
AI & Math Blog.
This MDP framework has been extended to the non-tabular case, with a very large number of states, and/or continuous states, see e.g., Long and Han (2022, 2023). Li
et al. (2023) consider the case of continuous action space $\mathcal{A}$ in a nonlinear constrained optimization setting.
Using Lagrangian and Fenchel duality, they transform a nonlinear constrained convex minimization problem into an unconstrained primal-dual
optimization problem.
Generative AI
Generative AI is a part of automation technologies: it generates text, programming code and images using as database quite the exhaustive content of the web, using attention mechanisms, transformer architectures and autoregressive probabilistic models. While the first generations of large language models (LLMs) lacked deductive closure, aligning with Kahneman's (2011) System 1, recent hybrid and self-reasoning architectures, combining LLMs with search, tool use, and machine-checked formal verification, have closed this gap at the highest level of mathematics, including Millennium Prize problems.
For a detailed discussion of the reasoning properties of LLMs, Chain-of-Thought (CoT) and Tree of Thoughts (ToT) prompting, and LLM-assisted mathematical proofs, see the section
Natural Language Processing in the
companion page of the
AI & Math Blog.
References
-
Bahdanau, D., Cho, K., and Bengio, Y., 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
-
Bengio, Y., Courville, A. and Vincent, P., 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), 1798-1828.
-
Belkin, M., 2021. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation.
Acta Numerica 3, 203-248. MR4298218
-
Bellman, R., 1957. A Markovian decision process. Journal of mathematics and mechanics, 679-684.
MR0091859
-
Bellemare, M.G., Dabney, W. and Rowland, M., 2023. Distributional reinforcement learning. MIT Press.
MR4649766
-
Bertsekas D.P. and Tsitsiklis J.N., 1996. Neuro-dynamic programming. Athena Scientific, Belmont
-
Bishop, C. M., 2006.
Pattern recognition and machine learning.
Inf. Sci. Stat. Springer, New York.
MR2247587
-
Bishop, C.M. and Bishop, H., 2023. Deep learning: Foundations and concepts. Springer Nature.
MR4719738
-
Brown, T. B. et al., 2020. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165.
-
Da Silva, F.L., Glatt, R. and Costa, A.H.R., 2017. Simultaneously learning and advising in multiagent reinforcement learning. Proceedings of the 16th conference on autonomous agents and multiagent systems 1100-1108.
-
Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y. and Goldblum, M., 2023. A Cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210.
-
Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep learning. Adapt. Comput. Mach. Learn.
MIT Press, Cambridge, MA.
MR3617773
-
Goyal, A. and Bengio, Y., 2022. Inductive biases for deep learning of higher-level cognition.
Proc. R. Soc. A.478 no.2266, Paper No. 20210068, 35 pp.
MR4505795
-
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X. and Zhang, X., 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.
arXiv preprint arXiv:2501.12948,
-
Hastie, T., Tibshirani, R., Friedman, J., 2009. The elements of statistical learning.
Data mining, inference, and prediction. Second edition
Springer Ser. Statist. Springer, New York.
MR2722294
-
Hinton, G.E. and Salakhutdinov, R.R., 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507
MR2242509
-
Hinton, G., 2023. How to represent part-whole hierarchies in a neural network. Neural Computation, 35(3), 413-452.
MR4555492
-
Kahneman, D., 2011. Thinking fast and slow. Macmillan.
-
Lan, G., 2023. Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming 198, no.1, Ser. A, 1059-1106.
MR4550970
-
Li, G., Wei, Y., Chi, Y. and Chen, Y., 2024. Breaking the sample size barrier in model-based reinforcement learning with a generative model. Operation Research, 72, no.1, 203–221.
MR4705834
-
Li, Z., Liu, B., Yang, Z., Wang, Z. and Wang, M., 2023. Double duality: variational primal-dual policy optimization for constrained reinforcement learning. Journal of Machine Learning Research, 24(385), 1-43.
MR4720841
-
Long, J. and Han, J., 2022.
Perturbational complexity by distribution mismatch: a systematic analysis of reinforcement learning in reproducing kernel Hilbert space.Journal of Machine Learning 1, no. 1, 1–37.
MR4716372
-
Long, J. and Han, J. 2023.
Reinforcement learning with function approximation: from linear to nonlinear.
Journal of Machine Learning 2, no. 3, 161-193.
MR4734710
-
Mallat, S., 2012. Group invariant scattering. Communications on Pure and Applied Mathematics, 65(10), 1331-1398.
MR2957703
-
Mallat, S., 2016. Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A, 374(2065), 20150203.
-
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G. and Petersen, S., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.
-
Mohri, M., Rostamizadeh, A. and Talwalkar, A., 2018. Foundations of machine learning, second edition.
Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA.
MR3931734
-
Northcutt, C., Jiang, L. and Chuang, I., 2021. Confident learning: Estimating uncertainty in dataset labels.
Journal of Artificial Intelligence Research 70, 1373-1411.
MR4306614
-
Pan, S.J. and Yang, Q., 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359.
-
Plaat, A., van Duijn, M., Van Stein, N., Preuss, M., van der Putten, P. and Batenburg, K.J., 2025. Agentic large language models, a survey. Journal of Artificial Intelligence Research, 84 74. 74 p.
MR5022806
-
Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S. and Rutherford, E., 2021. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446.
-
Russell, S.J. and Norvig, P., 2016. Artificial intelligence: a modern approach. Pearson.
-
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A. and Chen, Y., 2017. Mastering the game of go without human knowledge. Nature, 550(7676), 354-359.
-
Szepesvàri, C., 2022. Algorithms for reinforcement learning, reprint of the 2010 original,
Synthesis Lectures on Artificial Intelligence and Machine Learning, 9, Springer, Cham,
MR4647095
-
Tao, T., 2026. Local Bernstein theory, and lower bounds for Lebesgue constants. arXiv preprint arXiv:2603.21453.
-
Tao, T., 2026. Mathematics in the Age of AI. arXiv preprint arXiv:2608.16753.
-
Taylor, M.E. and Stone, P., 2009. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7).
1633-1685.
MR2534874
-
Teyssière, G., 2022. Review of "Belkin, M. (2021). Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation". Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2023. Review of "Dixon, M.F., Halperin, I. and Bilokon, P. 2020. Machine learning in finance—from theory to practice." Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2023. Review of "Lan, G., 2023. Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes". Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2025. Review of "Long, J. and Han, J., 2022.
Perturbational complexity by distribution mismatch: a systematic analysis of reinforcement learning in reproducing kernel Hilbert space". Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2025. Review of "Li, G., Wei, Y., Chi, Y. and Chen, Y., 2024. Breaking the sample size barrier in model-based reinforcement learning with a generative model". Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2025. Review of "Li, Z., Liu, B., Yang, Z., Wang, Z. and Wang, M., 2023. Double duality: variational primal-dual policy optimization for constrained reinforcement learning." Mathematical Reviews, American Mathematical Society.
-
Teyssière, G., 2026. Review of "Zhang, L., Peng, Y., Liang, J., Yang, W. and Zhang, Z., 2025. Estimation and inference in distributional reinforcement learning". Mathematical Reviews, American Mathematical Society.
-
Van Hasselt, H., Guez, A. and Silver, D., 2016. Deep reinforcement learning with double Q-learning. Proceedings of the AAAI conference on artificial intelligence , 30, No. 1.
-
Vaswani, A. et al., 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
-
Wang, J., Gao, R., Zha, H., 2024. Reliable off-policy evaluation for reinforcement learning.
Operation Research, 72, no. 2, 699-716.
MR4677384
-
Wiatowski, T., Grohs, P. and Bölcskei, H., 2018. Energy propagation in deep convolutional neural networks. IEEE Transactions on Information Theory, 64(7), 4819-4842
MR3819343
-
Wiatowski, T. and Bölcskei, H., 2017. A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory, 64(3), 1845-1866.
MR3766318
-
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J. and Ouyang, T., 2025. Toward large reasoning models: A survey of reinforced reasoning with large language models. Patterns, 6(10).
-
Yamaguchi, S.Y., Kanai, S., Kumagai, A., Chijiwa, D. and Kashima, H., 2025. Transfer learning with pre-trained conditional generative models. Machine Learning, 114(4), p.96.
MR4867330
-
Zbontar, J., Jing, L., Misra, I., LeCun, Y. and Deny, S., 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. Proceedings of the 38th International Conference on Machine Learning, 139, 12310-12320.
-
Zhang, L., Peng, Y., Liang, J., Yang, W. and Zhang, Z., 2025. Estimation and inference in distributional reinforcement learning. The Annals of Statistics, 53(5), pp.1987-2011.
MR4985257
-
Zhu, Z., Lin, K., Jain, A.K. and Zhou, J., 2023. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 13344-13362.
Updated September 2026.