Seminar: Scientific Computing

Deep learning techniques have been applied with great success to many problems in machine learning. The objective of a deep learning model is to predict the output of a considered system. The parameters of such model are assigned through a learning process based on available data. In some applications, the reliability of a deep learning model, i.e. model uncertainty due to limited data or data error, is also required, e.g. under the context of Artificial Intelligence (AI) safety. Bayesian probability theory is a mathematical tool to reason about uncertainty. Applying Bayesian inference to deep learning techniques, one can build a model that can predict in a probabilistic way the system's outputs. In this seminar work, student will investigate the theory of Bayesian deep neural networks, and the inference techniques in these.

[1] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). [2] Gal, Yarin. "Uncertainty in deep learning." University of Cambridge (2016).

This seminar topic reviews numerical optimization algorithms in the context of maching learning applications.

[1] Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." SIAM Review 60.2 (2018): 223-311.

[2] Jordan, Michael I. "Dynamical, Symplectic and Stochastic Perspectives on Gradient-Based Optimization." University of California, Berkeley(2018).

An important step in building a neural network is a training process, when a set of training examples are passed through the neural network. Difference between the actual results and the desired results is described by a loss function. When it is big, then the neural network does not perform well. There are various methods for minimizing the loss function. One such efficient method is the stochastic gradient method. In this seminar topic the stochastic gradient method and its properties will be addressed.

[1] Bottou, L. (2010): Large-Scale Machine Learning with Stochastic Gradient Descent. In: Lechevallier Y., Saporta G. (eds) Proceedings of COMPSTAT'2010. Physica-Verlag HD: p. 177-186. [2] Chen, J., Luss, R. (2018): Stochastic Gradient Descent with Biased but Consistent Gradient Estimators.

[1]Cohen, N., Sharir, O., & Shashua, A. (2016, June). On the expressive power of deep learning: A tensor analysis. In Conference on Learning Theory (pp. 698-728).

Recent researches reveal the close connection between tensor decomposition and Deep Neural networks(DNN). In this topic we concentrate on why and how tensor decomposition techniques are utilised to greatly improve the efficiency of DNN and gives insight to the understanding and further development of the latter.

[1] Novikov, Alexander, et al. "Tensorizing neural networks." Advances in Neural Information Processing Systems. 2015.

[2] Cichocki, Andrzej. "Era of big data processing: A new approach via tensor networks and tensor decompositions." arXiv preprint arXiv:1403.2048(2014).

[3] Cichocki, Andrzej, et al. "Tensor decompositions for signal processing applications: From two-way to multiway component analysis." IEEE Signal Processing Magazine 32.2 (2015): 145-163.

[4] Max Jaderberg, Andrea Vedaldi,Andrew Zisserman, "Speeding up Convolutional Neural Networks with Low Rank Expansions", British Machine Vision Conference, 2014, arxiv.org/abs/1405.3866.

[5] Cheng Tai, Tong Xiao,Yi Zhang, Xiaogang Wang, Weinan E, "Convolutional neural networks with low-rank regularization", arxiv.org/abs/1511.06067.