Daniel Eftekhari
PhD Student in Machine Learning
Department of Computer Science
University of Toronto
defte@cs.toronto.edu
CV/Resume
Publications
On the Importance of Gaussianizing Representations
Daniel Eftekhari, Vardan Papyan
ICML 2025: Paper | Code
Keywords:
mutual information game, gaussianization, power transform, noise robustness, normalization layer, deep learning, information theory
Abstract:
The normal distribution plays a central role in information theory – it is at the same time the best-case signal and worst-case noise distribution, has the greatest representational capacity of any distribution, and offers an equivalence between uncorrelatedness and independence for joint distributions. Accounting for the mean and variance of activations throughout the layers of deep neural networks has had a significant effect on facilitating their effective training, but seldom has a prescription for precisely what distribution these activations should take, and how this might be achieved, been offered. Motivated by the information-theoretic properties of the normal distribution, we address this question and concurrently present normality normalization: a novel normalization layer which encourages normality in the feature representations of neural networks using the power transform and employs additive Gaussian noise during training. Our experiments comprehensively demonstrate the effectiveness of normality normalization, in regards to its generalization performance on an array of widely used model and dataset combinations, its strong performance across various common factors of variation such as model width, depth, and training minibatch size, its suitability for usage wherever existing normalization layers are conventionally used, and as a means to improving model robustness to random perturbations.
Teaching Assistantships
- CSC412/CSC2506 (2024) Probabilistic Learning and Reasoning
- STA414/STA2104 (2023) Statistical Methods for Machine Learning II
- STA314 (2022) Statistical Methods for Machine Learning I
Courses
- CSC2541 (2024) Topics in Machine Learning: Generative AI for Images
- ECE1508 (2024) Multiuser Information Theory
- CSC2412 (2023) Algorithms for Private Data Analysis
- ECE1502 (2022) Information Theory
Awards
- Graduate Scholarship in Science & Technology (2024)
- Canada Graduate Scholarship M NSERC (2023)