Recent Advancements in Optimization Methods for Machine Learning I – Invited Special Session

B1L-D: Recent Advancements in Optimization Methods for Machine Learning I - Invited Special Session

Session Type: Lecture
Session Code: B1L-D
Location: Room 4
Date & Time: Thursday March 23, 2023 (09:00 - 10:00)
Chair: Nicolas Loizou
Track: 12

Paper No.	Paper Name	Authors	Abstract
3058	Understanding and Enriching the Algorithmic Reasoning Abilities in Deep Learning Models	Jingling Li, John Dickerson	Learning to reason is an essential step to achieving general intelligence. As the reasoning process can be described as a step-by-step algorithmic procedure, understanding and enriching the algorithmic reasoning capabilities has drawn increasing attention in deep learning communities. Therefore, this work starts from the algorithmic perspective to understand the essence of reasoning, building upon both theoretical and empirical evidence. To bridge algorithms and neural networks, we propose a framework, algorithmic alignment, which connects neural networks with algorithms in a novel manner and advances our understanding of how these two fields can work together to solve complex reasoning tasks. Intuitively, the algorithmic alignment framework evaluates how well a neural network\'s computation structure aligns with the algorithmic structure in a reasoning process. Utilizing algorithmic alignment, we are able to understand the limitations of machine learning models in the context of reasoning (e.g., generalization, extrapolation, and robustness) and know where to make improvements.
3039	ProxSkip for Stochastic Variational Inequalities: a Federated Learning Algorithm for Provable Communication Acceleration	Siqi Zhang, Nicolas Loizou	Recently Mishchenko et al. (2022) proposed and analyzed ProxSkip, a provably efficient method for minimizing the sum of a smooth function and an expensive nonsmooth proximable regularizer function. The main advantage of ProxSkip, is that in the federated learning (FL) setting, offers provably an effective acceleration of communication complexity. This work extends this approach to the more general regularized variational inequality problems (VIP). In particular, we propose ProxSkip-VIP algorithm, which generalizes the original ProxSkip framework of Mishchenko et al. (2022) to VIP, and we provide convergence guarantees for a class of structured non-monotone problems. In the federated learning setting, we explain how our approach achieves acceleration in terms of the communication complexity over existing state-of-the-art FL algorithms.
3042	Efficient Machine Learning at the Edge in Parallel	Furong Huang	Since the beginning of the digital age, the size and quantity of data sets have grown exponentially because of the proliferation of data captured by mobile devices, vehicles, cameras, microphones, and other internet of things (IoT) devices. Given this boom in personal data, major advances in areas such as healthcare, natural language processing, computer vision, and more have been made with the use of deep learning. Federated Learning (FL) is an increasingly popular setting to train powerful deep neural net- works with data derived from an assortment of devices. It is a framework for use-cases involving training machine learning models on edge devices without transmitting the collected data to a central server. In this talk, I will address some major challenges of efficient machine learning at the edge in parallel. Model efficiency, data efficiency and learning paradigm efficiency will be discussed respectively. As some highlights, I will introduce our recent progress on model compression via tensor representation, data efficiency through the lens of generalization analysis and a decentralized federated learning framework via wait-free model communication.