Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Tue, 15 Oct 2024 00:00:00 +0000

Authors: Minghui Chen, Meirui Jiang, Xin Zhang, Qi Dou, Zehua Wang, Xiaoxiao Li

Published in: The Thirty-Eighth Conference on Neural Information Processing Systems (NeurIPS 2024)

Abstract

Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called “Local Superior Soups.” Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets.

Benchmarks for Corruption Invariant Person Re-Identification

Wed, 01 Dec 2021 00:00:00 +0000

Authors: Minghui Chen (Equal contribution), Zhiqiang Wang (Equal contribution), Feng Zheng

Published in: Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2021)

Abstract

When deploying person re-identification (ReID) model in safety-critical applications, it is pivotal to understanding the robustness of the model against a diverse array of image corruptions. However, current evaluations of person ReID only consider the performance on clean datasets and ignore images in various corrupted scenarios. In this work, we comprehensively establish five ReID benchmarks for learning corruption invariant representation. In the field of ReID, we are the first to conduct an exhaustive study on corruption invariant learning in single- and cross-modality datasets, including Market-1501, CUHK03, MSMT17, RegDB, SYSU-MM01. After reproducing and examining the robustness performance of 21 recent ReID methods, we have some observations, 1) transformer-based models are more robust towards corrupted images, compared with CNN-based models, 2) increasing the probability of random erasing (a commonly used augmentation method) hurts model corruption robustness, 3) cross-dataset generalization improves with corruption robustness increases. By analyzing the above observations, we propose a strong baseline on both single- and cross-modality ReID datasets which achieves improved robustness against diverse corruptions. Our codes are available on github( https://github.com/MinghuiChen43/CIL-ReID).

NeurIPS on Minghui Chen

Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Abstract

Benchmarks for Corruption Invariant Person Re-Identification

Abstract