ICLR on Minghui Chen

Textual Equilibrium Propagation for Deep Compound AI Systems

Wed, 28 Jan 2026 00:00:00 +0000

Authors: Minghui Chen, Wenlong Deng, James Zou, Han Yu, Xiaoxiao Li

Published in: Accepted to The Fourteenth International Conference on Learning Representations (ICLR 2026)

Abstract

Large language models (LLMs) are increasingly deployed as part of compound AI systems that coordinate multiple modules, such as retrievers, tools, and verifiers, over long-horizon workflows. Recent approaches that propagate textual feedback globally, such as TextGrad, make it feasible to optimize such pipelines, but we find that performance degrades as system depth grows.

Can Textual Gradient Work in Federated Learning?

Fri, 24 Jan 2025 00:00:00 +0000

Authors: Minghui Chen, Ruinan Jin, Wenlong Deng, Yuanyuan Chen, Zhi Huang, Han Yu, Xiaoxiao Li

Published in: The Thirteenth International Conference on Learning Representations (ICLR 2025)

Abstract

Recent studies highlight the promise of LLM-based prompt optimization, especially with TextGrad, which automates “differentiation” via texts and backpropagates textual feedback provided by LLMs. This approach facilitates training in various real-world applications that do not support numerical gradient propagation or loss calculation. It opens new avenues for optimization in decentralized, resource-constrained environments, suggesting that users of black-box LLMs (e.g., ChatGPT) could enhance components of LLM agentic systems (such as prompt optimization) through collaborative paradigms like federated learning (FL).

Revisiting Delta-Parameter Pruning For Fine-Tuned Models

Thu, 23 Jan 2025 00:00:00 +0000

Authors: Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, Christos Thrampoulidis

Published in: The Thirteenth International Conference on Learning Representations (ICLR 2025, Spotlight)

Abstract

Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters—the differences between fine-tuned and pre-trained model weights—while typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure - (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters. To address these, we develop two algorithmic improvements - (1) DARq, which modifies the rescaling factor in DARE, leading to significant performance gains at high pruning rates (e.g., >30% on COLA and SST2 for encoder models, with even larger improvements in decoder models), and (2) AdamR, an in-training modification that incorporates appropriate Delta regularization before applying DPP. We also demonstrate that DARq can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-based pruning techniques within DPP, demonstrating that they outperform random-based methods when delta parameters are large. Through this comprehensive study, we develop a pipeline for selecting the most appropriate DPP method under various practical scenarios.