Imerge修改（Introduction）

1 introduction

%背景介绍

近年来，基于深度神经网络（DNN）的深度学习发展迅速。然而，随着数据规模和网络模型复杂度的指数级增长，在单一设备上完成训练已变得极其耗时。例如，在 ImageNet 数据集上使用单张 Nvidia Tesla V100 GPU 训练 ResNet-50 模型可能需要超过两天的时间 \cite{wang2019benchmarking}。因此，分布式深度学习的需求正稳步增长。

In recent years, deep learning based on deep neural networks (DNNs) has developed rapidly. However, with the exponential growth of data volume and network model complexity, training on a single device has become extremely time-consuming. For example, training the ResNet-50 model on the ImageNet dataset using a single Nvidia Tesla V100 GPU can take more than two days \cite{wang2019benchmarking}. Therefore, the demand for distributed deep learning has been steadily increasing.

修改：

近年来，基于深度神经网络（DNN）的深度学习快速发展，并被广泛应用于多个领域 [1]–[6]。然而，随着数据规模和模型复杂度的指数级增长，在单一设备上完成训练变得极为耗时 [7]。因此，为加速 DNN 的训练，分布式深度学习的需求日益增长 [8]。
在分布式训练策略中，数据并行已成为最主流的方法 [9-12]。其基本思想是将训练数据划分给多个计算节点，每个节点独立执行前向与反向传播，并计算局部梯度。随后，这些梯度需要通过参数同步在所有节点之间进行聚合，以更新全局模型参数 [13], [14]。然而，在大规模分布式训练中，梯度聚合往往带来密集的通信开销，成为限制系统可扩展性的关键瓶颈 [13], [15]。

In recent years, deep learning based on deep neural networks (DNNs) has developed rapidly and has been widely applied in various domains~\cite{krizhevsky2012imagenet,simonyan2014very,graves2013speech,amodei2016deep,deng2018deep,collobert2008unified}. However, with the exponential growth of data volume and model complexity, training on a single device has become extremely time-consuming~\cite{wang2019benchmarking}. As a result, distributed deep learning has become increasingly important to accelerate the training of DNNs~\cite{ben2019demystifying}.

Among distributed training strategies, data parallelism has emerged as the predominant approach~\cite{shallue2019measuring,dean2012large,dekel2012optimal,meng2019convergence}. The basic idea is to partition the training data across multiple workers, where each worker performs forward and backward propagation independently and computes local gradients. These gradients must then be aggregated through parameter synchronization across all workers to update the global model parameters~\cite{shi2021exploiting,shi2018performance}. However, in large-scale distributed training, gradient aggregation often introduces intensive communication overhead, which becomes a key bottleneck that limits scalability~\cite{shi2021exploiting,duan2022mercury}.

Therefore, parameter synchronization has become a critical factor affecting the performance of distributed deep learning.

%PS vs. All-Reduce,(引入两种架构，介绍PS优势)

在分布式深度学习中，有两种最流行的系统架构：All-Reduce架构和参数服务器架构，分别采用去中心化和中心化的设计思想，具体定义了计算节点间如何通信、与谁通信等一系列完整的参数同步规则。

在All-Reduce 架构中，参数同步的过程直接在 worker 节点内部执行，通过去中心化的点对点通信实现全局梯度聚合。Allreduce 有多种实现方式，其中最为著名的是 ring allreduce，它通过将张量分成小消息并以流水线方式同时交换这些消息来实现带宽优化。然而，它主要适用于传输少量的大消息，事实上对于一个分布式训练任务而言，传输大量的小消息是更为普遍的现象。通信次数和启动时延随集群规模线性增长。此外 ring 算法的链式通信结构也使其容灾能力较差，限制其扩展性。

相比之下，PS 架构采用集中式参数管理方式，在容错性、灵活性以及异构环境适配方面展现出独特优势。PS在逻辑上是一个中央服务器，它聚合来自worker的梯度，更新模型参数，并将最新的模型发回给worker，为系统实现提供了一个简单而灵活的框架。

---

修改版本（适当精简内容，与前面背景描述衔接）：

%PS与All-Reduce的对比

因此，在分布式深度学习中，参数同步是影响训练性能的核心环节。目前最常见的两类系统架构是 参数服务器（Parameter Server, PS） 和 All-Reduce。All-Reduce 架构通过去中心化的点对点通信实现全局梯度聚合，能够高效利用带宽。然而，它更适合少量大消息的传输，而在实际分布式训练中，大量小消息的通信更为常见，因而其优势受限。同时，其通信次数与启动延迟随集群规模线性增长，其链式结构在容错性上也较为薄弱，从而进一步限制了系统的可扩展性。相比之下，PS 架构采用集中式参数管理方式，在容错性、灵活性以及异构环境适配方面展现出独特优势。

In distributed deep learning, parameter synchronization is a critical factor that affects training performance. The two most common system architectures are the \textbf{Parameter Server (PS)} and \textbf{All-Reduce}. The All-Reduce architecture~\cite{awan2017s,hoefler2010toward,rabenseifner2004optimization,jia2018highly} performs global gradient aggregation through decentralized peer-to-peer communication, which enables efficient bandwidth utilization. However, it is more suitable for transmitting a small number of large messages, whereas in practical distributed training, the communication of numerous small messages is more common, which limits its advantages. Meanwhile, its communication rounds and startup latency grow linearly with cluster size, and its chain-based structure exhibits weak fault tolerance, further constraining system scalability. In contrast, the PS architecture\cite{dean2012large, smola2010architecture,li2013parameter,li2014communication } adopts a centralized parameter management approach, demonstrating unique advantages in fault tolerance, flexibility, and adaptability to heterogeneous environments.

---

%PS的问题

然而，原生的 PS 架构在设计上也存在不足。首先，其高度中心化的多对一通信模式容易造成带宽瓶颈与网络拥塞，使单个 PS 节点成为分布式系统扩展效率的关键瓶颈，从而显著限制整体的可扩展性与训练性能。其次，计算与通信之间的强串行关系也使得集群的资源利用率普遍不高。为了能更高效地发挥出多算力设备并行处理的优势，减少非必要的通信开销，研究参数服务器的通信优化问题具有重要的应用价值。

However, the original PS architecture also suffers from inherent limitations. Its highly centralized many-to-one communication pattern can lead to bandwidth bottlenecks and network congestion, making a single PS node the critical bottleneck that restricts system scalability and significantly degrades training performance.

修改版本（增加计算与通信的强串行关系问题）：

However, the original PS architecture exhibits inherent drawbacks.First, its highly centralized many-to-one communication pattern tends to incur bandwidth bottlenecks and network congestion, rendering a single PS node the critical factor that limits system scalability and impedes training performance. Second, the strong serialization between computation and communication often leads to suboptimal resource utilization across the cluster. Therefore, optimizing communication in PS-based systems is of practical significance for fully exploiting parallelism across multiple computing devices while reducing unnecessary communication overhead.

%相关工作：

最近的研究主要从 通信架构优化、通信压缩 和 通信调度 三个方面对参数服务器进行改进，以提升其通信效率和系统扩展性。

在 通信架构优化 方面，核心思想是改进参数服务器运行时遵循的逻辑拓扑。在默认的星型拓扑下，单一 PS 节点作为中心汇聚所有 worker 的梯度与参数更新，极易因带宽压力成为性能瓶颈。已有工作主要沿两条路径展开：一类方法通过增加 PS 节点数量实现横向扩展，如 BytePS；另一类方法通过引入多级服务器实现纵向扩展，以缓解跨层通信负载。

在 通信压缩 方面，研究者尝试通过减少传输数据量来降低通信开销。常见做法包括梯度量化，将高比特梯度转换为低比特表示，以及梯度稀疏化，只传输绝对值超过阈值的梯度。尽管这些方法能够显著减少通信量，但也可能导致模型精度下降，需要梯度补偿机制来弥补，同时压缩与解压缩过程本身会引入额外计算开销，从而在某些场景下削弱整体加速效果。

在 通信调度 方面，主要思想是打破计算与通信的串行关系，通过任务调度实现二者的重叠。DNN 训练通常可以抽象为有向无环图，其中计算（前向和反向传播）与通信（梯度聚合）占用的资源不同，因此通信可隐藏在计算过程中。在不影响模型收敛的前提下，已有方法分别探索了通信与反向传播阶段的重叠，以及通信与前向传播阶段的重叠，从而缩短一次迭代所需时间。

%相关工作：通信架构优化、通信压缩、通信调度
Recent research has primarily focused on three directions to optimize the Parameter Server (PS) architecture, 
namely \textbf{communication architecture optimization}, \textbf{communication compression}, and \textbf{communication scheduling}, 
with the aim of improving communication efficiency and system scalability.  

%通信架构优化
For \textbf{communication architecture optimization}, the core idea is to refine the logical topology followed by PS during runtime. 
In the default star topology\cite{kairouz2021advances}, a single PS node serves as the central aggregator for gradients and parameter updates from all workers, which easily becomes a performance bottleneck due to bandwidth pressure. 
Existing works mainly follow two approaches: one line of research increases the number of PS nodes to achieve horizontal scaling, as exemplified by BytePS\cite{jiang2020unified}; 
another introduces hierarchical PS designs\cite{wan2020rat,luo2020plink,yang2018tree,huang2021communication} to enable vertical scaling and alleviate cross-layer communication overhead.  

%通信压缩
In terms of \textbf{communication compression}, researchers aim to reduce the volume of transmitted data to lower communication costs. 
Typical techniques include gradient quantization\cite{seide20141, wen2017terngrad} , which converts high-precision gradients into low-bit representations, 
and gradient sparsification\cite{strom2015scalable, aji2017sparse, shi2020layer}, which transmits only gradients whose absolute values exceed a predefined threshold. 
Although these methods can significantly reduce communication overhead, they may lead to accuracy degradation\cite{grubic2018synchronous}, 
which often requires error compensation mechanisms to mitigate. 
Furthermore, the compression and decompression processes introduce additional computational overhead, 
which in some cases offsets the benefits of communication reduction.  

%通信调度
Regarding \textbf{communication scheduling}, the main idea is to break the strict serialization between computation and communication by overlapping the two. 
The training of deep neural networks (DNNs) can be abstracted as a directed acyclic graph (DAG), 
where computation (forward and backward propagation) and communication (gradient aggregation) utilize different resources. 
As a result, communication can be overlapped with computation without affecting convergence. 
Existing studies have explored overlapping communication with the backward propagation as well as with the forward propagation, 
thereby reducing the iteration time.  

首段（引出相关工作部分）修改：不要有过多加粗，改为

Recent research has primarily focused on three directions to optimize the Parameter Server (PS) architecture, namely (i) communication architecture optimization, (ii) communication compression, and (iii) communication scheduling, with the aim of improving communication efficiency and system scalability.

通信调度修改：

在通信调度方面，主要思想是打破计算与通信的串行关系，通过任务调度实现二者的重叠。在分布式深度学习的训练过程中，计算任务和通信任务可以用有向无环图
(DAG)来描述[]。其中计算任务（前向和反向传播）与通信任务（梯度聚合）占用的资源不同，因此通信可隐藏在计算过程中，从而隐藏部分通信开销[]。在不影响模型收敛的前提下，已有方法分别探索了通信与反向传播阶段的重叠，以及通信与前向传播阶段的重叠，从而缩短一次迭代所需时间。

Regarding \textbf{communication scheduling}, the main idea is to break the strict serialization between computation and communication by overlapping the two through task scheduling. 
In distributed deep learning training, computation tasks and communication tasks can be represented by a directed acyclic graph (DAG)\cite{shi2018dag}. 
Since computation (forward and backward propagation) and communication (gradient aggregation) utilize different resources, communication can be overlapped with computation to hide part of the communication overhead\cite{hashemi2019tictac,jayarajan2019priority,li2018pipe,narayanan2019pipedream,sergeev2018horovod,peng2019generic,shi2020communication}. 
Without affecting model convergence, existing methods have explored overlapping communication with the backward propagation as well as with the forward propagation, thereby reducing the iteration time.

%过渡

尽管在通信架构优化、通信压缩和通信调度方面已有许多研究工作，并在一定程度上缓解了参数服务器的通信瓶颈，但在中心化 PS 架构下仍显不足。特别是多对一的通信模式依然导致带宽瓶颈和尾延迟，严重制约了系统的可扩展性和效率。

修改版本

%我们的工作

为此，本文提出 IMerge —— 一种结合梯度融合与交错通信的无等待反向传播算法，有效缓解通信延迟与带宽瓶颈。

首先，我们采用梯度融合策略，将部分相邻层的参数合并发送以减少启动延迟，并通过建立准确合理的融合条件与通信模型，作为动态判别融合层的依据。其次，针对参数服务器架构下潜在的网络拥塞问题，我们设计一种交错通信的调度策略，包括迭代之间的交错和迭代之内的交错，以减少每轮迭代中通信的参数量和并发数，从而缓解不同工作节点对网络资源的竞争。最后，我们基于Pytorch设计了一种包含多 PS 节点的参数服务器原型系统以分担通信带宽的压力，并采用一种参数轮询分配的机制保证各 PS 节点的工作负载均衡。通过将交错融合通信策略集成至本文搭建的参数服务器中，可有效提升系统的训练效率，实验结果表明本文设计的方案相较默认的参数服务器同步训练机制对于 ResNet18、Vgg16 和 AlexNet 模型分别可获得约 84.4%、211.5%和 100.9%的性能加速效果。

最新版本：

\section{Introduction}
\label{sec:introduction}

% 背景描述
In recent years, deep learning based on deep neural networks (DNNs) has developed rapidly and has been widely applied in various domains~\cite{krizhevsky2012imagenet,simonyan2014very,graves2013speech,amodei2016deep,deng2018deep,collobert2008unified}. However, with the exponential growth of data volume and model complexity, training on a single device becomes prohibitively time-consuming~\cite{wang2019benchmarking}. As a result, distributed deep learning has become increasingly important to accelerate the training of DNNs~\cite{ben2019demystifying}.

Among distributed training strategies, data parallelism has emerged as the predominant approach~\cite{shallue2019measuring,dean2012large,dekel2012optimal,meng2019convergence}. The basic idea is to partition the training data across multiple workers, where each worker performs forward and backward propagation independently and computes local gradients. These gradients must then be aggregated through parameter synchronization across all workers to update the global model parameters~\cite{shi2021exploiting,shi2018performance}. However, in large-scale distributed training, gradient aggregation often introduces intensive communication overhead, which emerges as a key scalability bottleneck~\cite{shi2021exploiting, duan2022mercury}.

% PS vs. all-reduce
Therefore, parameter synchronization has become a critical factor affecting the performance of distributed deep learning. The two most common system architectures are the \textbf{Parameter Server (PS)} and \textbf{All-Reduce}. The All-Reduce architecture~\cite{awan2017s,hoefler2010toward,rabenseifner2004optimization,jia2018highly} performs global gradient aggregation through decentralized peer-to-peer communication, which enables efficient bandwidth utilization. However, it is more suitable for transmitting a small number of large messages, whereas in practical distributed training, the communication of numerous small messages is more common, limiting its advantages. Meanwhile, its communication rounds and startup latency grow linearly with cluster size, and its chain-based structure exhibits weak fault tolerance, further constraining scalability. In contrast, the PS architecture~\cite{smola2010architecture,li2013parameter,li2014communication} adopts a centralized parameter management approach, demonstrating unique advantages in fault tolerance, flexibility, and adaptability to heterogeneous environments.

% 参数服务器架构问题
However, the original PS architecture exhibits inherent limitations. First, its highly centralized many-to-one communication pattern often leads to bandwidth bottlenecks and network congestion, making a single PS node the critical bottleneck, which constrains scalability and degrades training performance. Second, the strong serialization between computation and communication results in suboptimal resource utilization across the cluster. Therefore, optimizing communication in PS-based systems is of practical importance for fully leveraging the parallelism of multiple computing devices while reducing unnecessary overhead.

%相关工作：通信架构优化、通信压缩、通信调度
Recent research has primarily focused on three directions to optimize the PS architecture, namely (i) communication architecture optimization, (ii) communication compression, and (iii) communication scheduling, with the aim of improving efficiency and scalability.

%通信架构优化
For \textbf{communication architecture optimization}, the core idea is to refine the logical topology followed by PS during runtime. 
In the default star topology~\cite{kairouz2021advances}, a single PS node serves as the central aggregator for gradients and parameter updates from all workers, which easily becomes a performance bottleneck due to bandwidth pressure. 
Existing works mainly follow two approaches: one line of research increases the number of PS nodes to achieve horizontal scaling, as exemplified by BytePS~\cite{jiang2020unified}; 
another introduces hierarchical PS designs~\cite{wan2020rat,luo2020plink,yang2018tree,huang2021communication} to enable vertical scaling and alleviate cross-layer communication overhead.  

%通信压缩
In terms of \textbf{communication compression}, researchers aim to reduce the volume of transmitted data to lower communication costs. 
Typical techniques include gradient quantization~\cite{seide20141, wen2017terngrad}, which converts high-precision gradients into low-bit representations, 
and gradient sparsification~\cite{strom2015scalable, aji2017sparse, shi2020layer}, which transmits only gradients whose absolute values exceed a predefined threshold. 
Although these methods can significantly reduce communication overhead, they may cause accuracy degradation~\cite{grubic2018synchronous}, 
which often requires error compensation mechanisms. 
Furthermore, the compression and decompression processes introduce additional computation, which in some cases offsets the benefits of communication reduction.  

%通信调度
Regarding \textbf{communication scheduling}, the main idea is to break the strict serialization between computation and communication by overlapping the two through task scheduling. 
In distributed deep learning training, computation tasks and communication tasks can be represented by a directed acyclic graph (DAG)~\cite{shi2018dag}. 
Since computation (forward and backward propagation) and communication (gradient aggregation) utilize different resources, communication can be overlapped with computation to hide part of the overhead~\cite{hashemi2019tictac,jayarajan2019priority,li2018pipe,narayanan2019pipedream,sergeev2018horovod,peng2019generic,shi2020communication}. 
Without affecting model convergence, existing methods have explored overlapping communication with the backward propagation as well as with the forward propagation, thereby reducing iteration time.

%由当前工作问题过渡到我们的工作
Although prior efforts on communication architecture optimization, compression, and scheduling have alleviated the bottlenecks of PS to some extent, they remain insufficient under the centralized PS architecture. In particular, the many-to-one communication pattern still causes bandwidth saturation and long-tail latency, posing critical challenges to scalability and efficiency. 

% 我们的工作
To address these issues, in this paper, we propose \textbf{IMerge}, an Interleaved-communication and Merged-gradient Wait-Free Backpropagation (WFBP) algorithm that integrates gradient fusion with interleaved scheduling to mitigate both latency and bandwidth constraints in PS-based distributed deep learning. 
First, we introduce a \textbf{gradient fusion strategy} that aggregates parameters from adjacent layers to reduce startup latency. We establish a mathematical communication model and well-defined fusion conditions, which dynamically determine whether layers should be fused during training.  
Second, to mitigate potential network congestion in PS-based systems, we design an \textbf{interleaved communication scheduling strategy} that operates both across iterations and within a single iteration. This strategy reduces the volume and concurrency of parameter transmissions per round, thereby alleviating contention among workers for network resources.  
Third, we implement a \textbf{PyTorch-based multi-PS prototype} to distribute communication bandwidth. A parameter polling mechanism is adopted to balance workloads across servers and prevent hotspot bottlenecks. By integrating our interleaved and fused communication strategy into this prototype, we effectively decouple computation and communication.  
Extensive experiments demonstrate that our approach significantly improves training efficiency compared to the default PS-based synchronization mechanism. On ResNet18, VGG16, and AlexNet, our method achieves performance speedups of approximately 84.4\%, 211.5\%, and 100.9\%, respectively.  

%我们的贡献总结
Our contributions are summarized as follows:
\begin{itemize}
	\item We provide an in-depth analysis of communication scheduling strategies in distributed deep learning, investigating how parameter transmission granularity and timing affect communication performance and system scalability, and comparing the characteristics of different approaches.
	
	\item We propose \textbf{IMerge}, a novel scheduling strategy that integrates gradient fusion with interleaved communication. IMerge leverages a PS-specific model to predict the computation--communication timeline and dynamically determine fusion conditions, while its interleaved scheduling mechanism reduces bandwidth contention and overlaps communication with computation to sustain high throughput.
	
	\item We develop a PyTorch-based multi-PS prototype system with a parameter polling mechanism for balanced load distribution. By integrating IMerge into this prototype, we achieve significant performance improvements on CNN benchmarks, including ResNet18, VGG16, AlexNet, and ResNet50.
\end{itemize}

% 论文结构
The remainder of this paper is organized as follows. Section \ref{sec:Background and Motivation} introduces the background and motivation, including distributed deep learning and the PS architecture. Section \ref{sec:IMerge Design} describes the design of the IMerge algorithm, including gradient fusion and interleaved communication strategies. Section \ref{sec:Implementation} presents the system implementation based on a multi-PS prototype. Section \ref{sec:Evaluation} reports experimental results and performance evaluations. Finally, Section \ref{sec:Conclusions} concludes the paper and discusses future directions.

2025/09/01

目标：修改贡献总结部分

（1）与我们的实际实验（成果）一一对应

参考老师点评：我们针对计算通信重叠的问题：融合通信（新的融合模型对通信的评估更加准确、精确控制）、多并发导致的网络拥塞提出交错通信。最后，我们基于Pytorch搭建了多参数服务器系统，并整合我们的交错融合通信策略，

先修改我们的工作介绍部分：

为此，本文提出 IMerge —— 一种结合梯度融合与交错通信的无等待反向传播算法，在参数服务器架构下有效缓解通信延迟与带宽瓶颈问题。

首先，我们采用融合梯度通信策略，将部分相邻层的小参数合并发送以减少启动延迟。
针对传统的线性通信模型 [xx] 难以准确刻画参数服务器架构下“多对一”通信模式的不足，本文建立了一个新的通信模型，能够更好地适配 PS 架构，并准确预测模型训练过程中计算与通信的时间线（timeline），为动态判断融合条件提供理论依据。
其次，为进一步缓解参数服务器架构下潜在的网络拥塞和带宽瓶颈问题，我们在融合通信的基础上设计了一种交错通信策略，涵盖迭代之间与迭代之内的两种交错调度机制，以减少每轮迭代中通信的参数量和并发数，从而缓解不同工作节点对网络资源的竞争。
最后，我们基于Pytorch搭建了包含多 PS 节点的参数服务器实物平台，并采用一种参数轮询分配的机制保证各 PS 节点的工作负载均衡，同时将所提出的交错与融合通信策略（IMerge）集成至该系统中。
为了验证我们提出的IMerge的有效性，我们在一个一个最大规模为32个节点的GPU集群上用4个典型的CNNs来评估其性能。
实验结果表明，在 32 节点规模的参数服务器训练平台下，基于本文建立的新的通信模型的融合梯度通信策略相较于基于传统的通信模型的融合通信策略可获得 28.5\%-182.3%的加速效果。
进一步，本文提出的IMerge相较默认的参数服务器同步训练机制可获得约 84.4%-211.5%的加速效果。

对应英文：

To address these issues, in this paper, we propose \textbf{IMerge}—an interleaved-communication and merged-gradient wait-free backpropagation (WFBP) algorithm designed to alleviate communication latency and bandwidth bottlenecks in parameter server (PS) architectures.

First, we introduce a \textbf{gradient fusion strategy} that merges small parameters from adjacent layers to reduce startup latency. Observing that traditional linear communication models [xx] fail to accurately characterize the many-to-one communication pattern inherent to PS-based systems, we establish a new communication model tailored for the PS architecture. This model enables accurate prediction of the computation–communication timeline during training and provides a theoretical basis for dynamically determining fusion conditions.

Second, to further mitigate potential network congestion and bandwidth contention in PS systems, we propose an  \textbf{interleaved communication scheduling strategy} on top of gradient fusion. This strategy includes both inter-iteration and intra-iteration interleaving mechanisms, which reduce the communication volume and concurrency per iteration, thereby alleviating contention among workers for network resources.

Finally, we implement a multi-PS prototype system based on PyTorch, incorporating a parameter polling mechanism to balance workloads across PS nodes. The proposed interleaved and fused communication strategies (IMerge) are fully integrated into this system.

To evaluate the effectiveness of IMerge, we conduct experiments on a GPU cluster with up to 32 nodes using four representative CNN models. Experimental results show that, under the 32-node PS training setting, our fusion strategy guided by the proposed new communication model achieves 28.5%–182.3% speedup over the same strategy guided by the traditional communication model. Furthermore, IMerge achieves 84.4%–211.5% performance improvement over the default PS-based synchronization mechanism.

遗留问题：目前整体稍显较长，可以适当压缩

2.修改我们的贡献总结部分

本文的主要贡献如下：

本文提出了一种新的通信调度算法 IMerge，适用于参数服务器架构的分布式训练。

我们建立了一种新的通信模型，更准确地刻画了参数服务器的多对一通信模式，并在此基础上设计了更加准确的梯度融合策略。

在融合通信策略的基础上，进一步设计了交错通信调度，缓解参数服务器架构下潜在的网络拥塞和带宽瓶颈问题。

我们基于 PyTorch 搭建了多 PS 原型系统并集成 IMerge，在一个 32 节点的 GPU 集群上进行真实实验评估其性能。

Our main contributions are summarized as follows:
\begin{itemize}
    \item We propose a new communication scheduling algorithm, \textbf{IMerge}, for distributed training under the parameter server architecture.

    \item We establish a new communication model that more accurately characterizes the many-to-one transmission pattern in PS systems, based on which we design a model-guided gradient fusion strategy.

    \item On top of the fusion strategy, we further design an interleaved communication scheduling strategy that incorporates both inter-iteration and intra-iteration interleaving mechanisms.

    \item We implement a PyTorch-based multi-PS prototype system, integrate IMerge into it, and conduct real-world experiments on a 32-node GPU cluster to evaluate its performance.
\end{itemize}