GSM: Efficient Language Modeling with Shared Global State

Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng +10

Improving language model efficiency requires not only cheaper operations but also eliminating redundant computation. We argue that effective modeling requires information accessibility and computational depth, but not repeated historical retrieval across layers. Guided by this principle, we introduce the Global State Model (GSM), a causal encoder--decoder architecture that separates historical information acquisition from deep computation. The encoder progressively aggregates long-range information through multi-stage sparse retrieval into a fixed-size shared state, while decoder layers perform successive transformations over this state without repeated historical access. This design makes decoder attention and cache costs independent of history length, while dependency-aware pruning further reduces prefill computation. Experiments across multiple model scales show that GSM achieves downstream performance comparable to matched DeepSeek-V4.1-Flash-based baselines, with improvements at some scales, while preserving long-context retrieval capabilities. GSM also improves training and inference throughput and reduces cache overhead. These results demonstrate that eliminating…