When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study

Gerardo Iuliano, Gabriele De Vito, Dario Di Nucci

Smart contracts are immutable programs deployed on blockchain platforms, where vulnerabilities can translate directly into financial loss. Among the techniques proposed for their detection, recent work has relied increasingly on Large Language Models (LLMs) guided by carefully crafted prompts. However, the evidence accumulated so far rests almost entirely on the dense Transformer architecture behind the GPT line and on model snapshots already scheduled for deprecation, leaving open the question of whether the reported behaviors reflect a general property of LLMs or an artifact of that specific design. This paper presents an extended replication of the empirical study by Chen et al. on LLM-based smart contract vulnerability detection. We replicate the original protocol on GPT-4 as a baseline, followed by a differentiated replication, executing the protocol on representative models based on Mixture-of-Experts and RNN-like designs, together with an attempted preliminary evaluation of State Space Models. The exact replication confirms the findings of Chen et al., including the high-recall, low-precision detection pattern they report on GPT-4. Nevertheless, the differentiated…