Large language model-enhanced probabilistic modeling for effective static analysis alarms

Abstract

Static analysis presents significant challenges in alarm handling, where probabilistic models and alarm prioritization are essential methods for addressing these issues. These models prioritize alarms based on user feedback, thereby alleviating the burden on users to manually inspect alarms. However, they often encounter limitations related to efficiency and issues such as false generalization. While learning-based approaches have demonstrated promise, they typically incur high training costs and are constrained by the predefined structures of existing models. Moreover, the integration of large language models (LLMs) in static analysis has yet to reach its full potential, often resulting in lower accuracy rates in vulnerability identification. To tackle these challenges, we introduce BinLLM, a novel framework that harnesses the generalization capabilities of LLMs to enhance alarm probability models through rule learning. Our approach integrates LLM-derived abstract rules into the probabilistic model, using alarm paths and critical statements from static analysis. This integration enhances the model's reasoning capabilities, improving its effectiveness in prioritizing genuine bugs while mitigating false generalizations. We evaluated BinLLM on a suite of C programs and observed 40.1% and 9.4% reduction in the number of checks required for alarm verification compared to two state-of-the-art baselines, Bingo and BayeSmith, respectively, underscoring the potential of combining LLMs with static analysis to improve alarm management.

1. Introduction

Static analysis is a powerful technique for detecting bugs in software systems, yet it faces significant challenges in managing alarms effectively (Beller et al., 2016; Christakis and Bird, 2016; Muske and Serebrenik, 2022). A key post-processing method is user-guided ranking (Shen et al., 2011; Mangal et al., 2015), which prioritizes alarms based on user feedback. Bayesian program analysis formalizes this into a probabilistic framework: alarms are ranked by inferred probabilities, and user feedback iteratively refines the model (Raghothaman et al., 2018; Heo et al., 2019; Chen TY et al., 2021; Kim et al., 2022; Zhang et al., 2024). Its effectiveness depends on the generalization ability of the Bayesian network, derived from inference graphs built using Datalog rules. More accurate rules yield stronger generalization, reducing user workload and improving alarm prioritization. However, this approach still relies on user expertise, and Datalog rules lack intelligent optimization (Muske and Serebrenik, 2022).

The integration of large language models (LLMs) into static analysis has gained attention (Chen M et al., 2021; Ma et al., 2023; Sun et al., 2024). However, challenges such as hallucinations and randomness persist (Ji et al., 2023; Touvron et al., 2023), limiting their effectiveness in vulnerability detection. The state-of-the-art methods based on LLMs have improved accuracy to 67.6% (Zhou et al., 2025), but these methods are constrained by data processing granularity, typically limited to the function or line level, which hampers their ability to detect vulnerabilities across entire libraries. Existing methods use mainly LLMs as judgment substitutes or static analysis supplements (Gao et al., 2023; Mohajer et al., 2023; Li HN et al., 2024; Li ZY et al., 2024), lacking deep integration. This suggests a paradigm shift where LLMs actively enhance static analysis for improved accuracy (Li HN et al., 2024).

To address these challenges, we propose BinLLM, a framework that leverages LLMs to refine alarm probability models through rule learning. By integrating LLM-derived rules with static analysis insights, including alarm paths and critical statements, our approach enhances reasoning capabilities and reduces user intervention, fostering a more synergistic relationship between LLMs and static analysis.