Mind the Gap: towards generalizable autonomous penetration testing via domain randomization and meta-reinforcement learning

Abstract

With the increasing number of vulnerabilities exposed on the Internet, autonomous penetration testing (pentesting) has emerged as a promising research area. Reinforcement learning (RL) is a natural fit for studying this topic. However, two key challenges limit the applicability of RL-based autonomous pentesting in real-world scenarios: the training environment dilemma—training agents in simulated environments is sample-efficient while ensuring that their realism remains challenging; poor generalization ability—agents’ policies often perform poorly when transferred to unseen scenarios, with even slight changes potentially causing a significant generalization gap. To address both challenges, we propose a generalizable autonomous pentesting framework termed GAP, which aims to achieve efficient policy training in realistic environments and train generalizable agents capable of drawing inferences about other cases from one instance. GAP introduces a real-to-sim-to-real pipeline that enables end-to-end policy learning in unknown real environments while constructing realistic simulations and improves agents’ generalization ability by leveraging domain randomization and meta-RL learning. We are among the first to apply domain randomization in autonomous pentesting and propose a large language model-powered domain randomization method for synthetic environment generation. We further apply meta-RL to improve agents’ generalization ability in unseen environments by leveraging synthetic environments. Combining the two methods effectively bridges the generalization gap and improves agents’ policy adaptation performance. Simulations are conducted on various vulnerable virtual machines, with results showing that GAP can enable policy learning in various realistic environments, achieve zero-shot policy transfer in similar environments, and achieve rapid policy adaptation in dissimilar environments.

1. Introduction

Penetration testing (pentesting) is an authorized simulated cyberattack methodology for identifying security vulnerabilities, allowing organizations to proactively enhance their defenses. However, as more vulnerabilities are exposed on the Internet, traditional manual-based pentesting becomes costly, time-consuming, and personnel-constrained. Autonomous pentesting has emerged as a promising research area, and reinforcement learning (RL) is a suitable method for optimizing the sequential decision-making process involved. RL trains agents through trial and error without needing predefined environmental models, making it a natural fit.

RL-based autonomous pentesting aims to train agents to explore and exploit vulnerabilities in target hosts. Yet, RL algorithms typically require large numbers of training samples, which is impractical in real-world environments where interactions are time-consuming and risky. A common solution is to train agents in simulated or emulated environments and then transfer the learned policies to real scenarios. This approach introduces two key challenges: the training environment dilemma (balancing realism and efficiency) and poor generalization ability.

Challenge 1—the training environment dilemma—represents a conflict between the realism of the training environment and the efficiency of the training process. Prior work has used simulated environments such as network simulators (e.g., NASim), which are sample-efficient but lack realism. Alternatively, training in realistic environments is more faithful but can be inefficient and risky. GAP aims to overcome these limitations by integrating domain randomization and meta-RL to train generalizable policies in realistic environments.