Efficient privacy-preserving scheme for secure neural network inference

Abstract

The increasing adoption of smart devices and cloud services, coupled with limitations in local computing and storage resources, prompts numerous users to transmit private data to cloud servers for processing. However, the transmission of sensitive data in plaintext form raises concerns regarding users' privacy and security. To address these concerns, this study proposes an efficient privacy-preserving secure neural network inference scheme based on homomorphic encryption and secure multi-party computation, which ensures the privacy of both the user and the cloud server while enabling fast and accurate ciphertext inference. First, we divide the inference process into three stages, including the merging stage for adjusting the network structure, the preprocessing stage for performing homomorphic computations, and the online stage for floating-point operations on the secret sharing of private data. Second, we propose an approach of merging network parameters, thereby reducing the cost of multiplication levels and decreasing both ciphertext–plaintext multiplication and addition operations. Finally, we propose a fast convolution algorithm to enhance computational efficiency. Compared with other state-of-the-art methods, our scheme reduces the linear operation time in the online stage by at least 11%, significantly reducing inference time and communication overhead.

1. Introduction

With the rapid development of smart devices, multimedia data such as images are playing an increasingly prominent role in various fields, and many users choose to outsource their data to cloud servers for processing and storage (Chai et al., 2022). Although the cloud servers provide convenient services, they also introduce additional security concerns. When the users upload private data containing sensitive information to the cloud servers, the latter ones may stealthily access the former ones’ data, leading to the leakage of sensitive information (Riazi et al., 2018; Ng and Chow, 2021). Furthermore, the parameters of intelligent models (e.g., neural networks) stored on the cloud servers are considered private, posing a risk of the model parameters inferred by the users, thus requiring protection (Chaudhari et al., 2020; Ma et al., 2021; Wang Y et al., 2023).

To protect the users’ privacy and ensure the cloud servers’ security, homomorphic encryption technology for computations under ciphertext has emerged as a preferable solution (Li JS et al., 2020). The users encrypt their private data before uploading them to the cloud servers for inference calculations, and the cloud servers return the encrypted prediction results for decryption by the users, thereby achieving inference while protecting the privacy of both the users and servers (Liu et al., 2017). In recent years, convolutional neural networks (CNNs) have flourished, demonstrating their capability to extract more abstract and complex features from data (Schroff et al., 2015) and further enhancing inference accuracy. This advancement has led to widespread applications in encrypted image prediction (Li Y et al., 2024).

Dowlin et al. (2016) proposed CryptoNets, which encrypts multiple inputs into a single ciphertext for parallel computation, replacing max pooling with average pooling and employing the square function as the activation function, marking the first instance of using CNN for encrypted image prediction. Hesamifard et al. (2017) proposed CryptoDL, which replaces the activation functions in neural network models with polynomial functions and provides mathematical expressions for low-degree polynomial approximation activation functions. Chou et al. (2018) devised Faster CryptoNets, accelerating inference by integrating neural network pruning methods to reduce the number of parameters in the original model. Chabanne et al. (2017) attempted to integrate batch normalization (BN) layers commonly used in deep learning with existing encryption schemes, effectively deepening the network layers. Ishiyama et al. (2020) proposed a scheme for normalizing inputs using BN layers and approximating the Swish and rectified linear unit (ReLU) with polynomial functions. They mitigated multiplication levels to reduce the multiplication cost in the ciphertext domain by preprocessing the coefficients of the highest-order terms in the polynomial. They provided evaluation results for different fitting intervals. However, despite the advancements made by the aforementioned studies, which leverage polynomial approximation and square function for nonlinear operations, they often suffer from a notable drawback: the loss of inference accuracy (Li QF et al., 2020; Iha et al., 2021; Lou et al., 2021).