Redefining IoT Threat Detection: The Power of Cumulative Analysis in the CUMAD Framework | HackerNoon

Table of Links

Abstract and 1. Introduction

2. Related Work

3. Background on Autoencoder and SPRT and 3.1. Autoencoder

3.2. Sequential Probability Ratio Test

4. Design of CUMAD and 4.1. Network Model

4.2. CUMAD: Cumulative Anomaly Detection

5. Evaluation Studies and 5.1. Dataset, Features, and CUMAD System Setup

5.2. Performance Results

6. Conclusions and References

3.2. Sequential Probability Ratio Test

Sequential probability ratio test (SPRT) is a simple yet powerful statistical tool that has found applications in many different domains, in particular, fault detection or quality control [7]. SPRT is a variant of the traditional probability ratio test for testing under what distribution (or with what distribution parameters), it is more likely to have the observed sequence of samples. Unlike traditional probability ratio test that requires a pre-defined fixed number of samples to carry out the test, SPRT works in an online fashion; it updates the corresponding statistical measure as samples arrive sequentially, and can conclude when sufficient samples have arrived to reach a decision. In its simplest form, SPRT is a statistical method to test a simple null hypothesis against a simple alternative hypothesis. In the following we will more formally describe the operation of SPRT.

Let y denote a Bernoulli random variable with an unknown parameter θ, and let yi , for i = 1, 2, . . . denote the corresponding successive observations of y. SPRT can be used to test a simple null hypothesis H0 that θ = θ0 against a simple alternative hypothesis H1 that θ = θ1, more specifically,

As a simple and powerful statistical tool, SPRT possesses a few critical and desired properties that lead to the wide-spread application of the technique in many different domains. First, the false positive and false negative rates of SPRT can be specified by user-desired error rates, which in turn control the thresholds of the model. Second, it has been proved that, among all sequential and non-sequential probability ratio testing algorithms, SPRT minimizes the expected number of observations to reach a decision with no greater errors. Put in another way, on average SPRT can reach a conclusion quickly compared to other probability ratio testing algorithms.

4. Design of CUMAD

In this section we will first discuss the considered network model, where CUMAD will be deployed, and then we will present the design of the CUMAD framework.

4.1. Network Model

Figure 2 illustrates the conceptual network model, where CUMAD is deployed. As shown in the figure, in order for CUMAD to carry out its task to detect compromised IoT devices in a network, CUMAD needs to have access to the network traffic associated with the IoT devices in the network. Depending on the deployment scenarios of CUMAD in the network and the corresponding network architecture, there can be a few different ways for CUMAD to obtain the corresponding network traffic of IoT devices. In essence, CUMAD as a network-based solution can be deployed in a similar way as network-based intrusion detection systems.

In the current design of CUMAD, (statistical) features from raw network traffic will be extracted and fed to CUMAD for detecting compromised IoT devices. Each input data point fed to CUMAD comprises these extracted features, and can be summarized at different levels of granularity of network traffic, such as packets, flows, and time windows. These features will capture the network behavioral characteristics of the corresponding IoT devices. In Section 5 we will discuss the network traffic features contained in the public-domain N-BaIoT dataset when we perform evaluation studies on CUMAD [8].

4.2. CUMAD: Cumulative Anomaly Detection

Figure 3 illustrates the high-level architecture of the CUMAD framework. CUMAD consists of two main components: an anomaly detection component (ADC) and a cumulative anomaly component (CAC). Assuming the model has been properly trained (will be discussed shortly), given an input data point with the corresponding features, the main responsibility of ADC is to classify an input data point as either normal or anomalous. After the classification of the input data point, the result is passed to the second component (CAC), which will maintain a cumulative view of the network traffic behavior of the corresponding IoT device, by sequentially merging the individual classification results into the view. When sufficient evidence on an IoT device has been collected to indicate that it has been compromised, an alert will be generated. In the following we will describe each component in details, both in model training and deployment to detect compromised IoT devices.

We note that different types of IoT devices perform drastically different functionalities, and in addition, we would like to detect which IoT device is compromised, we need to develop a separate CUMAD model for each IoT device and monitor their network traffic behaviors separately using their corresponding CUMAD models. Therefore, the following discussions are for one IoT device. We note that, although there are vastly diverse types of IoT devices on the Internet, autoencoder is a powerful neural network that is capable of learning different models. Therefore, we are able to build diverse autoencoder models, one for each IoT device, despite their vastly different network traffic behaviors of

these IoT devices. In addition, IoT devices also provide us with unique opportunities in establishing the models of normal behaviors, compared to traditional computer systems. In particular, each IoT device only performs a few well-defined simple functionalities in an autonomous or semiautonomous fashion, with very limited user interactions after the initial device configuration and setup. This makes it simpler to establish a model of normal behaviors in carrying out anomaly detection.

4.2.1. Model Training and Setup. Before CUMAD can be used to monitor network traffic to detect compromised IoT devices, we need to train a CUMAD system for each IoT device so that it can learn the normal model of the device. During the training stage of a deployed CUMAD system, it is critical that we should only feed normal (benign) network traffic of the device to the system. This can be done, for example, when an IoT device is first deployed in the network. In order to minimize false positives during the detection stage, it is also important that CUMAD has a reasonably complete view of all the normal network traffic behavior of the device.

As discussed above, the premise of using an autoencoder as an anomaly detection mechanism is that, although it can effectively reconstruct data points that are similar to the data points that it has seen previously during the training stage, it in general performs poorly to reconstruct data points that substantially differ from the training data. This is manifested in large reconstruction errors. Therefore, we will use the reconstruction error as the anomaly score, and when the anomaly score is greater than the pre-defined threshold, we classify the corresponding input data point as an anomalous sample.

The parameters α and β are the user-desired false positive rate and false negative rate, respectively. They normally have small values for all practical applications, for example, in the range 0.01 to 0.05. The initial value of Λn in Eq. (1) is set to 0 during the setup stage of the system. The functionality of the Alert module is to generate proper alert to inform system administrators of the detection of a compromised IoT device. Other actions can also be taken based on the local security policies, for example, informing proper agents to isolate the compromised IoT device.

4.2.2. Detection. After the model has been trained and the required parameters have been set for the CUMAD system, it can be used to monitor network traffic to detect if the corresponding IoT device has been compromised. In the following we describe the basic steps of a CUMAD system in carrying out the detection task (see Algorithm 1).

generated to indicate a normal data point. The output of the Detector module is then passed to the SPRT module to determine if sufficient evidence has been accumulated to make a decision regarding the nature of the IoT device (compromised or normal; line 10 of the algorithm). SPRT updates the probability ratio measure Λn according to Eq. (1), as the 0 (normal data point) and 1 (anomalous data point) output sequence of the Detector module arrives (lines 13 to 18). After the value of Λn is updated for each input data point, SPRT compares the value of Λn with the two boundaries A and B to determine if a decision can be made (lines 20 to 29). When the value of Λn hits or crosses the upper bound B, SPRT will conclude that the alternative hypothesis H1 is true, that is, the IoT device has been compromised. In this case, SPRT will inform the Alert module the detection of an compromised IoT device. Proper alert will be generated and corresponding system administrators will be informed. In addition, from this time on, it is not necessary for CUMAD to monitor the IoT device anymore, until proper actions have been taken to clean up or remove the device.

When the value of Λn is equal to or smaller than the lower bound A, SPRT reaches the conclusion that H0 is true, that is, the IoT device is not compromised. From the viewpoint of detecting compromised IoT devices, this conclusion is less interesting in that we cannot terminate the monitoring of the device as we have done when a compromised IoT device is detected. A normal IoT device may become compromised at a later time. Therefore, in this case, we will reset the state of SPRT to restart the monitoring of the IoT device, in particular, we will reset the value of Λn to zero. If a decision cannot be reached at this time (line 28), SPRT will simply wait for additional input data points and repeat the same procedure.