In recent years, deep learning has been applied to a variety of scenarios in Industrial Internet of Things (IIoT), including enhancing the security of IIoT. However, the existing deep learning methods utilised in IIoT security are manually designed by heavily relying on the experience of the designers. The authors have made the first contribution concerning the joint optimisation of neural architecture search and hyper-parameters optimisation for securing IIoT. A novel automated deep learning method called synchronous optimisation of parameters and architectures by GA with CNN blocks (SOPA-GA-CNN) is proposed to synchronously optimise the hyperparameters and block-based architectures in convolutional neural networks (CNNs) by genetic algorithms (GA) for the intrusion detection issue of IIoT. An efficient hybrid encoding strategy and the corresponding GA-based evolutionary operations are designed to characterise and evolve both the hyperparameters, including batch size, learning rate, weight optimiser and weight regularisation, and the architectures, such as the block-based network topology and the parameters of each CNN block. The experimental results on five intrusion detection datasets in IIoT, including secure water treatment, water distribution, Gas Pipeline, Botnet in Internet of Things and Power System Attack Dataset, have demonstrated the superiority of the proposed SOPA-GA-CNN to the state-of-the-art manually designed models and neuron-evolutionary methods in terms of accuracy, precision, recall, F1-score, and the number of parameters of the deep learning models. 

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1049/csy2.12085

SOPA‐GA‐CNN: Synchronous optimisation of parameters and architectures by genetic algorithms with convolutional neural network blocks for securing Industrial Internet‐of‐Things

Neural architecture search (NAS) has been an active direction of automatic machine learning (Auto-ML), aiming to explore efficient network structures. The searched architecture is evaluated by training on datasets with fixed data augmentation policies. However, recent works on auto-augmentation show that the suited augmentation policies can vary over different structures. Therefore, this work considers the possible coupling between neural architectures and data augmentation and proposes an effective algorithm jointly searching for them. Specifically, 1) for the NAS task, we adopt a single-path based differentiable method with Gumbel-softmax reparameterization strategy due to its memory efficiency; 2) for the auto-augmentation task, we introduce a novel search method based on policy gradient algorithm, which can significantly reduce the computation complexity. Our approach achieves 97.91% accuracy on CIFAR-10 and 76.6% Top-1 accuracy on ImageNet dataset, showing the outstanding performance of our search algorithm.

/pdf/daas-differentiable-architecture-and-augmentation-policy-4nikw1rzyo.pdf

DAAS: Differentiable Architecture and Augmentation Policy Search.

Automated machine learning (AutoML) usually involves several crucial components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization (HPO), and Neural Architecture Search (NAS). Although many strategies have been developed for automating these components in separation, joint optimization of these components remains challenging due to the largely increased search dimension and the variant input types of each component. Meanwhile, conducting these components in a sequence often requires careful coordination by human experts and may lead to sub-optimal results. In parallel to this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from low performance correlation between the search and retraining stages. An end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search is desirable. In view of these, we propose DHA, which achieves joint optimization of Data augmentation policy, Hyper-parameter and Architecture. Specifically, end-to-end NAS is achieved in a differentiable manner by optimizing a compressed lower-dimensional feature space, while DA policy and HPO are updated dynamically at the same time. Experiments show that DHA achieves state-of-the-art (SOTA) results on various datasets, especially 77.4\% accuracy on ImageNet with cell based search space, which is higher than current SOTA by 0.5\%. To the best of our knowledge, we are the first to efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end manner without retraining.

Kaichen Zhou

Papers

DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture.