Termite Pest Identification Method Based on Deep Convolution Neural Networks
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Journal of Economic Entomology, XX(XX), 2021, 1–8
https://doi.org/10.1093/jee/toab162
Research
Household and Structural Insects
Termite Pest Identification Method Based on Deep
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Convolution Neural Networks
Jia-Hsin Huang,1, Yu-Ting Liu,1 Hung Chih Ni,1 Bo-Ye Chen,2 Shih-Ying Huang,2
Huai-Kuang Tsai,1,3, and Hou-Feng Li2,3,
Institute of Information Science, Academia Sinica, Taipei, Taiwan, 2Entomology Department, National Chung Hsing University, 145 Xingda
1
Road, South Dist., Taichung City 402204, Taiwan, and 3Corresponding authors, e-mail: hktsai@iis.sinica.edu.tw; houfeng@nchu.edu.tw
Subject Editor: Claudia Husseneder
Received 24 May 2021; Editorial decision 2 August 2021
Abstract
Several species of drywood termites, subterranean termites, and fungus-growing termites cause extensive
economic losses annually worldwide. Because no universal method is available for controlling all termites,
correct species identification is crucial for termite management. Despite deep neural network technologies’
promising performance in pest recognition, a method for automatic termite recognition remains lacking. To
develop an automated deep learning classifier for termite image recognition suitable for mobile applications,
we used smartphones to acquire 18,000 original images each of four termite pest species: Kalotermitidae:
Cryptotermes domesticus (Haviland); Rhinotermitidae: Coptotermes formosanus Shiraki and Reticulitermes
flaviceps (Oshima); and Termitidae: Odontotermes formosanus (Shiraki). Each original image included multiple
individuals, and we applied five image segmentation techniques for capturing individual termites. We used
24,000 individual-termite images (4 species × 2 castes × 3 groups × 1,000 images) for model development and
testing. We implemented a termite classification system by using a deep learning–based model, MobileNetV2.
Our models achieved high accuracy scores of 0.947, 0.946, and 0.929 for identifying soldiers, workers, and
both castes, respectively, which is not significantly different from human expert performance. We further ap-
plied image augmentation techniques, including geometrical transformations and intensity transformations, to
individual-termite images. The results revealed that the same classification accuracy can be achieved by using
1,000 augmented images derived from only 200 individual-termite images, thus facilitating further model de-
velopment on the basis of many fewer original images. Our image-based identification system can enable the
selection of termite control tools for pest management professionals or homeowners.
Key words: deep learning, pest identification, termite, pest control, image classification
Termites are destructive urban pests, causing more than US$40 formosanus Shiraki (Blattodea: Rhinotermitidae) and C. gestroi
billion in economic loss annually worldwide (Rust and Su 2012). (Wasmann) (Blattodea: Rhinotermitidae), which are the major
Invasive termite pests, including subterranean and drywood ter- construction pests responsible for 95% of house termite infest-
mites, continue expanding their territories (Evans et al. 2013), thus ations (Huang 2020); another subterranean termite, Reticulitermes
increasing the damage inflicted and cost of termite control. Various flaviceps (Oshima) (Blattodea: Rhinotermitidae); a drywood termite,
control methods have been invented and applied to different species, Cryptotermes domesticus (Haviland) (Blattodea: Kalotermitidae);
such as creating underground soil barriers, injecting liquid pesticides and a fungus-growing termite, Odontotermes formosanus (Shiraki)
into wood, fumigation, and baiting subterranean colonies (Potter (Blattodea: Termitidae)—the last three are minor construction pests.
2011). However, no universal method is available for controlling all To control subterranean termites in the urban environment, baiting
termite species, and accurate identification of termite species is essen- and remedial treatment with liquid pesticide injections are the two
tial for controlling specific termites. most commonly used tools. However, current baiting tools are inad-
In Taiwan, five termite species are commonly encountered in equate for controlling drywood termites. O. formosanus, the most
the urban environment: two subterranean termites, Coptotermes frequently encountered termite in the lowland areas of Taiwan and
© The Author(s) 2021. Published by Oxford University Press on behalf of Entomological Society of America. 1
All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
2 Journal of Economic Entomology, 2021, Vol. XX, No. XX
the major food source of the endangered pangolin (Liang 2017), number of images as the training data and applying common aug-
builds mud shelter tubes on urban trees or gardening wood (Li et al. mentation techniques to enrich the training set.
2011). It rarely infests modern construction (Huang 2020) or dam- This study used image recognition and deep learning techniques
ages urban trees (Lai 2019). Identifying termite species helps not to classify economically important pest termite species. Our main
only in choosing proper control tools but also in differentiating pest contributions include the curation of numerous smartphone-based
species from ecologically friendly species. Accurate and efficient ter- images of termite pests and the successful implementation of deep
mite identification can help control termite pests precisely and re- learning models for termite classification. To the best of our know-
duce termiticide use. ledge, our study is the first to demonstrate the use of deep learning
Termite taxonomy and identification, especially of pest species, for termite recognition.
has been extensively researched regionally and worldwide in the past
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
few decades (Weesner 1965, Edwards and Mill 1986, Yang and Li
2012, Krishna et al. 2013). Termite identification requires expertise Materials and Methods
and can be labor intensive; thus, an automatic method could be Termite Collection
helpful. However, despite great achievements in automatic pest rec- Four construction termite pests were collected in Taiwan be-
ognition in recent years (Cardim Ferreira Lima et al. 2020), termite tween June and September 2018—Kalotermitidae: Cr. domesticus;
recognition has not been studied. Rhinotermitidae: C. formosanus and R. flaviceps; and Termitidae:
Recent developments in deep learning technologies based on arti- O. formosanus (Fig. 1). Five colonies of Cr. domesticus were collected
ficial neural networks have achieved state-of-the-art performance in the Siangjiao Bay Ecological Reserve Area, Kenting National Park,
in various image classification tasks. Compared with conventional Hengchun Township, Pingtung County (120.82°E, 21.92°N, C.d.
image-based methods, deep learning techniques do not require fea- colonies 1–5). Three colonies of C. formosanus were collected—one
ture extraction by domain experts, which is a time-consuming and colony/site—from National Chung Hsing University, South District,
subjective manual process (Wäldchen and Mäder 2018). Instead, deep Taichung City (120.67°E, 24.12°N, C.f. colony 1), Dakeshan,
learning approaches, such as convolutional neural networks (CNNs), Zhuolan Township, Miaoli County (120.89°E, 24.31°N, C.f. colony
can be applied to automatically extract the essential features from im- 2), and Taichung Port, Qingshui District, Taichung City (120.53°E,
ages to solve classification problems. CNNs have achieved good re- 24.29°N, C.f. colony 3). Three colonies of R. flaviceps were collected
sults for agricultural pest identification. For example, Xie et al. (2015) in Fushan, Yuanshan Township, Yilan County (121.58°E, 24.76°N,
applied multiple-task sparse representation to enhance performance R.f. colonies 1–3). One colony of O. formosanus was collected at
in recognizing 24 common field crop pest species. Cheng et al. (2017) National Chung Hsing University, South District, Taichung City
recognized 10 pest species with an accuracy of 98.67% using deep re- (120.67°E, 24.12°N, O.f. colony 1), and two colonies were collected
sidual learning. With the application of transfer learning, a pretrained at Tunghai University, Xitun District, Taichung City (120.60°E,
version of AlexNet also obtained a recognition rate of 93.84% for 10 24.18°N, O.f. colonies 2 and 3). All termite colonies were further
types of pest species (Dawei et al. 2019). A recent study developed a divided into three groups for the following experiments (Fig. 1).
sensor device to automatically monitor mosquito species using deep
learning (Huang et al. 2018). However, deep learning methods have
Image Acquisition
not yet been used for termite recognition.
To advance termite pest control strategies with image recogni- A cellular phone (ZenFone 3, ZE552KL, Asustek Computer, Taipei,
tion, we generated a new large-scale termite image dataset and an Taiwan) was placed on an acrylic stage to photograph five termite
automated deep learning classifier of termite pest species. Although workers or three termite soldiers in the acrylic ring (Fig. 2). The ring
several state-of-the-art deep learning frameworks, such as AlexNet, was placed on a black panel to ensure that the termite shape could
VGG-16, GoogleNet, ResNet, Xception, and Inception-V3, have be clearly distinguished. Three colonies each of C. formosanus,
achieved promising performance in pest recognition (Cheng et al. R. flaviceps, and O. formosanus were used in the experiment,
2017, Leonardo et al. 2018, Martineau et al. 2018), the parameters whereas five colonies of Cr. domesticus were used to compensate
of those frameworks are too large to be trained and used for mobile for the shortage of members of the soldier caste. For C. formosanus,
devices. We therefore took advantage of MobileNet, a well-known R. flaviceps, and O. formosanus, worker and soldier castes of each
deep learning architecture (Howard et al. 2017) designed for image colony were studied in three replicates without reusing any indi-
recognition and particularly suited to mobile applications. In add- vidual (2 castes × 3 colonies × 3 replicates). For Cr. domesticus, the
ition to traditional convolution layers and the fully connected layers 18 replicates were conducted with five colonies, and no individual
of the MobileNet architecture, depthwise convolution and pointwise was reused. For each replicate, 1,000 photos were shot continuously
convolution are designed to tune the layers with less loading of with 1-s intervals. For each termite species, 18,000 original images
computation. In this study, we used an improved lightweight were available for further individual-termite segmentation.
MobileNetV2 (Howard et al. 2017; Sandler et al. 2018); this version
has an improved reverse residual module (C-inverted residual block) Individual-Termite Segmentation
for rapid classification of images. The lightweight model makes it Original images of termites comprised several individuals in the
suitable for mobile devices that have lower hardware specifications. ring (Fig. 3A). We employed several image-processing algorithms to
In training deep neural network models, a large amount of insect automatically capture individual termites from the original images.
images is required, but obtaining them is time consuming and diffi- First, a red–green–blue (RGB) image was converted into a binary
cult. Image augmentation is a commonly used strategy to increase image using Otsu thresholding (Otsu 1979). Because in the histo-
the quantity and variety of training data when the number of im- gram of Otsu-threshold segmentation the black background and
ages in a data set is limited. Proper data augmentation can effect- pale-colored termites in the images of our dataset represent high
ively fine-tune and increase the efficiency of models (Perez and Wang contrast, we could obtain well-segmented binary images with highly
2017, Shorten and Khoshgoftaar 2019). Therefore, in this study, we precise outlines of termites (Fig. 3B). Second, for connected compo-
also examined the performance of our model by using a reduced nent filtering, we reversed the binary images by subtracting from 1
Journal of Economic Entomology, 2021, Vol. XX, No. XX 3
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Fig. 1. Four species of termite pests used for MobileNetV2 training and testing. The drywood termite, Cryptotermes domesticus; two subterranean termites,
Coptotermes formosanus and Reticulitermes flaviceps; and the fungus-growing termite, Odontotermes formosanus, were collected from three to five colonies.
Images of soldier and worker castes collected from specific colonies were further divided into three groups for training and testing models (Table 1).
(Haralick and Shapiro 1991). We labeled every connected compo- We used eight morphotypes (4 species × 2 castes, Fig. 1) to train
nent as 1, 2, 3,…, etc. using the D8 connecting mode to allow for and test the model performance. For each morphotype of each group,
area calculation and individual separation. For termites in images, 3,000 original images were randomly chosen and subjected to the
the area they occupy is usually approximately between 13,000 and automatic segmentation pipeline to obtain the first 1,000 individual-
50,000 pixels. In the third step, we filtered the areas by pixel size for termite images for model development. In total, our dataset included
termites to form individual-termite masks, with 0 as background and 24,000 individual-termite images (4 species × 2 castes × 3 groups ×
1 as termite (Fig. 3C). Because having different angles of termites in- 1,000 images).
creases the difficulty of cropping individual termites, we applied the
skeleton method (Ju et al. 2007) to create 1-pixel bones to represent Termite Classification Using MobileNetV2 With
the termite area (Fig. 3D). Next, we used several times of opening
Pretrained Weights
technique to form a thicker skeleton with 10 pixels and subsequently
We adapted the MobileNetV2 models (Sandler et al. 2018) with
applied the Hough transform (Duda and Hart 1972) to rotate en-
pretrained weights (or parameters) as a classifier for termite identifi-
tire images to straighten the given termite images (Fig. 3E). We then
cation. With the aim of termite recognition in a source-limited envir-
applied a bounding box to frame the termite area by using a rect-
onment, we employed transfer learning, which is a useful approach
angle as an image mask and cropped the original image to acquire
that entails fine-tuning the parameters of a network pretrained on
individual-termite areas (Fig. 3F).
a large data set (Martineau et al. 2018). For faster learning, we
Because the side lengths of the rectangles varied, an individual-
trained our MobileNetV2 models using the initial weights generated
termite image was padded into a squared image with an equal value
from the pretrained network on the ImageNet data set (Deng et al.
of width and height and zero padding. Furthermore, we downsized
2009), which consists of 1.4 million images and 1,000 classes of
each image of an individual termite to 130 × 130 pixels. This re-
web images.
duced the image sizes from approximately 3 mb to 23 kb, making
processing much more practical and faster during both model devel-
opment and execution. Our automatic segmentation pipeline led to Leave-One-Group-Out Cross-validation
individual-termite images containing minimum background; consid- We implemented classification tasks with the MobileNetV2 frame-
erable variations in termite posture, angles, and light intensity were work in Python 3.7, PyTorch 1.3.0, Torchvision 0.4.1, and CUDA
observed among the images (Fig. 3). 10. We used adaptive moment estimation (Kingma and Ba 2015) as
4 Journal of Economic Entomology, 2021, Vol. XX, No. XX
the optimizer and categorical cross-entropy (De Boer et al. 2005) as
the loss function for training classifiers. Each classifier was trained
for 100 epochs, with an initial learning rate of 0.0001 and a batch
size of 64. Within each group, the dataset contained 1,000 images
of individual termites of each morphotype (Fig. 1, Table 1). We per-
formed experiments for termite classification on our termite images
using a leave-one-group-out cross-validation (LOGO CV) procedure
in which models are repeatedly trained by leaving the images from
one of the three groups out (Table 1). Because our data set was spa-
tially and temporally dependent, LOGO CV was used to avoid the
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
optimistic bias of k-fold cross-validation, which randomly shuffles
datasets (Roberts et al. 2017). Additionally, LOGO CV provided
three times the model performance independently, thus revealing the
robustness of our termite classification system. Fig. 2. Termite photography device. Termites were placed in the acrylic ring.
The cellular phone was placed on an acrylic stage with its lens aiming at the
Testing a Sufficient Number of Training Images and termite through the 1.5-cm gap. The termites moved freely inside the ring
during photo capture. In each replication, 1,000 images were continuously
Applying Image Augmentation
taken with a 1-s interval.
Because the number of training data also influences model per-
formance (Perez and Wang 2017), we examined the drop in clas-
pixelated images. The user-view screenshot of the program is illus-
sification performance with a reduced number of training images.
trated in Supp Fig. 2 (online only). A user saw one termite image at a
We randomly eliminated 200, 400, 600, and 800 training images
time and was required to select any one of eight given morphotypes
per morphotype in each group and then used the remaining im-
before proceeding to next image. During the test, the program auto-
ages to train the other MobileNetV2 models of both termite castes.
matically recorded both selected answers and time spent automat-
Subsequently, we applied these models to predict the same 1,000
ically. To prevent visual fatigue, users could save their classification
images of each caste in the test set.
progress and resume at any time. After finishing all 8,000 images, the
We also tested whether prediction accuracy can be raised by
user received a report of classification accuracy.
augmenting the fewest number of images (200 images) per termite
morphotype to 1,000 images. Therefore, we applied several image
Statistical Analysis
augmentation techniques, including geometrical transformations
(rotation and flipping) and intensity transformations (RGB color Statistical analyses were performed using the computing software
filter and adding Gaussian noise; examples in Supp Fig. 1 [online R (v.4.0.2). The statistical significance of prediction results was
only]). The rotation, flipping, RGB filter, and Gaussian noise tech- evaluated using the t test for two-group comparisons and one-way
niques were used randomly on 30%, 30%, 30%, and 10%, respect- analysis of variance (ANOVA) followed by Tukey’s honestly sig-
ively, of the individual-termite images. nificant difference (HSD) post hoc test for multiple comparisons.
Correlations between the number of input images for model training
and classification accuracy were analyzed using Pearson’s correl-
Termite Classification by Human Experts
ation coefficient. A P-value of < 0.05 was set as significant.
To compare the termite classification performance between the
MobileNetV2 model and human experts, we prepared a human clas-
sification program written in Python 3 and compiled as an execution
file with compatibility for only Windows 10. Three human experts Results
chose one of three groups of termite images and then conducted clas- Termite Classification Performance by MobileNetV2
sification task. Each group contained 8,000 images, which were the and Human Experts
same images used for testing the MobileNetV2 model. Because the The three classifiers achieved average accuracy scores of 0.947,
images used for MobileNetV2 were downsized to 130 × 130 pixels, 0.946, and 0.930 for soldier, worker, and both castes, respect-
thus differing from the original resolution, we designed an analo- ively (Table 2). Underlying the LOGO CV procedure, three inde-
gous visual experience for humans by magnifying the low-resolution pendent cross-validation models demonstrated similar performance,
images to 500 × 500 pixels. Magnifying low-resolution images, indicating the robustness of the MobileNetV2 model for termite
however, does not increase the resolution; it only results in larger classification (soldier and worker, P = 0.98; soldier and both castes,
P = 0.47, and worker and both castes, P = 0.25). It took 231–243 s
Table 1. Leave-one-group-out cross-validation procedure
to identify 8,000 images of both termite castes, which was signifi-
Termite compositionb cantly longer than the time to identify only a single caste (30–38 s,
P < 0.001, Table 2). In addition to the MobileNetV2 model, we shuf-
Cross-validationsa Group 1 Group 2 Group 3 fled our data sets for classification by three human termite experts.
CV1 Test Train Train The identification accuracy of MobileNetV2 models and human
CV2 Train Test Train experts was not significantly different (P = 0.61), indicating that
CV3 Train Train Test our deep learning models’ capabilities were fairly close to those of
the human experts in image recognition of the four termite species.
a
In each cross-validation, testing data sets were independent from training Notably, the machine was 65 times faster in classifying 8,000 im-
data sets. ages of the termite species than the human experts were (P < 0.001).
b
For detailed grouping information of termite species, colonies, and castes, Thus, the automatic classification system was deemed to be highly
refer to Fig. 1. efficient for termite identification.
Journal of Economic Entomology, 2021, Vol. XX, No. XX 5
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Fig. 3. Pipeline for image segmentation of individual termites. (A) The original full image; (B) a binary image by Otsu threshold; (C) filtering termites by certain
area size; (D) skeleton of individual-termite object; (E) rotating image straight by Hough transform; and (F) bounding box of a single termite and mask.
Table 2. Classification performance of MobileNetV2 (MobNet) models and human experts
Classifiers Experiments Testing time (s) Accuracy Average accuracy
CV1 33 0.963
MobNet-Soldiera CV2 38 0.973 0.947
CV3 36 0.905
CV1 36 0.930
MobNet-Workera CV2 30 0.941 0.946
CV3 35 0.968
CV1 243 0.919
MobNet-both-castesa CV2 235 0.936 0.930
CV3 231 0.934
Expert1 17,043 0.963
Human-both-castes Expert2 13,039 0.898 0.942
Expert3 15,934 0.966
a
Experiments were conducted using the LOGO CV procedure (Table 1).
Figure 4 presents the confusion matrix of the final testing results Although the performance of the classifiers of both castes was
of termite classification. Accordingly, we can visually evaluate the not significantly different from the classifiers of a single caste
performance of each classifier and determine which classes were more (Table 2), the MobNet both-caste models could mistakenly clas-
easily misidentified by the deep learning models or human experts. sify termites as the wrong morphotype in the same species, e.g.,
For the classifiers using soldier caste images as the input, the majority in O. formosanus (Fig. 4I–K). Notably, C. formosanus workers
of images were correctly predicted (Fig. 4A–C). By examining the in- and R. flaviceps workers were most often falsely classified by
correct predictions of other classes for each species, the MobileNetV2 the MobileNetV2 models of both castes (Fig. 4L). The three
model of the soldier caste did not exhibit preference bias (Fig. 4D). human experts experienced the same problem in classifying the
For the classifiers using worker caste images as the input, the numbers C. formosanus and R. flaviceps workers, as indicated by the high
of correct predictions of C. formosanus and R. flaviceps workers were number of wrong answers for these 2 morphotypes in their con-
significantly less than those of Cr. domesticus and O. formosanus fusion matrixes (Fig. 4N–P). The statistical analyses also indi-
by the MobileNetV2 models (Fig. 4E–G, P = 0.0041). In addition, cated that the number of R. flaviceps workers falsely classified as
the C. formosanus workers were more often falsely classified as C. formosanus workers was significantly higher than that of other
R. flaviceps workers than the other two species (Fig. 4H). morphotypes (Fig. 4Q).6 Journal of Economic Entomology, 2021, Vol. XX, No. XX
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Fig. 4. Confusion matrix of three MobileNetV2 classifiers and human experts. The number of testing data in each class was 1,000. The testing images in each CV
experiment were from different termite colonies. The MobileNetV2 classification results of soldier (A–C), worker (E–G), and both castes (I–K) are shown. Termite
image classification results of three human experts are also shown (N–P). Multiple comparison tests on the falsely classified results across three CV experiments
using one-way ANOVA, followed by the post hoc Tukey’s HSD test for paired difference with a P < 0.05 (D, H, L, and Q). C.d., Cr. domesticus; C.f., C. formosanus;
R.f., R. flaviceps; O.f., O. formosanus; S, soldier; W, worker.
Increasing Prediction Performance by Augmentation popularity of smartphones leads to an increasing case report of new
of a Small Number of Images termite infestation from the general public annually. According to
Classification accuracy decreased when fewer input images were our own experience, however, the number of termite taxonomists
used in training, in contrast to the models that achieved the highest who could provide identification service is much fewer than actual
accuracy by using 1,000 images per morphotype (Fig. 5). This ef- demand. Herein, we proposed a MobileNet model, which is suitable
fect was confirmed by a significantly negative correlation between for mobile phones, to classify worker and soldier castes of four ter-
mean accuracy and a reduction in the number of training images mite pest species in Taiwan. Among the three major termite castes,
(Pearson’s correlation coefficient = −0.9825, P = 0.0028). When we the alate and soldier castes present more morphological character-
used extensive data augmentation to increase the quantity of images istics for species identification than the worker caste. Hence, these
in the smallest dataset (i.e., 200 images per morphotype) to 1,000 two castes were used in most previous termite taxonomic studies
images, the model performance significantly increased to the average (Krishna et al. 2013). Of the four termite species submitted to this
accuracy of 0.907 across three cross-validations compared with the study, alates are the most distinctive caste for species identifica-
model using only 200 images per morphotype (P = 0.0017; Fig. 5). tion because of their varying body size, body color, and wing color
In addition, the test accuracy of three CV experiments using data characteristics (Yang and Li 2012). However, use of the alate caste
augmentation reached a similar performance as that of models using for species identification encounters several limitations: 1) alates
a complete data set containing 1,000 images per morphotype per only occur seasonally, 2) only mature colonies produce alates, and
group (P = 0.05; Fig. 5). In short, we demonstrated that augmen- 3) alates are usually collected during dispersal flights. Furthermore,
tation methods can improve the performance of the deep learning locating the infestation sites of swarming alates is challenging. Even
model if the image data set is limited. though soldiers and workers exhibit fewer distinctive morphological
characteristics than alates, they are commonly found at infestation
sites, facilitating identification and termite control implementation.
Our deep learning model was designed to solve the practical termite
Discussion control problem instead of serving as the basis for conducting taxo-
In this study, we explored a possibility to develop a technique, which nomic studies.
could be installed in a regular smartphone, for identifying common On the basis of termite behavior and ecology, termite pests can be
termite pest species based on images captured by smartphones. The divided into three groups: subterranean termites, drywood termites,Journal of Economic Entomology, 2021, Vol. XX, No. XX 7
(Inception v3) achieved state-of-the-art performance in identifying
crop pests because it could deal with the complicated backgrounds
presented by farmland scenes with several techniques, including
image augmentation, transfer learning, and parameter adjustment
(Li et al. 2020). Further experiments are required to fine-tune our
MobileNetV2 models to classify termite images from complex nat-
ural environments. However, the collection of numerous termite pest
images in household environments could improve the termite clas-
sification system through a social media curation strategy, which is
currently running in Taiwan (Facebook Group 2021: Termite and
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Alates Forum). Yet another difficult challenge is to recognize other
termite species, which are not collected in this study. In a very recent
paper (Goodwin et al. 2021), the authors proposed a two-step image
recognition process to extract relevant features from the image for
classification of known species and detection of unknown species
by comparison with all CNN features in training, respectively. Their
detection algorithm in a separate step obtained a promising perform-
ance of the CNN model to address the open-set novel species detec-
tion problem. This method can be developed further for real-world
applications.
Fig. 5. Classification accuracy of MobileNetV2 trained with different numbers Termites, as social insects, appear as multiple castes and individ-
of input images. Image classification accuracy decreased along with a uals in nests, tunnels, and infestation sites, which makes classification
decreased number of training images from 1,000 to 200 per morphotype. challenging but can also be advantageous. When using smartphones
*The 1,000 augmented images derived from 200 individual-termite images to acquire an image, multiple termite individuals belonging to the
per morphotype by using the method illustrated in Supp Fig. 1 (online only).
same colony usually appear. Hence, they must be the same species.
The classification accuracy of models was significantly increased with data
augmentation (P = 0.0017, t test) and was not significantly different from that
We obtained multiple individuals’ images by using the image seg-
trained by using 1,000 original individual-termite images (P = 0.05, t test). mentation methods developed in this study and classified them in-
dividually. In this manner, we could generate multiple identification
results on which to base the final decision, preventing incorrect judg-
and gardening and agriculture termite pests. No universal control ments based on only one individual image. In addition, we could
methods are applicable to all three termite groups; therefore, correct combine the characteristics of both intraspecific castes, workers, and
identification is vital for choosing termite control tools. Practically, soldiers, to increase the accuracy.
classifying infesting termites into respective termite control groups is In addition to classifying individual termites, the interaction
sufficient for termite control, and identification of the correct species patterns of intracolonial individuals or distinctive behaviors could
or even caste is not necessary. For example, the workers and sol- also be potential characteristics for species classification. For ex-
diers of O. formosanus are more likely misclassified as each other by ample, when termite’s foraging mud tubes are broken, soldiers of
MobileNetV2 models (Fig. 4I–K), which may be due to their similar C. formosanus guard at the exit, but soldiers of O. formosanus hide
head color. The head of O. formosanus workers is yellowish, which and workers defense intruders, which are usually ants. Even though
is much more similar to the soldier caste, compared with the other the morphological characteristics of soldiers of C. formosanus and
three species (Fig. 1). In this case, misidentification between intra- O. formosanus are similar, their behaviors markedly differ. In add-
specific castes of O. formosanus leads to the same termite control ition to termite images, the shape, color, and size of termite-made
group: gardening and agriculture termite pest. constructions, such as tunnels, shelter tube, and carton materials
In this study, a high percentage of misidentification occurred filling vacant space, are also distinctive images for termite species
between C. formosanus and R. flaviceps by human experts and identification. The drywood termites excrete fecal pellets, which ex-
MobileNetV2 models (Fig. 4H, L, Q). Because both C. formosanus hibit a distinctive size and a hexagonal cylinder shape. No other type
and R. flaviceps are subterranean termites that share the same con- of termite or wood-destroying organism produces such fecal pellets.
trol tools, for termite control implementation, misidentification can To summarize, the MobileNetV2 classifiers of both castes yielded
still lead to choosing the correct control tool. If we were to analyze satisfactory performance, nearly reaching the level of human experts,
the classification accuracy of the three termite control groups, the but improvements can be made by increasing the number and quality
accuracy would be higher than the current results (Table 2, Fig. 4). of termite images. In addition, termite pest species occurring in other
Although MobileNetV2 can achieve satisfactory performance— countries or geographic regions should be considered for examin-
similar to that of human experts (Table 2, Fig. 4), certain limita- ation. An image-based identification system can facilitate precise ter-
tions were evident, such as it being trained and tested on cropped mite control by providing immediate support to pest control agents
termite images with a minimal and simple background. Most real- as well as ordinary citizens. Further research should evaluate how
world pests appear against highly complex backgrounds, which in- the deep learning model performs when using other computer vision
creases the range of classification challenges and errors as well as techniques to improve the recognition outcome. Our ultimate goal is
problems of termite detection. In recent years, several deep learning providing the termite image classifiers for installing in regular smart-
architectures have been applied to classify pest images in natural phones under commercial operation systems. We also have released
scenes, achieving a high classification accuracy of approximately the full model trained on our complete data set, so other researchers
90%. For example, generative adversarial networks have been ap- might benefit by the pretrained parameters to develop sophisticated
plied to extend training datasets and improved the plant disease models including other termite species or different CNN architec-
classification accuracy to 92% using a pretrained CNN model tures in mobile devices. The immediate identification result will
(Gandhi et al. 2018). In another approach, the fine-tuned GoogleNet lead the user to biological information of the termite pest, control8 Journal of Economic Entomology, 2021, Vol. XX, No. XX
options, control products, and even professional pest management Huang, S. Y. 2020. Survey of household termite species composition,
service in reachable distance. The high identification efficiency and swarming season, and termite hybridization. M.S. thesis, National Chung
accuracy will improve the integrated pest management of termites. Hsing University, Taichung, Taiwan.
Huang, L. P., M. H. Hong, C. H. Luo, S. Mahajan, and L. J. Chen. 2018.
A vector mosquitoes classification system based on edge computing and
Supplementary Data deep learning, pp. 24–27. In Proceedings of the 2018 Conference on
Technologies and Applications of Artificial Intelligence, Taichung, Taiwan.
Supplementary data are available at Journal of Economic doi:10.1109/TAAI.2018.00015
Entomology online. Ju, T., M. L. Baker, and W. Chiu. 2007. Computing a family of skeletons
of volumetric models for shape description. Comput. Aided Des. 39:
352–360.
Acknowledgments
Downloaded from https://academic.oup.com/jee/advance-article/doi/10.1093/jee/toab162/6359981 by guest on 07 September 2021
Kingma, D. P., and J. Ba. 2015. Adam: a method for stochastic optimization,
We thank three termite experts, Chun-I Chiu, Chia-Chien Wu, and pp. 1–15. In Proceedings of the 3rd International Conference for Learning
Representations, San Diego, CA. https://arxiv.org/abs/1412.6980
Wei-Ren Liang, for helping with the classification of termite images.
Krishna, K., D. A. Grimaldi, V. Krishna, and M. S. Engel. 2013. Treatise on the
This study was supported by Ministry of Science and Technology,
Isoptera of the world. Bull. Am. Mus. Nat. Hist. 377: 1–2704.
Taiwan, grants MOST108-2221-E-001-014-MY3 to H.-K.T. and
Lai, Y. Y. 2019. Termite infestation survey of protected old trees and
MOST105-2628-B-005-003-MY3 to H.-F.L. baiting Formosan subterranean termite (Blattodea: Rhinotermitidae) in
mango trees. M.S. thesis, National Chung Hsing University, Taichung,
Taiwan.
Data Availability
Leonardo, M. M., T. J. Carvalho, E. Rezende, R. Zucchi, and F. A. Faria.
The code developed for this work is available at Github: https:// 2018. Deep feature-based classifiers for fruit fly identification (Diptera:
github.com/bio-it-station/DL-termite-identification. Of note, the Tephritidae), pp. 41–47. In Proceedings of the 31st Conference on Graphics,
pretrained weights using all images of three termite groups (the full Patterns and Images, Parana, Brazil. doi:10.1109/SIBGRAPI.2018.00012
model) are released as well. Li, Y., H. Wang, L. M. Dang, A. Sadeghi-Niaraki, and H. Moon. 2020. Crop
pest recognition in natural scenes using convolutional neural networks.
Comput. Electron. Agric. 169: 105174.
References Cited Li, H. F., H. T. Yeh, Y. N. Wang, and M. J. Tsai. 2011. Termite diversity and
Cardim Ferreira Lima, M., M. E. Damascena de Almeida Leandro, C. Valero, damage pattern in tropical botanical garden of Taiwan. J. Exp. For. Nat.
L. C. Pereira Coronel, and C. O. Gonçalves Bazzo. 2020. Automatic detec- Taiwan Univ. 25: 139–147.
tion and monitoring of insect pests – a review. Agriculture 10: 161. Liang, C. C. 2017. Termite species composition in soil and feces of Formosan
Cheng, X., Y. Zhang, Y. Chen, Y. Wu, and Y. Yue. 2017. Pest identification via pangolin (Manis pentadactyla pentadactyla) at Luanshan, Taitung.
deep residual learning in complex background. Comput. Electron. Agric. M.S. thesis, National Pingtung University of Science and Technology,
141: 351–356. Pingtung, Taiwan.
De Boer, P. T., D. P. Kroese, S. Mannor, and R. Y. Rubinstein. 2005. A tutorial Martineau, M., R. Raveaux, C. Chatelain, D. Conte, and G. Venturini. 2018.
on the cross-entropy method. Ann. Oper. Res. 134: 19–67. Effective training of convolutional neural networks for insect image recog-
Deng, J., W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: a nition, pp. 426–437. In J. Blanc-Talon, D. Helbert, W. Philips, D. Popescu,
large-scale hierarchical image database, pp. 248–255. In Proceedings of and P. Scheunders (eds.), Advanced concepts for intelligent vision systems.
the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Springer, Cham, Switzerland.
Miami, FL. doi:10.1109/CVPR.2009.5206848 Otsu, N. 1979. A threshold selection method from gray-level histograms. IEEE
Dawei, W., D. Limiao, N. Jiangong, G. Jiyue, Z. Hongfei, and H. Zhongzhi. Trans. Syst. Man Cybern. 9: 62–66.
2019. Recognition pest by image-based transfer learning. J. Sci. Food Perez, L., and J. Wang. 2017. The effectiveness of data augmentation in image
Agric. 99: 4524–4531. classification using deep learning. arXiv:1712.04621.
Duda, R. O., and P. E. Hart. 1972. Use of the Hough transformation to detect Potter, M. F. 2011. Termites, pp. 292–441. In D. Moreland (ed.), Handbook of
lines and curves in pictures. Commun. ACM 15: 11–15. pest control. The Mallis Handbook Company, Cleveland, OH.
Edwards, R., and A. E. Mill. 1986. Termites in buildings: their biology and Roberts, D. R., V. Bahn, S. Ciuti, M. S. Boyce, J. Elith, G. Guillera-Arroita,
control. The Rentokil Library, Kent, United Kingdom. S. Hauenstein, J. J. Lahoz-Monfort, B. Schröder, W. Thuiller, et al. 2017.
Evans, T. A., B. T. Forschler, and J. K. Grace. 2013. Biology of invasive ter- Cross-validation strategies for data with temporal, spatial, hierarchical, or
mites: a worldwide review. Annu. Rev. Entomol. 58: 455–474. phylogenetic structure. Ecography 40: 913–929.
Facebook Group. 2021. Termite and alates forum. (https://www.facebook. Rust, M. K., and N. Y. Su. 2012. Managing social insects of urban importance.
com/groups/398182670532605). Annu. Rev. Entomol. 57: 355–375.
Gandhi, R., S. Nimbalkar, N. Yelamanchili, and S. Ponkshe. 2018. Plant Sandler, M., A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen. 2018.
disease detection using CNNs and GANs as an augmentative approach, MobileNetV2: inverted residuals and linear bottlenecks, pp. 4510–4520.
pp. 1–5. In Proceedings of the 2018 IEEE International Conference on In Proceedings of the IEEE Conference on Computer Vision and Pattern
Innovative Research and Development, Bangkok, Thailand. doi:10.1109/ Recognition, Salt Lake City, UT. doi:10.1109/CVPR.2018.00474
ICIRD.2018.8376321 Shorten, C., and T. M. Khoshgoftaar. 2019. A survey on image data augmen-
Goodwin, A., S. Padmanabhan, S. Hira, M. Glancey, M. Slinowsky, tation for deep learning. J. Big Data 6: 1–48.
R. Immidisetti, L. Scavo, J. Brey, B. M. M. Sai Sudhakar, T. Ford, et al. Weesner, F. W. 1965. The termites of the United States. The National Pest
2021. Mosquito species identification using convolutional neural net- Control Association, Elizabeth, NJ.
works with a multitiered ensemble model for novel species detection. Sci. Wäldchen, J., and P. Mäder. 2018. Machine learning for image based species
Rep. 11: 13656. identification. Methods Ecol. Evol. 9: 2216–2225.
Haralick, R. M., and L. G. Shapiro. 1991. Computer and robot vision, Vol. 1. Xie, C., J. Zhang, R. Li, J. Li, P. Hong, J. Xia, and P. Chen. 2015. Automatic
Addison-Wesley, Boston, MA. classification for field crop insects via multiple-task sparse represen-
Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, tation and multiple-kernel learning. Comput. Electron. Agric. 119:
M. Andreetto, and H. Adam. 2017. MobileNets: efficient convolutional 123–132.
neural networks for mobile vision applications. arXiv:1704.04861. Yang, R. L., and H. F. Li. 2012. Taxonomy and identification of the five
https://arxiv.org/abs/1704.04861 common termite species in Taiwan. Formosan Entomol. 32: 169–198.You can also read
