A Novel Hybrid Autoregressive Integrated Moving Average and Artificial Neural Network Model for Cassava Export Forecasting

Warut Pannakkong; Van-Nam Huynh; Songsak Sriboonchitta

doi:10.2991/ijcis.d.190909.001

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Volume 12, Issue 2, 2019, Pages 1047 - 1061

A Novel Hybrid Autoregressive Integrated Moving Average and Artificial Neural Network Model for Cassava Export Forecasting

Authors

Warut Pannakkong¹, Van-Nam Huynh²^{, *}, Songsak Sriboonchitta³

¹School of Manufacturing Systems and Mechanical Engineering, Sirindhorn International Institute of Technology, Thammasat University, Pathumthani, Thailand

²School of Knowledge Science, Japan Advanced Institute of Science and Technology, Ishikawa, Japan

³School of Economics, Chiang Mai University, Chiang Mai, Thailand

^*Corresponding author. Email: huynh@jaist.ac.jp

Corresponding Author

Van-Nam Huynh

Received 11 April 2019, Accepted 3 September 2019, Available Online 26 September 2019.

DOI: 10.2991/ijcis.d.190909.001 How to use a DOI?
Keywords: Time series forecasting; Hybrid model; Artificial neural network; ARIMA; Cassava export
Abstract: This paper proposes a novel hybrid forecasting model combining autoregressive integrated moving average (ARIMA) and artificial neural network (ANN) with incorporating moving average and the annual seasonal index for Thailand's cassava export (i.e., native starch, modified starch, and sago). The comprehensive experiments are conducted to investigate the appropriate parameters of the proposed model as well as other forecasting models compared. In particular, the proposed model is experimentally compared to the ARIMA, the ANN and the other hybrid models according to three popular prediction accuracy measures, namely mean square error (MSE), mean absolute error (MAE) and mean absolute percentage error (MAPE). The empirical results show that the proposed model gives the lowest error in all three measures for the native starch and the modified starch which are major cassava exported products (98% of the total export volume). However, the Khashei and Bijari's model is the best model for the sago (2% of the total export volume). Therefore, the proposed model can be used as an alternative forecasting method for stakeholders making a decision in cassava international trading to obtain better accuracy in predicting future export of native starch and modified starch which are the majority of the total export.
Copyright: © 2019 The Authors. Published by Atlantis Press SARL.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

Time series forecasting is an important research area which has attracted a lot of attention from research communities in numerous practical fields including statistics, business, econometrics, finance, weather forecasting, earthquake prediction, etc. Over the past several decades, many attempts have been made by researchers for the development of efficient forecasting models to continuously improve forecasting accuracy [1].

Cassava (Manihot esculenta Crantz) is a main source of calories, after rice and maize, for the world's population particularly in developing countries. In addition, animal foods and ethanol production for alternative energy industries use the cassava as a raw material [2]. It is worth emphasizing that most available cassava in the international market are exported from Thailand such that Thailand's future cassava export quantity can support decision-makers involved in the cassava supply chain to improve production planning, policies making for helping cassava farmers, the profit of cassava trading in the future market, etc. Although the future cassava export quantity from Thailand is important in the cassava international trading, but to our best knowledge, so far, the researches of the cassava export forecasting are limited to using the autoregressive integrated moving average (ARIMA) model [3,4], and in Pannakkong et al. [5], we have applied the artificial neural network (ANN) model for cassava export forecasting and compared its performances to the ARIMA model. This paper is a further attempt in hybridizing the ARIMA model and ANN in forecasting cassava export.

2. LITERATURE REVIEW

One of the most popular statistical techniques for time series forecasting is ARIMA model, which has been widely used due to its capability in dealing with both stationary and nonstationary time series. The ARIMA model is good at linear modeling and has an assumption of the linearity of the associated time series [6]. In fact, the linearity assumption of time series made in the ARIMA model or its special instances is difficult to meet in many practical situations. In order to overcome this limitation, various nonlinear forecasting models have been proposed in the literature; among them the ANN model has received increasing attention for nonlinear time series forecasting due to its benefits of being considered as a universal function approximator and data-driven model without any assumptions [7]. The ANN model has been applied in several practical situations [8–12]. The comparative studies between the ANN and the ARIMA models have been performed as well. In most cases, the comparative results show that the ANN model has better prediction accuracy than the ARIMA model; however, there are still some cases that the ARIMA can outperform the ANN model [13].

The mixed results implied that neither the ARIMA model nor the ANN model is the best model for all problems. In particular, it seems to be inappropriate to use the ARIMA model to fit nonlinear data and apply the ANN model to linear data as well. Furthermore, it is difficult to clearly identify the properties of the data from real-world applications. To reduce the risk of selecting unsuitable model, recently, hybrid models of the ARIMA and the ANN models, which have the capability in both linear and nonlinear modeling, have been developed [14,15]. Such models use the ARIMA model to capture linear component from the time series, and then the ANN model is used to capture the nonlinear component from residuals of the ARIMA model.

The hybridization has been proved that it can provide a higher prediction accuracy than using individual models alone in several applications. The followings are researches adopting the hybrid model of Zhang [14]. Faruk [16] proposed a hybrid ANN and ARIMA model for water quality time series prediction. Pai and Lin [17] developed a hybrid ARIMA and support vector machines (SVM) model for stock price forecasting. Chun-Ling Lin and Shyu [18] presented an application of hybrid multi-model forecasting system involving ARIMA and ANN models focusing on market demand of the display market in Taiwan. He et al. [19] proposed a hybrid ARIMA and SVM for short term load forecasting in Hebei province, China. Bouzerdoum et al. [20] also proposed a hybrid model of seasonal autoregressive integrated moving average (SARIMA) and SVM for short-term power forecasting but it is for a small-scale grid-connected photovoltaic plant.

In day-ahead electricity price forecasting, Zhang et al. [21] introduced a new hybridization of ARIMA and least squares support vector machine (LSSVM) for the Australian national electricity market. Shafie-Khah et al. [22] presented a hybrid model based on ARIMA and radial basis function neural networks (RBFN) for mainland Spain electricity price. These two papers used wavelet transform to decompose the time series and applied particle swarm optimization (PSO) to optimize the models. Furthermore, Chaâbane [23] developed a hybrid model of autoregressive fractionally integrated moving average (ARFIMA) and neural network model for electricity price prediction in Nordpool, Norway.

Chen and Wang [24] proposed a hybrid SARIMA and SVM to predict the production value of Taiwan machinery industry. Chen [25] Combining linear and nonlinear model using ARIMA, ANN and SVM in forecasting tourism demand of Taiwanese outbound tourists. Aslanargun et al. [26] compared the performance of ARIMA, ANN, and their hybrid models in forecasting amount of monthly tourists visiting Turkey.

Lo [27] conducted a study of applying ARIMA and SVM models to predict the number of failure in software execution. Shi et al. [28] assessed capability of hybrid approaches of ARIMA, ANN, and SVM models in forecasting wind speed and power in Colorado. Cadenas and Rivera [29] developed a hybrid ARIMA and ANN model to predict wind speed in three districts of Mexico. Díaz-Robles et al. [30] proposed a hybrid model of ARIMA and ANN models in order to forecast particulate matter occurring in metropolitan using Temuco in Chile as the case study. Barak and Sadegh [31] introduced ARIMA–adaptive neuro fuzzy inference system (ANFIS) hybrid algorithm, a combination of ARIMA and ensemble ANFIS, to forecast energy consumption in Iran.

Moreover, there are several research works applied Khashei and Bijari's model [15] to real-world applications. Zhu and Wei [32] presented a novel combination of ARIMA and LSSVM for carbon price forecasting. Ruiz-Aguilar et al. [33,34] introduced hybrid approaches based on SARIMA and ANN to forecast the number of inspection goods in customs and border controls at the Port of Algeciras Bay in Spain, which is the top ten of Europeans ports. [35] developed a hybrid model by combining SARIMA and ANN to predict annual energy cost budget in South Korea. Recently, Babu and Reddy [36] extended Khashei and Bijari Khashei and Bijari [15] model by investigating the nature of volatility of the data with moving-average filter before applying ARIMA and ANN models.

It is of interest to note that, however, these hybrid models consider only lagged values of the time series as their input, there may be an opportunity to improve their prediction quality by including processed variables such as moving average (MA) and annual seasonal index into the models. In this paper, we propose a new hybrid model that also combines the ARIMA and the ANN models but additionally incorporates the MA and the annual seasonal index into the model for Thailand's cassava export forecasting.

To determine the appropriate structure of the cassava export forecasting models, the comprehensively experimental comparisons are conducted, and then the performances of the proposed, the ARIMA, the ANN, and the Khashei and Bijari's models [15] are evaluated and compared.

The rest of paper is organized as follows. Section 3 introduces the time series models for the cassava export forecasting. Section 4 explains the formulation of the proposed model. Section 5 presents and compares the experimental results of cassava export forecasting. Section 6 shows additional experiments with other two seasonal time series related to agriculture. Finally, Section 7 provides conclusions.

3. TIME SERIES FORECASTING MODELS

3.1. ARIMA Model For Cassava Export Forecasting

The ARIMA model is a popular statistical model for stationary and nonstationary time series forecasting during several past decades. Typically, this model is an integration of autoregressive (AR) and MA models, including data transformation term called differencing. However, as mentioned above, the ARIMA model has several limitations due to the linearity assumption which is hard to be fully satisfied in real-world applications, or the use of only historical time series as the model's inputs.

The formulation of the ARIMA model comes up with autoregressive moving average (ARMA) model, which is a special case of the ARIMA model, as in (1).

(1)Zt=c+∑i=1pϕiZt−i+at−∑j=1qθjat−j

The ARMA model predicts a time series at period t(Zt) by using lagged values of time series (Zt−1,…,Zt−p) and lagged random errors (at−1,…,at−q) where ϕi and θj are the model parameters; c is a constant; at is a random error with mean of zero and a constant variance of σ2; p and q are orders of the lagged values included in the model.

In order to simplify the mathematical formula, backward shift operator (B), defined as BiZt=Zt−i, is substituted for the ordinary algebraic symbol in (1), thus, the ARMA model can be formulated as (2) below.

(2)Zt=c+∑i=1pϕiZtBi+at−∑j=1qθjatBj

Then, by rearranging the terms related to Zt in (2), we obtain the ARMA model as (3).

(3)(1−∑i=1pϕiBi)Zt=c+(1−∑j=1qθjBj)at

which is compactly rewritten as

(4)ϕp(B)Zt=c+θq(B)at

where

ϕp(B)=1−∑i=1pϕiBi

θq(B)=1−∑j=1qθjBj

are called the AR operator (ϕp(B)) and the MA operator (θq(B)), respectively.

Note that, nevertheless, the ARMA model has no capability to deal with nonstationary time series, in this case, the differencing is required to transform the nonstationary into stationary time series by substitute (1−B)dZt for Zt in (4), where d is the degree of differencing. Then, we can obtain the ARIMA model as follows.

(5)ϕp(B)(1−B)dZt=c+θq(B)at

In a situation where there is seasonality in time series, the seasonal model components such as seasonal AR operator (ΦP(Bs)) and seasonal MA operator (ΘQ(Bs)) respectively defined by

ΦP(Bs)=1−∑k=1PΦkBks

ΘQ(Bs)=1−∑l=1QΘlBls

are included in (5) to capture the relationship of the seasonality. Then, the seasonal ARIMA model, or SARIMA for short, can be formulated as

(6)ϕp(B)ΦP(Bs)(1−B)d(1−Bs)DZt=c+θq(B)ΘQ(Bs)at

where s is the time span for a season, P is the seasonal order of AR, Q is the order of seasonal MA, and D is the degree of seasonal differencing.

In this study, the SARIMA model with 12 months seasonal time span is applied to the cassava starch export time series due to the characteristic of the cassava as an agricultural product that is influenced by the factors which have an annual seasonal pattern such as seasonal weather, harvesting cycle and government policy.

Box et al. [6] described a manual approach for obtaining best-fit parameters of ARIMA models. Nonetheless, in this study, the best-fit parameters are automatically determined by IBM SPSS Statistics software such that it can reduce error from a manual selection of the parameters. In addition, we can also construct a hundred scenarios of the models and compare them at once rather than doing it manually for one model at a time.

3.2. ANN Model for Cassava Export Forecasting

The ANN model, which is a kind of artificial intelligent technique mimicking biological neurons mechanism, is a well-known tool in time series forecasting in term of usage flexibility because there is no assumption on inputs and it has self-learning ability as human brain neurons [13].

The structure of the ANN model, graphically depicted in Fig. 1, consists of nodes which are located in three types of layers: input layer; hidden layer and output layer. Usually, there is only one input layer and one output layer, while the number of the hidden layer can be more than one. Nevertheless, ANN with one hidden layer is capable to approximate any continuous function [7].

In general, the number of nodes and layers depend on experience and the level of understanding in the problem of the architect because so far, there is no theory for selecting the best parameters for ANN models. Hence, a fashionable approach is the trial and error until obtaining the appropriate parameters [13,38].

At each node (Fig. 2), the relationship of the input and the output can be expressed by

(7)avl=f(∑u=1Uaul−1wuvl+bvl) =f(nvl)

where U denotes the total number of nodes in layer l−1, l ≤ L is an integer that represents the considered layer, L denotes the total number of layers in the network, and f denotes a transfer function.

For example, at node v in layer l, the inputs (aul−1) which are the output from nodes u=1,…,U in the previous layer (l−1) are aggregated with weights (wuvl) and a bias (bvl) to produce a net input (nvl). Then, the net input (nvl) is passed through a transfer function (f) to compute an output (avl) which will become an input for the next layer.

After the output of the ANN model (e.g., forecasted export) is generated from the output layer, the output is compared to the target (e.g., actual export) to measure prediction accuracy. The ANN model can learn for minimizing the forecasting error via a training algorithm attempting to determine the weight (wuvl) and bias (bvl) that fit the relationship between the inputs and the target.

In this research, a feed-forward ANN [37] is applied for the cassava starch export forecasting. The training algorithm is Levenberg-Marquardt algorithm with Bayesian regularization [39]. The idea of how to construct the ANN model for the cassava starch export forecasting in this paper is inspired by Pannakkong et al. [5] with an extension of the experimental scenarios. The following sections explain how the structure of the ANN model is identified.

3.2.1. Input layer

The input layer consists of the input nodes representing input variables (e.g., historical export quantity). Theoretically, the input variables can be divided into two types: technical variables and fundamental variables [38]. The technical variables are lagged values (e.g., time series value at time t−1) or processed values (e.g., MA) of time series. The fundamental input variables are other variables believed that there is an existing relationship between them and the dependent variable (e.g., month in the year).

In this study, there are totally nine input variables; six of them are the technical variables: three lagged values at time t−1,t−3 and t−12 (denoted by Zt−1, Zt−3 and Zt−12); two MAs with three and twelve previous periods (denoted by MA(3) and MA(12)); and an annual seasonal index. The remainings are fundamental variables corresponding to three time indices: number of the period (Sequence), month in the year (Month), and number of the quarter (Quarter).

In the experiments, there are three input scenarios that are designed based on the different combination of these nine inputs. The first input scenario is to include all nine inputs into the ANN model. For the second input scenario, the correlation analysis is performed first to screen the variables that have a statistically significant correlation (P-value ≤ 0.05) with the actual cassava export and then, the variables that can pass the correlation analysis are included in the ANN model. The last input scenario consists of the three time indices only. All three input scenarios of each cassava starch export are shown in Tables 1 and 2.

Scenario 1	Scenario 2	Scenario 3
All	Correlated	Time indices
Sequence	Sequence	Sequence
Month	Zt−1	Month
Quarter	Zt−3	Quarter
Zt−1	Zt−12
Zt−3	MA(3)
Zt−12	MA(12)
MA(3)	Seasonal index
MA(12)
Seasonal index

Table 1

ANN inputs for native starch and modified starch.

Scenario 1	Scenario 2	Scenario 3
All	Correlated	Time indices
Sequence	Sequence	Sequence
Month	MA(3)	Month
Quarter	MA(12)	Quarter
Zt−1	Seasonal index
Zt−3
Zt−12
MA(3)
MA(12)
Seasonal index

Table 2

ANN inputs for Sago.

3.2.2. Output layer

The output layer is the layer that produces the results of the ANN model by aggregating the outputs from the hidden layer as:

(8)akl=flin(∑ j=1Jajl−1wjkl+bkl) =flin(nkl)

where j and k are nodes in the hidden layer (the previous layer) and the output layer respectively, ajl−1 is the output from node j in the hidden layer, wjkl is the weight between node j of the hidden layer and node k of the output layer, bkl is the bias of node k, nkl is the net input of node k and akl is the output of node k. In order to obtain the output (akl) (i.e., forecasted exports), the net input (nkl) is passed through the pure linear transfer function (flin), then the outputs (akl) are stored in a memory for further analyses.

3.2.3. Hidden layer

The hidden layer, the middle layer between the input and the output layers, generates the outputs by aggregation of the outputs from the previous layer with the weights, the biases, and the nonlinear transfer function, like the following:

(9)ajl=ftan-sig(∑i=1Iail−1wijl+bjl) =ftan-sig(njl) =21+e−2njl−1

where i and j are nodes in the previous layer and the hidden layer respectively, ail−1 is the output from node i in the previous layer, wijl is the weight between node i of the previous layer and node j of the hidden layer, bjl is the bias of node j, njl is the net input of node j. The output of the hidden layer at node j (ajl) is the result of substituting the net input (njl) into the tan-sigmoid transfer function (ftan-sig), then, the output (ajl) is adopted as the input of the next layer.

In this study, only one hidden layer is used; however, the number of hidden nodes is varied from one to 20 nodes. Thus, totally, there are 180 ANN models constructed based on three types of the cassava export, three scenarios of the inputs and 20 scenarios of hidden nodes.

3.3. Khashei and Bijari's Hybrid Model

Generally, time series in real-world applications are rarely to have only pure linear or nonlinear characteristics [14]. In this case, the capability of the single forecasting models may not be enough to capture the relationship between the historical and future time series.

To overcome this circumstance, Zhang [14] has developed a hybrid model of the ARIMA and the ANN models in order to obtain capability in linear and nonlinear modeling from the ARIMA and the ANN modes respectively. However, in the Zhang's model, the relationship between linear and nonlinear components is assumed to be additive. This assumption may reduce the prediction accuracy of the Zhang's model if the relationship does not satisfy the assumption. Moreover, in some cases, the single forecasting models can perform even better than the Zhang's model [40].

Recently, Khashei and Bijari [15] modified the Zhang's model by proposing a novel hybrid model that defines time series as a function of its linear and nonlinear components. This relationship can be formally formulated as:

(10)yt=f(Lt,Nt)

where Lt and Nt denote the linear and nonlinear components, respectively.

This hybrid model includes three stages: 1) extracting linear component (Lt) from the time series by using the ARIMA model, and 2) defining nonlinear components as functions of lagged values of the ARIMA residuals (et) and lagged values of the time series (zt) and 3) applying the ANN model to determine the function representing the relationship between time series and the linear and nonlinear components. The detail of these three stages is described in the following paragraphs.

In the first stage, the ARIMA model is applied to time series as presented in Section 3.1. The results of the ARIMA model are considered as the linear component of the time series.

In the second stage, Khashei and Bijari assumes that the nonlinear pattern still exists in the ARIMA residuals and the time series. Thus, the nonlinear components are defined as the functions of the lagged values of the ARIMA residuals and the lagged values of the time series as (Nt1) and (Nt2) respectively.

(11)Nt1=f1(et−1,…,et−n)

(12)Nt2=f2(zt−1,…,zt−m)

where f1 and f2 are the nonlinear functions determined by the ANN model; n and m are integers representing the number of maximum previous periods included in the model. The residuals of the ARIMA model at time t (et) are computed as in (13).

(13)et=zt−L^t

where L^t denotes the result of the ARIMA model and zt denotes the time series at time t.

In the third stage, the time series can be represented by the function of the linear and the nonlinear components as:

(14)yt=f(L^t,Nt1,Nt2) =f(L^t,et−1,…,et−n,zt−1,…,zt−m)

where f is the nonlinear function determined by the ANN model; n and m are integers that are determined in the design process by varying them from one to twelve.

Furthermore, seasonal ANN (SANN) can be employed in the Khashei and Bijari's model rather than typical ANN. The SANN captures seasonal nonlinear patterns in data from values of the same period in the pervious cycle(s). Therefore, the Khashei and Bijari (seasonal) model can be derived as:

(15)yt=f(L^t,Nt1,Nt2) =f(L^t,et−1,…,et−n1,zt−s,zt−2s,…,zt−ks)

where s is the number of periods in the cycle (12 months), k is number of pervious cycles included in the model. This study uses both Khashei and Bijari model (14) and seasonal Khashei and Bijari model (15) as benchmarks in performance comparison.

4. PROPOSED HYBRID MODEL

In Pannakkong et al. [5], we have applied the ANN model for the cassava starch export forecasting which yielded a better accuracy than the ARIMA model. As discussed above, the hybrid models such as the Zhang's hybrid model [14] and the Khashei and Bijari's hybrid model [15] involve only lagged values of the time series for the prediction which may be inadequate for capturing complicated patterns in time series.

In the following, we develop a novel hybrid ARIMA and ANN model by further incorporating the MA and the seasonal index into the model, as graphically depicted in Fig. 3. The MA are the averages of previous N period of time series. The time series included in the MA correspond to the current period (t). The mathematical formula of the MA can be expressed as:

(16)MAt(N)=∑n=1Nzt−n

where MAt(N) is the average time series of pervious N periods at current period t, and N is number of previous periods included in the MA.

The seasonal index is the ratio between the time series at considered period and the average of all time series in the cycle. If the time series value in a period is higher than the average value of its cycle, its seasonal index will be greater than one. Conversely, the seasonal index will be lower than one when the time series value is lower than the average value of its cycle. The number of periods for each cycle is the length of the cyclic pattern that can be identified by autocorrelation analysis of the time series. Usually, time series related to agriculture have the cyclic pattern every 12 months as they are affected by seasonal weather. The seasonal index can be computed as below:

(17)Seasonal indext=zt∑p=1sz(p+((nc−1)×s))

where Seasonal indext is the seasonal index at period t, s is the number of periods in the cycle (12 months), nc is the number of the current cycle, and p is the period in the cycle. However, in case of future time series, their seasonal index cannot be calculated. Therefore, the seasonal index for future time series is the average of the seasonal index at the same period in all previous cycles.

The proposed model starts with separating the cassava time series into the training and testing set. Then, the linear and nonlinear components of the training set are determined as in the Khashei and Bijari's model.

Additionally, instead of taking only lagged values of time series into consideration, our proposed model also includes the correlated input variables shown in the second column of Tables 1 and 2 (denoted by Ct1,…,Cto) as an additional nonlinear component Nt3 as defined in (18).

(18)Nt3=f3(Ct1,…,Cto)

where f3 is the nonlinear function determined by the ANN model. The number of statistically correlated significant variables is o. For instance, from Table 1, the native starch has seven correlated significant variables (o = 7) including Sequence (Ct1), zt−1 (Ct2), zt−3 (Ct3), zt−12 (Ct4), MA(3) (Ct5), MA(12) (Ct6), and Seasonal index (Ct7). Eventually, the proposed model can be written as in (19) below:

(19)yt=f(L^t,Nt1,Nt2,Nt3) =f(L^t,et−1,…,et−n1,zt−1,…,zt−m1,Ct1,…,Cto)

where f is the nonlinear function determined by the ANN model; n1≤n and m1≤m are integers that are determined in the design process.

In time series forecasting problem, the number of input nodes corresponds to the number of the lagged values, which is used to discover the underlying pattern in a time series and to make forecasts for future values [13]. Therefore, n1 and m1 are varied in the experiments from one to n and m, which are set to 12 because of intention to cover annual seasonal pattern in time series.

To determine the appropriate parameters of the proposed model, firstly, the results of the ARIMA model are generated as discussed in Section 3.1 and their residuals are computed. Secondly, the suitable n1 and m1 are identified by running the hybrid model in Khashei and Bijari [15] while varying n1, m1 and the number of hidden nodes from one node to 20 nodes. The forecasting results are compared with the testing set to evaluated the performance. The values of n1 and m1 that give the lowest mean absolute percentage error (MAPE) are used for the proposed model. Finally, the proposed model is run while the number of hidden modes are varied as in the second step to find a suitable number of hidden nodes for the proposed model.

5. EXPERIMENTAL RESULTS

5.1. Cassava Starch Export Time Series

The cassava starch export time series contains monthly historical records of three types of the cassava export (i.e., native starch, modified starch, and sago) for 13 years (2001–2013). In term of average export quantity, the native starch is the first rank followed by the modified starch and the sago. In addition, these time series are nonstationary because their mean and variance are not constant as implied in Figs. 4–6.

There is uncertainty in the time series of cassava export quantity. In the long term, the cassava export quantity is increased due to the world's population growth and need for alternative energy resources. In short term, within a year, there is somewhat annual seasonality in the cassava time series due to the relevant events which are usually repeated annually such as seasonal weather, environmental issues (e.g., drought and pests), annual cassava pawn policy of the government, selling whole pawned cassava stock of the government in every September, and farmers' behaviors in the harvesting cycle.

Data partitioning is required to split the time series into training set and test set. The training set contains 144 monthly data during 2001-2012. The test set involves 12 monthly data in 2013 which is an unknown future for the forecasting models.

5.2. Forecasting Accuracy Measures

To measure the forecasting accuracy, three popular measures, namely mean square error (MSE), mean absolute error (MAE) and MAPE, are computed as in (20–22) respectively. Let Zt denotes the actual value at t, Zt^ denotes the forecasted value at t and N denotes the amount of total forecasting period. The smaller value of these measures expresses more accurate forecasting.

(20)MSE=1N∑t=1N(Zt−Zt^)2

(21)MAE=1N∑t=1N|Zt−Zt^|

(22)MAPE=1N∑t=1N|Zt−Zt^|Zt

5.3. Forecasting Performance Comparison

After the comprehensive experimental runs, the forecasting accuracy measures are computed and compared in order to find out a suitable prediction model for each cassava starch. The MSE, MAE and MAPE of the best models in each type of forecasting model and each cassava type are presented in Tables 3 and 4.

Cassava Export	Forecasting Model	MSE	MAE	MAPE (%)
	ARIMA (1, 1, 0)(0, 1, 1)12	501,751,335	16,699	13.96
Native Starch	ANN (7-15-1)^b	2,383,777	1,091	1.10
	Khashei and Bijari [15] (n=1,m=2)	432,234,143	15,411	13.07
	Khashei and Bijari [15] (n=1,k=4)	425,421,264	15,226	12.29
	Proposed^b(n=1,m=2,o=5)	87,655,928	7,522	6.60

	ARIMA (0, 1, 3)(1, 0, 1)12	41,578,514	5,186	9.84
Modified Starch	ANN (9-19-1)^a	3,948,821	1,464	2.75
	Khashei and Bijari [15] (n=6,m=2)	27,960,066	4,221	7.79
	Khashei and Bijari [15] (n=6,k=2)	42,033,243	5,220	9.34
	Proposed^a(n=6,m=2,o=6)	116,941	247	0.49

	ARIMA (0, 0, 1)(1, 0, 0)12	490,409	370	19.70
Sago	ANN (4-6-1)^b	19,679	111	6.03
	Khashei and Bijari [15] (n=12,m=12)	51,479	167	9.74
	Khashei and Bijari [15] (n=12,k=3)	998,749	617	33.95
	Proposed^b(n=12,m=12,o=4)	9,529	73	4.04

ANN, artificial neural network; ARIMA, autoregressive integrated moving average; MAE, mean absolute error; MAPE, mean absolute percentage error; MSE, mean square error.

a

With technical and fundamental inputs.

b

With significant correlated inputs.

Table 3

Training performance of cassava export forecasting.

Cassava Export	Forecasting Model	MSE	MAE	MAPE (%)
	ARIMA (1, 1, 0)(0, 1, 1)12	2,954,264,813	46,344	21.02
Native Starch	ANN (7-15-1)^b	978,697,046	23,738	12.19
	Khashei and Bijari [15] (n=1,m=2)	1,186,289,692	27,565	14.95
	Khashei and Bijari [15] (n=1,k=4)	2,807,270,432	45,063	20.58
	Proposed^b(n=1,m=2,o=5)	631,588,845	19,462	10.79

	ARIMA (0, 1, 3)(1, 0, 1)12	25,259,327	4,513	6.06
Modified Starch	ANN (9-19-1)^a	16,987,064	3,185	4.25
	Khashei and Bijari [15] (n=6,m=2)	21,870,151	3,797	5.01
	Khashei and Bijari [15] (n=6,k=2)	34,223,857	5,032	6.64
	Proposed^a(n=6,m=2,o=6)	12,628,092	2,893	3.84

	ARIMA (0, 0, 1)(1, 0, 0)12	306,717	460	19.54
Sago	ANN (4-6-1)^b	191,042	319	13.41
	Khashei and Bijari [15] (n=12,m=12)	131,358	302	12.44
	Khashei and Bijari [15] (n=12,k=3)	270,294	453	20.08
	Proposed^b(n=12,m=12,o=4)	229,176	369	16.10

ANN, artificial neural network; ARIMA, autoregressive integrated moving average; MAE, mean absolute error; MAPE, mean absolute percentage error; MSE, mean square error.

a

With technical and fundamental inputs.

b

With significant correlated inputs.

Table 4

Testing performance of cassava export forecasting.

The best model for the native starch is the proposed model followed by the ANN model, the Khasei and Bijari's model and the ARIMA model. From Fig. 7, The ANN is more accurate than the proposed in January to June. On the other hand, the proposed model outperforms the ANN in July to December. The ARIMA underestimates the actual export in all month except June. The Khasei and Bijari's model can capture the actual trend roughly.

For the modified starch, the proposed model is the best model followed by the ANN model, the Khasei and Bijari's model [15] and the ARIMA model respectively. From January to May, all models can capture the direction of the future export except the ARIMA model in May (see Fig. 8). After May, the forecasting direction seems to be opposite to the actual value until October but in November and December, all models give the same forecasting direction.

In the case of the sago, surprisingly, the best model is the Khasei and Bijari's model [15] followed by the ANN model, the proposed model and, the ARIMA model respectively. The Khasei and Bijari's model [15] can obviously track the direction of the future export from January until June. The other models can track the direction only in January, March, and April (see Fig. 9). In addition, the ANN and the proposed model perform almost the same. The ARIMA model gives somewhat similar values for the whole year. It seems that the 12 lagged values in the Khashei and Bijari's model can cover the pattern the time series of the sago. Therefore, including the correlated variables into the model incurs overfitting of the historical time series (i.e., training set).

From the overall results, prediction accuracy in testing is worse than training except for the ARIMA and seasonal Khashei and Bijari's models in sago prediction. The ARIMA model has almost similar MAPE in both training and testing because it is robust for the outlier in testing set of the sago. The seasonal Khashei and Bijari's model in testing gives better performance than training, however, its performance is still the worst among all forecasting models as the sago time series has no obvious seasonality.

In training, the proposed model performs the best in all types of the cassava starch except the native starch. The ANN model has 1.10% of MAPE following by the proposed model having 6.60% of MAPE. However, in testing, MAPE of the ANN model increases by 12 times (13.41% of MAPE) approximately while MAPE of the proposed model increases by less than 2 times (10.79% of MAPE). This evidence implies that the overfitting problem occurs during the training of the ANN model.

The seasonal Khashei and Bijari's model has less accurate prediction than typical Khashei and Bijari's model because the cassava export time series are not pure seasonal time series and the seasonal Khashei and Bijari's model can capture only the seasonal pattern from SARIMA and SANN without obtaining recent information from recent lagged values. However, typical Khashei and Bijari's model obtains seasonal pattern from SARIMA and also receives the pattern within cycle from recent lagged values.

In summary, for the reason that the proposed model outperforms the Khashei and Bijari's model [15] in predicting the native starch and the modified starch, there may be remaining nonlinear component that cannot be captured with the residuals of ARIMA and the lagged values but it can be captured in which the correlated inputs are included in the model. On the other hand, the Khashei and Bijari's model [15] is the best model for the sago because the proposed and the ANN models, which are more complex in term of inputs, give more weight to an extreme outlier that causes the overfitting problem as the proposed model has the best performance in training stage (Table 3).

6. EXPERIMENTS WITH OTHER SEASONAL TIME SERIES

The previous section implies the capability of the proposed model in cassava export forecasting. However, the structure of the proposed model is not limited for only cassava export time series. This section illustrates the effectiveness of the proposed model to the other two seasonal time series (i.e., milk production and temperature).

The monthly milk production time series [41] was collected during January 1962 to December 1975 (168 observations) as shown in Fig. 10. The cyclic pattern is obviously repeated annually. It has an increasing trend from 1962 to 1971. After that, the production mean seemed to be stable. The observations from 1962 to 1974 are used in the training stage. The remaining observations in 1975 are testing data.

The mean daily temperature of Fisher River near Dallas [42] was continuously recorded from 1988 to 1991. The daily observations are aggregated into 48 monthly observations as presented in Fig. 11. The mean of temperature is quite stable during these four years. The peak of temperature usually occurs in the middle of the year. However, the patterns in each year are slightly different. The first three years (1988 to 1990) observations are training set. The testing set is observations in 1991.

The correlation analyses are performed to find their significant correlated inputs as presented in Table 5. After these two time series are analyzed by the proposed model and benchmarking models, the performance in training and testing is shown in Tables 6 and 7. The forecasting values are plotted in Figs. 12 and 13.

Milk Production	Temperature
Sequence	Month
Zt−1	Quarter
Zt−3	Zt−1
Zt−12	Zt−12
MA(3)	MA(3)
MA(12)	MA(12)
Seasonal index	Seasonal index

Table 5

Significant correlated inputs for milk production and temperature.

Time Series	Forecasting Model	MSE	MAE	MAPE (%)
	ARIMA (0, 1, 1)(0, 1, 1)12	55.71	5.71	0.77
Milk Production	ANN (7-10-1)^a	5.19	1.75	0.24
	Khashei and Bijari [15] (n=6,m=12)	12.15	2.33	0.31
	Khashei and Bijari [15] (n=6,k=3)	44.66	4.85	0.64
	Proposed^a(n=6,m=12,o=6)	11.72	2.32	0.30

	ARIMA (0, 0, 0)(0, 1, 0)12	9.78	2.39	127.14
Temperature	ANN (7-2-1)^a	0.35	0.47	31.64
	Khashei and Bijari [15] (n=2,m=6)	2.68	1.14	94.19
	Khashei and Bijari [15] (n=2,k=1)	6.63	1.92	52.40
	Proposed^a(n=2,m=6,o=6)	2.44	1.05	92.08

ANN, artificial neural network; ARIMA, autoregressive integrated moving average; MAE, mean absolute error; MAPE, mean absolute percentage error; MSE, mean square error.

a

With significant correlated input.

Table 6

Training performance of milk production and temperature forecasting.

Time Series	Forecasting Model	MSE	MAE	MAPE (%)
	ARIMA (0, 1, 1)(0, 1, 1)12	160.65	10.99	1.28
Milk Production	ANN (7-10-1)^a	169.22	11.46	1.35
	Khashei and Bijari [15] (n=6,m=12)	147.07	10.77	1.24
	Khashei and Bijari [15] (n=6,k=3)	156.32	11.59	1.34
	Proposed^a(n=6,m=12,o=6)	132.55	10.58	1.22

	ARIMA (0, 0, 0)(0, 1, 0)12	17.24	3.35	949.43
Temperature	ANN (7-2-1)^a	11.21	2.75	839.90
	Khashei and Bijari [15] (n=2,m=6)	16.56	3.30	141.77
	Khashei and Bijari [15] (n=2,k=1)	28.43	4.90	803.96
	Proposed^a(n=2,m=6,o=6)	16.05	3.25	120.47

ANN, artificial neural network; ARIMA, autoregressive integrated moving average; MAE, mean absolute error; MAPE, mean absolute percentage error; MSE, mean square error.

a

With significant correlated input.

Table 7

Testing performance of milk production and temperature forecasting.

From the results, there are some interesting points to be noted. Training performance is better than testing performance because test data set in unknown for the prediction models. ANN gives the best performance in training for both milk production and temperature.

Training and testing MAPEs of milk production are not quite different. Conversely, in case of temperature, testing MAPE of some models is dramatically increased from training MAPE. The reason is that MAPE is sensitive when mean of data set is low. Mean of temperature is very low (approximately zero), therefore small predict error can cause huge changing in MAPE.

Considering the testing performance, they indicate that the proposed model outperforms all benchmarking models in milk production forecasting with lowest error in all three measures. For temperature forecasting, the proposed model can perform the best in only MAPE and give the second-lowest MSE and MAE, while ANN has the lowest MSE and MAE. This result implies that when mean of data set is very low, both the proposed model and ANN should be applied and then their results should be compared based on user preferences.

7. CONCLUSION

In recent years, hybridization of the ARIMA and the ANN models have been proved to often provide a more accurate prediction than individual models used separately. However, there is a limitation regarding the input adequacy of previously developed hybrid models. In this paper, we proposed a new hybrid model for the cassava export forecasting, which also combines the ARIMA model and ANN and additionally considers the MA and the annual seasonal index along with the lagged values of the time series as the inputs for fitting the nonlinear relationship. After conducting the comprehensive experiments, the prediction performances of the proposed hybrid model are compared to the ones by the ARIMA model, the ANN model and the Khashei and Bijari's model [15].

In conclusion, the proposed model shows its capability in the cassava export forecasting with the highest accuracy in MSE, MAE, and MAPE comparing with the ARIMA model, the ANN model, the Khashei and Bijari's model and the seasonal Khashei and Bijari's model in case of the native starch and the modified starch. However, for sago, the Khashei and Bijari's hybrid model is the best. Although, the proposed model is not suitable for sago, it can give the most accurate forecast for the native starch and the modified starch which covers around 98% of the total export volume.

Regarding the capability of the proposed model, it can contribute the cassava international trading market as an alternative forecasting model for the major volume of the cassava export. The forecasting results from the proposes model would be useful for the stakeholders who make a decision based on the future cassava starch export. Furthermore, as the cassava is a seasonal time series, this hybrid model can be applied to other seasonal time series that share the similar characteristic as it provides the best performance in all three measures in predicting milk production, and the best MAPE in predicting temperature.

Nevertheless, the experiments conducted are limited to 12 months ahead prediction, three replication runs for each scenario, one hidden layer and twenty hidden nodes. In future work, we intend to apply a technique (e.g., response surface methodology) to find optimal hyperparameters rather than trial and error to obtain more reliable results, and also make an attempt to find out a solution to deal awith the outlier of the sago.

CONFLICT OF INTEREST

The authors confirm that all funding sources supporting the work and all institutions or people who contributed to the work, but who do not meet the criteria for authorship, are acknowledged. The authors also confirm that all commercial affiliations, stock ownership, equity interests or patent licensing arrangements that could be considered to pose a financial conflict of interest in connection with the work have been disclosed.

AUTHORS' CONTRIBUTIONS

W. Pannakkong and V.N. Huynh conceived of the presented idea. W. Pannakkong developed the proposed model and performed the experiments. V.N. Huynh and S. Sriboonchitta supported W. Pannakkong in verifying the analytical methods. All authors discussed the results, provided critical feedback, and helped shape the research, analysis, and manuscript. W. Pannakkong and V.N. Huynh wrote the manuscript in consultation with S. Sriboonchitta. V.N. Huynh sent the manuscript for publication and communicated with the journal editor.

Funding Statement

This work was supported by the SIIT Young Researcher Grant, under contract no. SIIT 2017-YRG-WP05.

ACKNOWLEDGEMENTS

The authors would like to thank the anonymous reviewers for providing insightful comments and providing directions for additional work which has resulted in this paper. This work was supported by the SIIT Young Researcher Grant, under contract no. SIIT 2017-YRG-WP05.

REFERENCES

1.J.G. De Gooijer and R.J. Hyndman, 25 years of time series forecasting, Int. J. Forecast., Vol. 22, 2006, pp. 443-473.

2.Food and Agriculture Organization of the United Nations, The world cassava economy, 2000. http://www.fao.org/3/x4007e/X4007E00.htm#TOC

3.C. Kongcharoen and T. Kruangpradit, Autoregressive integrated moving average with explanatory variable (arimax) model for thailand export, in 33rd International Symposium on Forecasting (South Korea), 2013, pp. 1-8. https://forecasters.org/wp-content/uploads/gravity_forms/7-2a51b93047891f1ec3608bdbd77ca58d/2013/07/Kongcharoen_Chaleampong_ISF2013.pdf

4.C. Prapasornpittaya, Comparative study on ARIMA, intervention and transfer function models in forecasting Thailand's export value, Thammasat University, Pathum Thani, Thailand, 2013. Unpublished master's thesis https://koha.library.tu.ac.th/cgibin/koha/opac-detail.pl?biblionumber=680825

5.W. Pannakkong, V.-N. Huynh, and S. Sriboonchitta, Arima versus artificial neural network for thailands cassava starch export forecasting, V.-N. Huynh, V. Kreinovich, and S. Sriboonchitta (editors), Causal Inference in Econometrics, 2016, pp. 255-277.

6.G. Box, G. Jenkins, and G. Reinsel, Time Series Analysis: Forecasting and Control, 2008. https://books.google.co.jp/books?id=lJnnPQAACAAJ

7.R. Hecht-Nielsen, Theory of the backpropagation neural network, IEEE, in International Joint Conference on Neural Networks, IJCNN (Washington), 1989, pp. 593-605.

8.D.C. Park, M. El-Sharkawi, R. Marks, L. Atlas, and M. Damborg, Electric load forecasting using an artificial neural network, IEEE Trans. Power Syst., Vol. 6, 1991, pp. 442-449.

9.J.-H. Wang and J.-Y. Leu, Stock market trend prediction using arima-based neural networks, in IEEE International Conference on Neural Networks (Washington), 1996, pp. 2160-2165.

10.G. Zhang and M.Y. Hu, Neural network forecasting of the British pound/us dollar exchange rate, Omega, Vol. 26, 1998, pp. 495-506.

11.A. Chaouachi, R.M. Kamel, and K. Nagasaka, Neural network ensemble-based solar power generation short-term forecasting, J. Adv. Comput. Intell. Intell. Inform., Vol. 14, 2010, pp. 69-75.

12.K.G. Abistado, C.N. Arellano, and E.A. Maravillas, Weather forecasting using artificial neural network and bayesian network, J. Adv. Comput. Intell Intell. Inform., Vol. 18, 2014, pp. 812-817.

13.G. Zhang, B.E. Patuwo, and M.Y. Hu, Forecasting with artificial neural networks: the state of the art, Int. J. Forecast., Vol. 14, 1998, pp. 35-62.

14.G.P. Zhang, Time series forecasting using a hybrid arima and neural network model, Neurocomputing, Vol. 50, 2003, pp. 159-175.

15.M. Khashei and M. Bijari, A novel hybridization of artificial neural networks and arima models for time series forecasting, Appl. Soft Comput., Vol. 11, 2011, pp. 2664-2675.

16.D.Ö. Faruk, A hybrid neural network and arima model for water quality time series prediction, Eng. Appl. Artif. Intell., Vol. 23, 2010, pp. 586-594.

17.P.-F. Pai and C.-S. Lin, A hybrid arima and support vector machines model in stock price forecasting, Omega, Vol. 33, 2005, pp. 497-505.

18.C.-L.L. Chen-Chun Lin, Hybrid multi-model forecasting system: a case study on display market, Knowl. Based Syst., Vol. 71, 2014, pp. 279-289.

19.Y. He, Y. Zhu, and D. Duan, Research on hybrid arima and support vector machine model in short term load forecasting, in Sixth International Conference on Intelligent Systems Design and Applications (Jinan), Vol. 1, 2006, pp. 804-809.

20.M. Bouzerdoum, A. Mellit, and A.M. Pavan, A hybrid model (sarima–svm) for short-term power forecasting of a small-scale grid-connected photovoltaic plant, Sol. Energy, Vol. 98, 2013, pp. 226-235.

21.J. Zhang, Z. Tan, and S. Yang, Day-ahead electricity price forecasting by a new hybrid method, Comput. Ind. Eng., Vol. 63, 2012, pp. 695-701.

22.M. Shafie-Khah, M.P. Moghaddam, and M. Sheikh-El-Eslami, Price forecasting of day-ahead electricity markets using a hybrid forecast method, Energy Convers. Manag., Vol. 52, 2011, pp. 2165-2169.

23.N. Chaâbane, A hybrid arfima and neural network model for electricity price prediction, Int. J. Electr. Power Energy Syst., Vol. 55, 2014, pp. 187-194.

24.K.-Y. Chen and C.-H. Wang, A hybrid sarima and support vector machines in forecasting the production values of the machinery industry in Taiwan, Expert Syst. Appl., Vol. 32, 2007, pp. 254-264.

25.K.-Y. Chen, Combining linear and nonlinear model in forecasting tourism demand, Expert Syst. Appl., Vol. 38, 2011, pp. 10368-10376.

26.A. Aslanargun, M. Mammadov, B. Yazici, and S. Yolacan, Comparison of arima, neural networks and hybrid models in time series: tourist arrival forecasting, J. Stat. Comput. Simul., Vol. 77, 2007, pp. 29-53.

27.J.-H. Lo, A study of applying arima and svm model to software reliability prediction, in 2011 International Conference on Uncertainty Reasoning and Knowledge Engineering (URKE) (Bali), Vol. 1, 2011, pp. 141-144.

28.J. Shi, J. Guo, and S. Zheng, Evaluation of hybrid forecasting approaches for wind speed and power generation time series, Renew. Sustain. Energy Rev., Vol. 16, 2012, pp. 3471-3480.

29.E. Cadenas and W. Rivera, Wind speed forecasting in three different regions of mexico, using a hybrid arima–ann model, Renew. Energy, Vol. 35, 2010, pp. 2732-2738.

30.L.A. Díaz-Robles, J.C. Ortega, J.S. Fu, G.D. Reed, J.C. Chow, J.G. Watson, and J.A. Moncada-Herrera, A hybrid arima and artificial neural networks model to forecast particulate matter in urban areas: the case of Temuco, Chile, Atmos. Environ., Vol. 42, 2008, pp. 8331-8340.

31.S. Barak and S.S. Sadegh, Forecasting energy consumption using ensemble arima–anfis hybrid algorithm, Int. J. Electr. Power Energy Syst., Vol. 82, 2016, pp. 92-104.

32.B. Zhu and Y. Wei, Carbon price forecasting with a novel hybrid arima and least squares support vector machines methodology, Omega, Vol. 41, 2013, pp. 517-524.

33.J. Ruiz-Aguilar, I. Turias, and M. Jimenéz-Come, Hybrid approaches based on sarima and artificial neural networks for inspection time series forecasting, Transp. Res. Part E Log. Trans. Rev., Vol. 67, 2014, pp. 1-13.

34.J. Ruiz-Aguilar, I. Turias, and M. Jiménez-Come, M. A novel three-step procedure to forecast the inspection volume, Transp. Res. Part C Emer. Technol., Vol. 56, 2015, pp. 393-414.

35.K. Jeong, C. Koo, and T. Hong, An estimation model for determining the annual energy cost budget in educational facilities using sarima (seasonal autoregressive integrated moving average) and ann (artificial neural network), Energy, Vol. 71, 2014, pp. 71-79.

36.C.N. Babu and B.E. Reddy, A moving-average filter based hybrid arima–ann model for forecasting time series data, Appl. Soft Comput., Vol. 23, 2014, pp. 27-38.

37.J.E. Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold Co., New York, 1990.

38.I. Kaastra and M. Boyd, Designing a neural network for forecasting financial and economic time series, Neurocomputing, Vol. 10, 1996, pp. 215-236.

39.D.J. MacKay, A practical bayesian framework for backpropagation networks, Neural Comput., Vol. 4, 1992, pp. 448-472.

40.T. Taskaya-Temizel and M.C. Casey, A comparative study of autoregressive neural network hybrids, Neural Netw., Vol. 18, 2005, pp. 781-789.

41.J.D. Cryer, Time Series Analysis, 1, Wadsworth Publ. Co., Belmont, 1986.

42.K.W. Hipel and A.I. McLeod, Time Series Modelling of Water Resources and Environmental Systems, Elsevier, Vol. 45, 1994.

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Journal: International Journal of Computational Intelligence Systems
Volume-Issue: 12 - 2
Pages: 1047 - 1061
Publication Date: 2019/09/26
ISSN (Online): 1875-6883
ISSN (Print): 1875-6891
DOI: 10.2991/ijcis.d.190909.001 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - Warut Pannakkong
AU  - Van-Nam Huynh
AU  - Songsak Sriboonchitta
PY  - 2019
DA  - 2019/09/26
TI  - A Novel Hybrid Autoregressive Integrated Moving Average and Artificial Neural Network Model for Cassava Export Forecasting
JO  - International Journal of Computational Intelligence Systems
SP  - 1047
EP  - 1061
VL  - 12
IS  - 2
SN  - 1875-6883
UR  - https://doi.org/10.2991/ijcis.d.190909.001
DO  - 10.2991/ijcis.d.190909.001
ID  - Pannakkong2019
ER  -

download .riscopy to clipboard