a study of deep neural networks in stock trend prediction ......deep recurrent nn for forecasting....
TRANSCRIPT
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 1
A Study of Deep Neural Networks in Stock
Trend Prediction using Different Activation
Functions
B. Dhamayanthi, Ph.D. Research Scholar Dept. of Computer Applications
Madurai Kamaraj University Madurai, India.
J.Sharmila Vaiz, Ph.D. Research Scholar
Dept. of Computer Applications Madurai Kamaraj University
Madurai, India.
Dr. M. Ramaswami, Associate Professor Dept. of Computer Applications
Madurai Kamaraj University Madurai, India.
Abstract— Machine learning is a type of Artificial
Intelligence (AI) technique that analyzes the data and
finds hidden insights and automates analytical model
building process. Deep Neural Network (DNN) is one of
the paradigm for performing machine learning. DNN
consists of multiple processing layers that learn
representations of data with multiple levels of
abstraction. The purpose of an activation function in a
Deep Learning context is to ensure that the
representation in the input space is mapped to a different
space in the output. There are several activation functions
available in DNN such as Tanh and Tanh with Dropout,
Maxout and Maxout with Dropout, Rectifier and
Rectifier with Dropout. In this paper, we attempt to
implement effectiveness of different activation functions
of Deep Neural Network to make an optimal trade
decision by means of forecasting stock trend of six major
capitalization companies of NSE, India.
Keywords—Deep Neural Networks, Stock Trend,
Activation functions, Binary Classification
I. INTRODUCTION
A number of forecasting models have been developed over the past several years to predict the direction of movement of stock price. Conventional methods use historical data and develop the model for stock market forecasting. In these methods, it is assumed that the future will be exactly like the past except for those variables used by the model to develop a forecast. Major conventional methods used by stock analyst and researchers are regression analysis, time series decomposition, moving averages and smoothing methods, ARIMA (Box Jenkins methodology), etc. Regression models, for example, assume that the underlying population follows a normal distribution.
The field of AI research was started in a conference at Dartmouth College in 1956. AI is a branch of computer science dealing with the simulation of intelligent behavior of real-world problems in
computers. It is budding from the essence of physical, mathematical sciences and massive parallel computational power. Over the years, AI diversified into three major sub divisions: language processing, robotics and artificial neural network (ANN). ANN plays major role in most of the commercial applications, including forecasting. In stock market forecasting, AI can search through countless analysis and predict the stock price movement. In ANN processing system [9], the basic elements are called as neurons, transmitted the signals by connection links. The links possess an associated weight, which is multiplied along with the incoming signal (net input) for any typical neural net. The output signal is obtained by applying activations to the net input. One of the most important types of feed forward network is Back Propagation Network. Back Propagation using extend gradient-descent based delta-learning rule, commonly known as back propagation (of errors) rule.
Draw backs of neural network [3] are non convex problem, curse of dimensionality and vanishing gradient problem. DNN architecture used partial derivative method to avoid the limitations of basic neural network. In this study, we analyze the role of different types of activation functions of DNN in predicting the stock trend of six major capitalization companies of NSE of India.
II. LITERATURE REVIEW
Deep Learning Neural Networks [11] is the fastest growing field in machine learning. It serves as a powerful computational tool for solving prediction, decision, diagnosis, and detection based on a well-defined computational architecture. It has been successfully applied to a broad field of applications ranging from computer security, speech recognition, image and video recognition to industrial fault detection, medical diagnostics and stock market. Enzo Busseti et al [1] developed deep learning architectures to predict energy loads across different network grid
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 2
areas, using time and temperature data. They used hourly demand for four and a half years from 20 different geographic regions, and similar hourly temperature readings from 11 zones as dataset. They used different learning models such as Kernelized Regression, Frequency NN, Deep Feed forward NN, Deep Recurrent NN for forecasting. Finally, they concluded that deep neural network and recurrent neural network gives better accuracy for power load forecasting with evaluation measures of RMSE and Percentage of RMSE.
Matthew Dixon et al [2] described the application of deep neural networks on financial time series data in order to classify financial market movement directions. They used DNN application to back testing a simple trading strategy and demonstrate the prediction accuracy and its relation to the strategy profitability. They used 43 different Commodity and FX future 5-minute intervals mid-prices from June 1989 to March 2013 as dataset and implemented using C++.
Mladen Dalto et al [3] compared shallow and deep neural networks with two input variable selection algorithms on an ultra-short-term wind prediction task. Wind measured 10m relative heights from the ground in Split, Croatia from 2010 to 2012 were used as dataset. They used neural network with training (75%), validation (15%) and testing (15%). PMI and novel distance based IVS algorithm is used for reducing input size and overall number of parameters. Finally, they concluded that DNN obtained better result than shallow networks by using evaluation measure of MAE.
Wan He et al [4] proposed DNN based electric load forecast methods. Hourly data of weather information from February 1st 2000 to November 30th 2012 in North China were used for their study. They used RBM pre-training and discriminative pre-training technologies. Finally, they evaluated the performances of deep neural networks in load forecast using a large data set, which contains nearly three years loads.
Yangtuo Peng et al [5] proposed a simple method to leverage financial news to predict stock movements based on the popular word embedding and deep learning techniques. The financial news data were used, which contains 106,521 articles from Reuters and 447,145 from Bloomberg. The news articles were published in the time period from October 2006 to December 2013. The historical stock security data are obtained from the Centre for Research in Security Prices (CRSP). They concluded that prediction accuracy of the proposed model is higher than the baseline system that relies only on historical data.
Xiao Ding et al [6] perform a deep learning method for event-driven stock market prediction. Events are extracted from news text and represented as dense vectors and trained using a novel neural tensor network. 15 companies of S&P 500 were used as data set. They concluded that there is nearly 6% improvement on S&P 500 index prediction and individual stock prediction over baseline system.
III. MATERIALS AND METHODS
In this work we attempt to forecast the stock trend Using Deep Neural Networks using different activation functions. We use 22 technical indicators derived from OHLCV (Open, High, Low, Close Price, and Volume) historical data from January 2012 to December 2015 of six high market capitalization companies of NSE. Stock trend is forecasted using six activation functions of DNN using H2O package in R and compared their efficacy using performance measures like RMSE, MSE, F-measure, Classification Accuracy, AUC and Gini index.
A. Research data
In this paper, we use 22 technical indicators which are broadly classified under trend, momentum, volatility and volume indicators. The technical indicators used in this study are shown in Table 1. The six high market capitalization companies of NSE used in this study are: (i) Tata Consultancy Services Ltd. (TCS) (ii) Reliance Industries Ltd. (RIL) (iii) Housing Development Finance Corporation Ltd. (HDFC) (iv) Hindustan Unilever Ltd. (HUL) (v) Sun Pharmaceutical Industries Ltd. (SPIL) and (vi) I.T.C Ltd. (ITCL). Four year data from January 2012 to December 2015 of above six companies of NSE are used for our study.
B. Technical Indicators
The technical indicators and its usage are listed in Table 1. There are several types of moving averages SMA (Simple Moving Average), EMA (Exponential Moving Average), WMA (Weighted moving average), DEMA (Double Exponential Moving Average) and VWMA (Volume Weighted Moving Average). The usage of EMA indicator is if price is above EMA then it indicates buy signal else it indicates sell signal which is explained in Table 1.
C. Deep Neural Network
This basic framework of multi-layer neural networks can be used to accomplish Deep Learning tasks. Deep Learning architectures are models of hierarchical feature extraction, typically involving multiple levels of non linearity [2]. Deep Learning models are able to learn useful representations of raw data and have exhibited high performance on complex data such as images, speech, and text. The basic unit in the model is the neuron, a biologically inspired model of the human neuron. In humans, the varying strengths of neurons output signals travel along the synaptic junctions and are then aggregated as input for a connected neuron’s activation. In the model, the weighted combination
b+xwα iin
1i
of input signals is aggregated, and then an output signal f (α) is transmitted by the connected neuron.
Deep neural network is implemented using R platform. R is an open source programming language and software environment for statistical computing and it is widely used among statisticians and data miners
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 3
for developing statistical software and data analysis. Deep neural network algorithm can be implemented by the H20 package in R.
TABLE 1: LIST OF TECHNICAL INDICATORS
Technical Indicators Formula
Moving Averages(Var-EMA) If Price >= EMA then buy
If Price < EMA then sell
MACD (Var- MACD, signal) If MACD >= 0 or MACD
>= Signal then buy If MACD < 0 or MACD <
Signal then sell
ADX(Var - ADX, Dip, Din) If ADX > =25 and Dip >=
Din, then buy If ADX > 25 and Dip < Din
then sell
TDI(Var-TDI, DI) If TDI > 0 and DI > 0 then buy
If TDI > 0 and DI < 0 then
sell
Aroon(aroonup, aroondn) If aroonup > 70 and aroondn < 30 then buy
If aroondn > 70 and aroonup < 30 then sell
VHF(VHF)
If VHF is rising then buy
If VHF is falling then sell
RSI (RSI) If RSI > 70 then buy If RSI < 30 then sell
Stoch(fastD, slowD) If fastD>slowD then buy
If fastD<slowD then sell
SMI(SMI) If SMI > -40 then buy If SMI < 40 then sell
WPR(WPR) If WPR < -80 or WPR >-50
then buy
If WPR > -20 or WPR < -
50 then sell
CMO(CMO) If CMO < -50 then buy
If CMO > 50 then sell
CCI(CCI) If Price & CCI increase, same trend
If Price increase and CCI
decrease reverse trend
Bollinger Bands (BBands_pctB) If (BBands_pctB <=0) then
buy
If (BBands_pctB >0) then sell
Don chain Channel
(upper channel, lower channel)
If Price > upper channel
then buy
If Price < lower channel then sell
ATR(ATR) If ATR is high - volatile
else price is flat
CMF(CMF) If CMF > +0.05 then buy
If CMF < -0.05 then sell
OBV(OBV) Rising peaks and falling
troughs indicate trend else OBV is flat
MFI(MFI) If MFI < 20 then buy
If MFI > 80 then sell
1) Initialization: H2O uses a purely supervised training protocol. The default initialization scheme is the uniform adaptive option, which is an optimized initialization based on the size of the network. Deep Learning can also be started using a random initialization drawn from either a uniform or normal distribution, optionally specifying a scaling parameter.
2) Activation and Loss Functions: The choices for the nonlinear activation function f described in xi
and wi represent the firing neurons input values and their weights denotes the weighted combination
b+xwα iin
1i
The activation functions are Tanh, Rectifier, Maxout. Each of these activation functions can be operated with dropout regularization.
a) tanh: The tanh function is a rescaled and shifted logistic function; its symmetry around 0 allows the training algorithm to converge faster. Fig 1 shows the Tanh Activation Function Range plot and ―(1)‖ represents the formula of Tanh.
Figure 1 Tanh Activation Function plot
-(1)---[-1,1] f(.):Range αeαe
αeαeαf
b) Rectifier: The default activation function is the Rectifier. The Rectifier linear activation function has demonstrated high performance on image recognition tasks and is a more biologically accurate model of neuron activations. Fig. 2 shows the Rectifier Activation Function Range plot and ―(2)‖ represents the formula of Rectifier.
Figure 2. Rectifier Activation Function plot
(2)R.f:Range α0,maxαf
c) Maxout: Maxout is a generalization of the Rectified Linear activation, where each neuron picks the largest output of k separate channels, where each channel has its own weights and bias values. Fig. 3 shows the Maxout Activation function Range plot and ―(3)‖ denotes the Maxout activation formula.
(3)R.f:Range α2α1,maxα2α1,f
Figure 3 Maxout Activation Function plot
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 4
3) Overfitting: The huge dataset are able to implement complex nonlinear models with encountering the problem of overfitting [1]. Large networks are also slow to use [7], making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. By using regularization methods overfitting problem is avoided.
4) Regularization: H2O’s Deep Learning framework supports regularization techniques to prevent overfitting. (L1: Lasso) and (L2: Ridge) regularization enforce the same penalties as they do with other models: modifying the loss function so as to minimize loss:
-(4)---j) | B(W,Rλj) | B(W,Rλj) | BL(W,j) | B(W,L'2211
For L1 (Lasso) regularization, R1 (W, B/j) is the sum of all L1 norms for the weights and biases in the network; L2 (Ridge) regularization via R2 (W, B/j) represents the sum of squares of all the weights and biases in the network‖ (4)‖. The constants λ1 and λ2 are very small (for example 10
−5).
5) Dropout: The second type of regularization available for Deep Learning is a modern innovation called dropout. Dropout constrains the online optimization so that during forward propagation for a given training example, each neuron in the network suppresses its activation with probability P, which is usually less than 0.2 for input neurons and up to 0.5 for hidden neurons such as (TanhWithDropout, MaxoutWithDropout, or RectifierWithDropout).
IV. BUILDING THE MODEL
Four years OHLCV historical data from January 2012
to December 2015 are retrieved from Google Finance.
22 technical indicators values are derived from
OHLCV data for six major capitalization companies of
NSE. A binary classification variable price change is
calculated to decide the trade action buy/sell (Open
Price, Close Price). The difference between the close
price and the open price is assigned to price change
variable. If the difference is greater than zero, then the
close price is above the open price and it indicates buy
signal else it indicates sell signal. After deriving all the
variables it is fed into Deep Neural network which is
shown in Fig.4. All the derived technical indicator
variables are given as input to the DNN. B1, B2 and
B3 shows the bias value associated with DNN. Two
hidden layers are used in this study. Each hidden layer
is associated with ten neurons which are labeled as H1,
H2 … H10. Two output layers which are used to
decide the trade action (buy/sell) is labeled with O1
and O2. The deep neural network is trained with
different activation functions. The first hidden layer
and second hidden layer consist of 10 neurons. This
model is built by Bernoulli and multinomial loss
function and is primarily associated with cross-
entropy. And the models are trained by 6 different
activation functions with 400 epochs. The activation
function used in this model is listed in Table 2.
TABLE 2: DIFFERENT ACTIVATION FUNCTION’S
VARIABLE INDEX S No Activation functions Variable Index
1 Tanh TAN
2 Tanhwithdropout TANWD
3 Maxout MAX
4 Maxoutwithdropout MAXWD
5 Rectifier REC
6 Rectifierwithdropout RECWD
Fig 4: Best Deep Neural Network Architecture
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 5
V. RESULTS AND DISCUSSIONS
The proposed model is trained with six activation functions such as TAN, TANWD, MAX, MAXWD, REC and RECWD. The model is tested with various evaluation measures such as Accuracy, F-measure, RMSE, MSE, AUC and Gini etc.
Accuracy is a statistical measure to evaluate how well a binary classification correctly identifies or excludes a condition. It is the measure of the proportion of true results among the total number of cases examined. Among the six activation functions, MAXWD gives a high accuracy of 87.76% in HDFC and REC and RECWD gives an accuracy of 87.16% for TCS and ITCL which is listed in Table 3. Among six companies MAXWD gives high accuracy for HUL,
SPIL, RIL and HDFC which is depicted in Fig 5.
The traditional F-measure is the harmonic mean of precision and recall. Precision is the number of correct positive results divided by the number of all positive results and recall is the number of correct positive results divided by the number of positive results. The least F-measure accuracy is 76.9% for HUL using TAN function and the highest accuracy is 90% using REC function for TCS. From fig 5 it is noticed that MAXWD and REC gives high F-measure value.
Another performance indicator for binary classification is Receiver Operating Characteristic (ROC).It is graphical plot [8] of the true positive rate against the false positive rate for different possible cut points of a diagnostic test. The perfect classifier is
Company
id Activation functions Accuracy F-Measure AUC RMSE MSE Gini
HUL
TAN 0.7672 0.7692 0.8150 0.4746 0.2252 0.6291
TANWD 0.7582 0.8038 0.8107 0.4352 0.1894 0.6215
MAX 0.7403 0.7798 0.8122 0.4355 0.1896 0.6244
MAXWD 0.8418 0.8579 0.8944 0.3906 0.1526 0.7888
REC 0.7791 0.8127 0.8234 0.4140 0.1714 0.6467
RECWD 0.7552 0.7927 0.8040 0.4436 0.1968 0.6081
TCS
TAN 0.8328 0.8747 0.9140 0.3529 0.1246 0.8279
TANWD 0.8000 0.8553 0.8272 0.4539 0.2061 0.6544
MAX 0.8597 0.8834 0.9353 0.4020 0.1616 0.8705
MAXWD 0.8507 0.8809 0.9156 0.3401 0.1157 0.8312
REC 0.8716 0.9005 0.9252 0.3547 0.1258 0.8504
RECWD 0.8388 0.8726 0.9137 0.3430 0.1176 0.8273
SPIL
TAN 0.8030 0.8382 0.8704 0.3964 0.1572 0.7409
TANWD 0.7463 0.8114 0.8022 0.4251 0.1807 0.8022
MAX 0.8119 0.8454 0.8780 0.4003 0.1602 0.7560
MAXWD 0.8418 0.8723 0.8915 0.3852 0.1484 0.7829
REC 0.7881 0.8247 0.8664 0.4013 0.1610 0.7329
RECWD 0.7522 0.7896 0.8316 0.4099 0.1681 0.6633
RIL
TAN 0.803 0.8272 0.8771 0.4024 0.1619 0.7542
TANWD 0.7522 0.7852 0.8210 0.4288 0.1839 0.6421
MAX 0.8149 0.8447 0.8947 0.3766 0.1419 0.7894
MAXWD 0.8030 0.8290 0.8346 0.4127 0.1703 0.6692
REC 0.8537 0.8707 0.9250 0.3479 0.1210 0.8491
RECWD 0.7851 0.8209 0.8673 0.4041 0.1633 0.7346
ITCL
TAN 0.8299 0.8407 0.9037 0.3997 0.1597 0.8073
TANWD 0.7254 0.7737 0.7828 0.4662 0.2174 0.5657
MAX 0.8657 0.8703 0.9218 0.3684 0.1357 0.8436
MAXWD 0.8478 0.8661 0.9108 0.3556 0.126 0.8215
REC 0.8089 0.8120 0.8879 0.3987 0.1590 0.7757
RECWD 0.8716 0.8841 0.9281 0.3432 0.1178 0.8563
HDFC
TAN 0.7821 0.8184 0.8470 0.4208 0.1771 0.6940
TANWD 0.7881 0.8137 0.8471 0.4380 0.1918 0.6958
MAX 0.8328 0.8557 0.8869 0.4052 0.1642 0.7737
MAXWD 0.8776 0.8960 0.9386 0.3164 0.1001 0.8773
REC 0.8478 0.8722 0.9289 0.3865 0.1494 0.8577
RECWD 0.8081 0.8382 0.8714 0.3834 0.1470 0.7429
TABLE 3: ACCURACY MEASURES FOR 6 COMPANIES DIFFERENT ACTIVATION FUNCTIONS
International Conference on Recent Trends in Engineering, Computers, Information Technology and Applications (ICRTECITA-2017)
Seventh Sense Research Group www.internationaljournalssrg.org Page 6
located at (0, 1) with TPR=1 and FPR=0. It is difficult task to compare the classifiers when no classifier dominates the other.
In order to resolve this issue, we used another performance measure called Area under Curve (AUC). The AUC value for each ROC curve is measured by using trapezoidal rule and high AUC value among them indicates the corresponding classification method is close to (0,1).
The AUC values for each classification algorithm of six companies are calculated and shown in Table 3. Maximum AUC value is 93.8% for HDFC using MAXWD and minimum AUC value is 78.2% for ITCL using TANWD. From fig. 5 it is noticed that MAXWD gives high accuracy for HUL, SPIL, HDFC and MAX gives high accuracy for TCS and ITCL and RIL using REC activation function.
The Mean Squared Error (MSE) is a measure of how close a fitted line is to data points. Root-mean-square error (RMSE) is a measure of the differences between predicted by a model or an estimator and the values actually observed. MSE and RMSE for various activation functions are shown in Table 3.
The Gini index measures the inequality among the values of a frequency distribution. If Gini index value is 0, it expresses equality, where all values are the same. A Gini index of value 1 expresses maximal inequality among values. Gini index for six activation functions are listed in Table 3.
VI. CONCLUSION
In this work, we used Deep Neural Network with different activation functions to predict the stock market movement direction. We Compare the results of six companies and underline the fact that Rectifier and Maxout activation function provides better accuracy than Tanh activation function. The proposed model gives maximum accuracy of 87.76% with MAXWD function in predicting HDFC stock trend. In
this study we used 22 technical indicators in training Deep Neural Networks. In future researches, we aim at using feature selection to reduce the attributes and then train the model using DNN in order to increase the predicted accuracy.
References [1] Enzo Busseti, Ian Osband, Scott Wong, ―Deep Learning for
Time Series Modeling‖, CS 229 Final Project Report, December 14th, 2012.
[2] Matthew Dixon, Diego Klabjan, Jin Hoon Bang, ―Classification-based Financial Markets Prediction using Deep Neural Networks‖, arXiv:1603.08604v1 [cs.LG], 29 Mar 2016.
[3] Mladen Dalto, University of Zagreb, Faculty of Electrical Engineering and Computing ,‖Deep neural networks for time series prediction with applications in ultra-short-term wind forecasting".
[4] Wan He, ―Deep Neural Network Based Load Forecast‖, computer modelling & new technologies 2014 18(3) 258-26.
[5] Yangtuo Peng and Hui Jiang, ―Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks‖, Proceedings of NAACL-HLT 2016, pages 374–379,San Diego, California, June 12-17, 2016,Association for Computational Linguistics.
[6] Xiao Ding, Yue Zhang, Ting Liu, Junwen Duan, ―Deep Learning for Event-Driven Stock Prediction‖, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), pp: 2327-2333.
[7] Nitish Srivastava, Geoffrey Hinton hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, ―Dropout: A Simple Way to Prevent Neural Networks from Overfitting‖ , Journal of Machine Learning Research 15 (2014), pp: 1929-1958.
[8] Matjaz Majnik, Zoran Bosni, ―ROC Analysis of Classifiers in Machine Learning‖, Technical report MM-1/2011, University of Ljubljana, Faculty of Computer and Information Science, Trzaska cesta 25, Ljubljana, Slovenia.
[9] Bhagwant Chauhan, Umesh Bidave, Ajit Gangathade, Sachin Kale-―Stock Market Prediction Using Artificial Neural Networks‖, Bhagwant Chauhan et al, (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 5 (1) , 2014, pp: 904-907.
[10] Arno Candel Viraj Parmar Erin LeDell Anisha Arora Edited by: Jessica Lanford, Deep Learning with H2O, November 2015: Fourth Edition.
[11] Deep learning neural networks: design and case studies Kindle Edition by Daniel Graupe world scientific.
Figure 5: Accuracy, F-Measure, AUC values of 6 company’s different activation functions
0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
HUL
Accuracy
F-Measure
Auc 0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
TCS
Accuracy
F-Measure
AUC 0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
SPIL
Accuracy
F-Measure
AUC
0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
RIL
Accuracy
F-Measure
AUC0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
ITCL
Accuracy
F-Measure
AUC0
0.2
0.4
0.6
0.8
1
TA
N
TA
NW
D
MA
X
MA
XW
D
RE
C
RE
CW
D
HDFC
Accuracy
F-Measure
AUC