Prediction of Pineapple Sweetness from Images Using Convolutional Neural Network

The objective of this research is to propose a deep learning based-prediction model for pineapple sweetness. In this research, we use a Convolutional Neural Network (CNN) to predict sweetness of pineapples from images. The dataset contains 4,860 pineapple images for training. Based on the CNN designed it is found that the best image size is 300 × 300 pixels resized to 30 × 30 pixels. The classification accuracy of training and testing are 72.38% and 78.50%, respectively. In addition, the root mean square error values for training and testing are 0.1362 and 0.1156, respectively. When developed as a mobile application, the accuracy of the application is 80.15%, the root mean square error value is 0.0156 and the reliability is 95.00% Received on 21 June 2020; accepted on 13 July 2020; published on 13 July 2020


Introduction
Pineapples originate from South America, where pineapples are well tolerated for various environments and are considered fruit of Thailand's economy. Pineapple varieties that are grown in Thailand, which consists of many species such as Pattawia varieties, Intharachit varieties (Native varieties), Phuket varieties and Phan Nang Lare varieties, etc. [1]. Pineapples are one type of healthy fruits that are diverse because they are rich in minerals and vitamins. Therefore, most people choose to eat ripe fresh pineapple, preferable the sweetness. However, the main problem is how to select pineapple that are sweet, suitable for eating.
Tapping the sound of pineapple to listen to the density of pineapple is a method that farmers generally use to measure sweetness. However, measuring pineapple taste sensation requires experts that are farmers who have such knowledge to listen and analyse the tapped sound. Experienced farmers are not easily trained and usually not available. The traditional method is a unique ability for individuals whose values depend on listening [2]. In addition, there is a standardized and * Corresponding author. Email: adisak.s@mcru.mail.go.th acceptable sweetness measurement method, which uses a tool called brix-refractometer in which the measured values is in brix-degrees (% Brix). However, this method is must destroy the ripe fresh pineapple to separate the water for measuring the sweetness. Subsequently, the pineapple juice is separated for measuring by the instrument called brix-refractometer, showing the sweetness value (0-90% Brix). Anyway, this method of measurement is not appropriate to be used in everyday life because the tool is expensive and difficult to use.
In the age of digital economy, image processing has become more active in our daily life in a wide variety of disciplines and fields in science and technology. There are many applications such as television, photography, robotics, remote sensing, medical diagnosis, industrial inspection and agriculture [3]. Especially, there are many researchers that use image processing in agriculture to measure fruit sweetness. Khairunniza, et. al. [4] presented a way to determine Chokana mango sweetness using a non-destructive image processing technique. A machine vision system was employed to capture images of mangos in RGB and HSB colour space. Kaur,et. al. [5] presented image analysis technique for automatic characterization of fruits and measuring the sweetness. This research used Artificial Neural Network (ANN) to predict the sweetness of apples and pears. Muresan,et. al. [6] used convolutional neural networks for fruit recognition from images. The researchers presented results of some numerical experiments for training a neural network classifier to detect fruit sweetness. There are also many researches that use different techniques and methods yet they are no perfect in other similar tasks.
In this research, we propose the predictions of ripe fresh pineapple sweetness without destroying the fruit. A convolutional neural networks (CNN) , which is optimised the parameter by swarm intelligence is designed for predicting pineapple sweetness from images . After that, the algorithm was developed to be an application on the Android system to be easy to use and to enhance the quality of life and to promote sales to farmers.

Related Work
In this section we review several previous attempts to use neural networks and deep learning for fruit sweetness prediction.
There are quite a number of researchers to work in the area of fruit sweetness measurement. Sathiendee, et. al. [7] presented colour analysis of banana with spectrum sensor to compare with the sweetness of bananas. The research collected banana samples of about 70 days, which were in the harvesting of 33 samples from 33 bananas, and measure the sweetness of banana with a brix refractometer to analyse the relationship between colour of banana sweetness. The result found that bananas should be harvested for distribution with a colour intensity of 920-940 that would have an average sweetness of 21.47% Brix. In the research, light intensity was used to help determine the relationship of colour of ripe bananas and sweetness. Continued from the above research, Kirasak, et. al. [11] presented a determination of commercial cane sugar (CCS) using near infrared spectroscopy. Determination of CCS is commonly calculated from three factors: total soluble solid (%Brix), polarization (Pol) values and fiber content. This research investigated the use of Fourier transform near-infrared (FT-NIR) spectroscopy techniques for the rapid detection of CCS. CCS predicted equation from sugar-cane juice spectrum showed correlation coefficient (R), root mean square error of calibration (RMSEC) and root mean square error of prediction (RMSEP) of 0.996, 0.610 and 0.573, respectively. In addition, there is some research that uses image processing to measure sweetness. Phanomchai, et. al. [9] developed a system in which users determine the sweetness level of pineapple by using an image of the meat part. The researchers used Euclidean distance for determination sweetness. The precision of developed system was around 83.00%.
In this decade, there are widely used image processing methods with deep learning. Arad, et. al. [10] proposed deep learning for controlled lighting and illumination-independent target detection for real-time cost-efficient applications. The case study used the sweet pepper robotic harvesting. The researchers presented a Flash-No-Flash (FNF) controlled illumination acquisition protocol that frees the system from most ambient illumination effects and facilitates robust target detection while using only modest computational resources and no supervised training. A performance evaluation database was acquired in greenhouse conditions using an eye-in-hand RGB camera mounted on a robotic manipulator. Kaur, et. al. [5] and Rupa, et. al. [8] used the same method of deep learning in their research.

Convolutional neural networks
Convolutional neural networks (CNN) is a special architecture of artificial neural networks, proposed by Yann LeCun in 1988. CNN uses some features of the visual cortex. One of the most popular uses of this architecture is image classification [6].
In a typical CNN architecture, each convolutional layer is followed by a Rectified Linear Unit (ReLU) layer, a Pooling layer, one or more convolutional layer, and finally one or more fully connected layer. A characteristic that sets apart the CNN from a regular neural network is taking into account the structure of images while processing them. Note that a regular neural network converts the input in a one dimensional array which makes the trained classifier less sensitive to positional changes. Among the best results obtained on the MNIST [12] dataset is done by using multi-column deep neural networks. As described in paper [13], they use multiple maps per layer with many layers of non-linear neurons. Even if the complexity of such networks makes them harder to train, by using graphical processors and special code written for them. The structure of the network uses winner-take-all neurons with max pooling that determine the winner neurons.

Swarm Intelligence
Swarm Intelligence is a collective intelligence, a branch of artificial intelligence and a branch of biological inspiration from computer calculations. It is a study and imitation of natural methods by observing living organisms in a multitude of groups, consisting of working groups that are not complicated, and members can communicate within the group and communicate with the environment. However, operations are not controlled centrally, but instead are specific functions such as self organizing. When combined, it will be a big system, all relying on the principles of living organisms, 2 EAI Endorsed Transactions on Context-aware Systems and Applications 05 2020 -09 2020 | Volume 7 | Issue 21 | e4 collaborating, without competition such as ants, bees, per, termite, fish and bird etc [14].
In next section, we will describe each of the layers of a CNN network for prediction image processing.

Methodology
The Convolutional Neural Network Prediction on Pineapple Image for Sweetness Measurement (CNN-PPSM) architecture is proposed and formally defined in this section.

CNN-PPSM Architecture
CNN-PPSM Architecture, shown in Figure 1, consists of two phases. The first phase is an image dataset generation, and the second is feature learning phase. The details are as follows: In the first phase, the dataset generation contains the following steps.
Step 1: Take image pictures of the whole pineapple with the camera from a mobile phone during normal light hours, 6 shots for each pineapple. Cut pineapples into head, middle and tail parts. Then measure the sweetness with a refractometer and record as head, middle and tail classes.
Step 3: For each 700 × 700 image, slide a window with the size 300 × 300 pixels to obtain a total of 9 images.
Step 4: From 300 × 300 image, resize to 30 × 30 pixels, and use to generate training and testing datasets.
Step 5: Adaptive Swarm Intelligence which is used hybrid genetic algorithm (GA) and particle swarm optimization (PSO). We used genetic algorithm to generate the intelligence particle of swarm in PSO. The pseudocode is shown in Algorithm1. The objective function presented in Equ. (3),(4) and (5).
where V is velocity, X is position, c 1 , c 2 , ω, γ 1 and γ 2 are respectively inertia weight, two positive constants and two random parameters within [0, 1].
Equ. (3)and (4) presented equation of MLP of deep learning neural network, where w ij is weight, h is hidden layer and b is biasses.
Equ. (5) presented the accuracy of classification. Step 6 : Using adaptive swarm intelligence (Algorithm 1) to optimize CNN engine which is number of biases and layers.
Step 7 : Develop to application in mobile using Android operating system for sweetness measurement pineapple. 3 EAI Endorsed Transactions on Context-aware Systems and Applications 05 2020 -09 2020 | Volume 7 | Issue 21 | e4 The second phase is in fact the Convolutional Neural Network containing feature learning and prediction parts. The feature learning part is composed of Convolution and Pooling layers. The prediction part contains a flatten layer and a few Dense layers. Equations 1, 2 and Table 1 form the CNN designed.
where filter size (f), stride (s), pad (p), input size (n) andń c as the number of filters, which are detecting different features.
The designed CNN for predicting pineapple sweetness from images is shown in Table 3.

Images Dataset
This research goal was to design a deep learning neural network for predicting pineapple sweetness from images. The collection of pineapple images was necessary and the images of pineapples from every season were required because the colour of the pineapple shell and the value of sweetness are different from season to season. Thus we collected 30 pineapples per season and each pineapple were shot pictures to produces 6 images as shown in Figure 2. The total photographs of original pineapples were 540 images. By a sliding windows method each image generated 9 different images yielding 4,860 images, in which 3,780 images were used for training and 1,080 images were used for testing.
In addition, a refractometer( % Brix) was used to measure the sweetness of pineapple. The sweetness were classifier into three parts of pineapples: head, middle, and tail indicating three levels of sweetness.  The images and their responding sweetness values form data mining data and to be used in the data mining process as shown in Figures 2 and 3.
From the figure 4, take the pineapple images to find the suitable image size for use in learning. In this research, 300x300 pixel images and reduced to 30x30 pixel is the most suitable size for learning. The images used are RGB color images and use images that are basic color systems based on the human eye (HSB). The images of pineapples are divided into 3 parts: the header, the middle and the tail of the pineapples. Therefore, each pineapple will get 54 divided and scaled images. Then, all 4860 images were used for learning. After that, all images were grouped by sweetness. This research uses the sweetness values to group it into 4 groups as follows: Sweetness 5-7 4 EAI Endorsed Transactions on Context-aware Systems and Applications 05 2020 -09 2020 | Volume 7 | Issue 21 | e4

Experimental Result
This section presents the performance of CNN in predicting sweetness value of pineapple images. Accuracy (ACC) and mean squire error (MSE) are used as performance metrics. Since convolutional neural network (CNN) is neural one type of network that is often used in computer vision or image analysis such as image classification, object detection, face recognition, etc. CNN is a multi-layer neural network that has a unique structure designed to add the ability to extract more complex features from data. Therefore, the limitations of CNN are the difficulty in determining CNN's architecture to be fit that can extract features from data that is less organized or does not have a unique structure (unstructured data) such as specifying the number of convolution layers that extract important features from images, filter types, channel number, kernel size, stride, padding, etc. Therefore, finding the most suitable parameter in CNN's architecture will have a big impact on the overall model architecture's performance. It is one of the most important hyper parameters that will give the most satisfying results. Therefore, the results of this experiment will show the use of the developed algorithm to adjust the parameters of the CNN architecture to get the best output which can predict the sweetness of pineapples are as close as possible to the measured values ??from a refractometer.
The results are presented in the percentage of prediction accuracy both for training and test data. A better trained model will result in lower MSE and higher accuracy with the original images. Figure 5 shows variation of MSE and ACC for 500 epochs.  From the table 2, it shows that the appropriate size for learning is 300 × 300 pixel Resize to 30 × 30 pixel, the learning accuracy is 72.38%, the square root of the mean square error is 0.1362, and the accuracy of the test is 78.50%, the square root of the mean square error is 0.1156, which is a good value at a satisfactory level. But the problem with this experiment can be found whether the time-consuming operation and the CNN configuration parameters are appropriate. To solve this problem, the researchers developed the hybrid heuristic algorithms between genetic algorithm and particle swarm optimization algorithm to be used to optimize parameters with CNN. The algorithm used for image data that is suitable for learning is 300 × 300 pixel Resize to 30 × 30 pixel. All images in practice and test are 4,860 images. The less number of working times is 300 epoch, epoch 100 times, and the number of parameters of CNN is smaller as shown in table 3. The experimental results obtained from the adjustment of the parameter of CNN from Table 3 showed that the learning accuracy was 78.65%, the square root of the mean square error of 0.1350, and the accuracy of tested at 80.15% square root value of the mean square error value at 0.0156, respectively. 5 EAI Endorsed Transactions on Context-aware Systems and Applications 05 2020 -09 2020 | Volume 7 | Issue 21 | e4 After that, the researchers used the developed algorithm and the CNN with the appropriate parameter adjustment to develop the application to measure the sweetness of pineapples via mobile phone on the Android operating system as shown in figure 6 and 7. For efficient processing, database storage space, applications can be shared everywhere, without additional computer investment. In addition, operations that use image processing require fast processing and large storage space. Moreover, it must be easily accessible through the internet process. Therefore, the researcher has applied the developed application and algorithms to the cloud computing system, which results in the usage being able to be used wherever the internet access is as shown in figure 6 and 7.

Conclusion
From the research, it is found that the pineapple sweetness determination without damaging the fruit can be done by using a photograph of pineapples from the external appearance, the colours of the fruit, the size of the pineapple's eye, the tension of the pineapple skin, the glossy colours of the fruit. If considered superficial, the colour of the pineapple fruit may not be related to the sweetness, such as the colour of green fruit, but sweet or the colour of citrus but sour. In fact, the consideration of the sweetness of pineapples must take into account many variables. Therefore, using computer with appropriate methods to help train and test by combining from many variables together can predict the sweetness at a satisfactory level (measured by the value of accuracy and root mean square error from the experiment results). This research develops the suitable algorithm to find the suitable value of image processing with CNN to get the best results. After obtaining the suitable method for predicting the sweetness, it was developed to be a new innovation to measure the sweetness of fresh pineapple without damaging the result by using the mobile phone photography on the Android operating system. This innovation makes the measurement of the sweetness of pineapples convenient and easy to use for both producers and consumers. The results obtained from this innovation measure an accuracy of 80.15%, square root value of the mean square error value of 0.0156% and reliability of 95%, respectively. The main limitation of this research is that pineapples used in the experiment are mostly green pineapples, orange green pineapple that has a small amount of orange. Therefore, the learning of pineapples with orange is minimal. The results obtained from the prediction of the pineapples with orange have a relatively high error value. From the results of this research, it can get the idea of developing new innovations in the future such as the development of the predictive innovation in pineapple mealybug wilt by using the application on the Android operating system etc. The above development will enable farmers to use the appropriate technology for further development as a smart farmer. 6 EAI Endorsed Transactions on Context-aware Systems and Applications 05 2020 -09 2020 | Volume 7 | Issue 21 | e4