Acu Glass: Quantifying Acupuncture Therapy using Google Glass

Acupuncture is a therapeutic or preventative therapy through the stimulation on acupoints that located on the body surface across the anatomy to affect the functions of speciﬁc organs or soft tissues. During the acupuncture training process, mapping the acupuncture map to patients could be a challenging work since the trainees mainly rely on the instruction from experienced acupuncturists. In this study, a wearable acupuncture assistant system using augmented reality technology was introduced to record the whole treatment process with augmented information that quantiﬁed the traditional acupuncture practice. Through constructing a dynamic, personalized, and self-regulating map of virtual acupoints, the system quantiﬁed the coordinates of the acupoints to form a basic guide-line to help the trainee to recall the type and location of acupuncture map. The experimental results show that the proposed system could record the coordinate of acupoints in real time and the augmented information also has the potential to be a visual guide during treatment.


INTRODUCTION
Acupuncture is an alternative therapies practiced by inserting a fine needle at certain acupoints on the body for therapeutic purposes.Acupoints located on the body surface across the anatomy affect the functions of specific organs or soft tissues.Scientific evidence supports that acupuncture treatment can stimulate neurons under the skin and in muscle tissue for pain relieve [11,7,1,2].The training and treatment processes of acupuncture practice are still primarily and rely on instruction from the experienced acupuncturists to locate the acupoints from body acupuncture maps.The experienced-based methods is a large burden due to limited number of experienced acupuncturists and is difficult to fulfill the requirement for long-term training.An overlaid virtual acupuncture map on the acupuncturists' view of the patient's skin would improve the efficiency of their work and also the accuracy of localizing the acupoints.Augmented Reality (AR) is capable of providing the new experience by superimposing virtual information onto the real world objects.A portable hands-free Head-Mounted Displays (HMD) device that controlled by voice or hand gestures could free user's hand for acupuncture treatment.Google Glass [8] is a head-mounted wearable computer that uses a field sequential color to project rendered images through a prism and onto the retina, and could be controlled through user's voice or hand gestures.The users perceive a small translucent screen hovering at about an armslength away, extending upward and outward from the right eye.The proposed wearable acupuncture assistant system located the coordinates of the acupoints and constructed a virtual acupuncture map according to the relative position between the user and the patient.The user performed training treatment based on viewing the superimposed acupuncture map on the patient's face through the Google Glass.In addition to the hands-free feature, the system recorded the whole treatment process with augmented information that quantified the traditional acupuncture practice, and had the potential to be a visual guide during a treatment.

RELATED WORK
Head-Mounted Displays have a wide range of applications in medical and health areas.Zhang et al. introduced a personalized UV monitoring and notification system using AR glasses to display augmented sunburn effects on user's body parts [13].Yang et al. [6] reported a real-time acupuncture manipulation procession system using pattern identification of an autoregressive parameter model to identify various kinds of acupuncture manipulation.Various tracking algorithms have been developed for AR such as Scale-invariant feature transform (SIFT), Speeded Up Robust Features (SURF) [5], and Oriented FAST and Rotated BRIEF (ORB) [12] etc. Tracking features from frame-to-frame system gave stability to AR and the virtual object could be registered on the correct position.A study indicated that Binary Robust Independent Elementary Features (BRIEF) [3] is an efficient feature point descriptor, which only requires 256 bits or even 128 bits to obtain good matching results.The proposed work was to design an AR-based, hands-free acupuncture assistant system including face detection, image tracking and AR technology with a quantified evaluation.To the best of our knowledge, no pre-existing AR-based acupuncture training system using HMD was reported.

SYSTEM DESIGN
The acupuncture assistant system was designed for acupuncture training and treatment in the hospital with quantified practice approach to assist the trainee to recall the location and type of the acupoints.The acupuncture assistant system included three main components: (1) Coordinate Quantification and Map Generator (2) Tracking and Rendering (3) Visualization and Training.The Acu System flow is shown in Figure 1.The system acquired the coordinates of acupuncture points in the Cartesian coordinate system by using their relative positions on the base face.Then, the virtual acupoints were constructed as spheres and the map generator built a graph-based 3D face acupuncture point map.After getting the virtual map, rendering system put the virtual map on the real human's face over the consecutive camera frame by using the feedback from tracking system.The tracking and sensing system obtained an accurate overlay accordingly to improve the stability of performance in our AR system.Finally, the augmented reality system combined the human face in the real world with the virtual acupuncture map and displayed an enhanced view to the user through the projected screen of the Google Glass.Users could directly locate the specific acupoint according to the virtual acupoints in the AR view.If no face was detected, the system simply passed the original camera view back to the user.

Coordinate Quantification and Graph Generation
Considering a simple wire frame map with a list of nodes and edges, the proposed system visualized the spatial relationships between acupoints in a directed graph structure in which nodes represented virtual acupoints and edges represented spatial relationships between the pair of nodes (p, q).Assuming that we had N different nodes on our face, each pair of nodes had a geometric relationship that could be expressed using the standard computer graphics notation of a 4 × 4 homogeneous matrix.For constructing an initial virtual acupoint map, a frontal human face with standard acupuncture point map chart learned from acupuncture book was used as baseline.The geometrical relationship between each pair of (p, q) ∈ N was scaled according to one of the specific volunteer's face, which was set as the base face.In the proposed prototype system, a simple map was preferred, and therefore, all values were expressed as a ratio to the bounding box that was returned by the face detection algorithm.For each volunteer's frontal face, the proposed system generated different referenced virtual map.For each virtual map, geometrical widths were scaled based on the distance between the centers of the user's eyes relative to the geometrical heights of the base face by calculating the height of the whole face.The scaling process could be represented by the equation ( 1).
where (xo, yo) is the origin of the coordinate system, (x, y) and (x , y ) are the coordinate for each node in the image plane of base face and new face, respectively.W and H are the eye spacing of the new face and height of the whole new face, respectively.W and H are the corresponding value of base face.In response to the different volunteer, different referenced virtual map was used for image tracking to keep the AR system performance to be stabilized.

Tracking and Rendering
The Oriented Fast and Rotated Brief (ORB) keypoint detector and descriptor [12] were selected for the tracking system to search for the distinguishable features of the human face.ORB extended FAST [10] by efficiently computing orientations based on the intensity centroid moment and steered BRIEF according to the orientation of keypoints.Therefore, ORB was faster than SIFT or SURF and also rotation-invariant and scale-invariant.In order to find the latest relative positions relationship of corner features, the detected corner features were matched in the newly captured frame to the features in reference frame and the transformation matrix between current frame and referenced frame calculated.When the keypoints and the descriptors matched closely enough, a homography matrix could be estimated.The system crossed specific pixel to represent the virtual acupoint.With tracking, the rendering system would receive the result of a series of coordinate transformations from the tracking system and modified the relative positions of the acupoints automatically to compensate the change of view of the user.Therefore, the stability of AR system would be improved and the position bias would be reduced significantly.

Face Detection and Map Scaling
First, the face detection was implemented using OpenCV.The subsequent processing could be reduced to the region of interest (ROI) of the face.Before detecting both eyes, the ROI of the face could be further reduced to the approximate ROI of eyes.Then, the same OpenCV detection algorithm was used once again to detect both eyes.Note that another special haar cascade was used in this case.In this way, the distance between the two eyes was used to scale the acupoint map in the x-direction, and the height of the whole face was used to scale the acupoints map in y-direction.

Acupuncture Training Guide
Acupuncture Training Guide was about the interaction between the augmented information and the user.The virtual acupuncture map scaled according to the different face sizes, and the user could control the zoom level in and out via swiping the Glass's touchpad right and left.The user was able to select different labeling modes of the acupuncture assistant system for different purposes including color coded training or specific treatment sequence.The figure 2 shows training mode where the red rectangle representing ROI of face, green rectangles representing ROI of eyes, and the red line between eyes representing the eye spacing.

EXPERIMENTS AND RESULTS 4.1 Objective
To evaluate the usability of the system, two corresponding experiments with human subjects were conducted.The first experiment was to test the effectiveness of the tracking system for adjusting the geometrical relationship for each pair of virtual acupoints even on the different view angle of face.The second experiment was to validate if the rendering system could adjust and render the virtual acupoints map on different faces.

Experimental Setting
Six volunteers were selected for the two different experiments.The study included subjects who were more than 18 years old with normal facial features and were also willing to participate in acupoint detection by using the E-ACUPEN [4].To verify the reasonably accuracy of the position for each virtual acupoint, E-ACUPEN was used for detecting the actual acupoints.E-ACUPEN located the acupoints through measuring the electrical resistance of the skin and indicated the major acupoints by modulating the beeping sound frequency [4,9].The first experiment was to verify the effectiveness of the tracking algorithm of the system.The lighting conditions were kept unchanged because this experiment was designed to verify the basic effectiveness of face tracking.As shown in figure 3, experiment 1 was divided into two parts.First, in order to establish a basis for comparison, the position of the virtual points on the view of frontal face was used as the basic reference, and the face tracking module was temporarily removed from the proposed system while all of the other modules were retained.Then, several black plus-sign-shaped markers were drawn on the volunteer's face to save the position of the virtual acupoints.After the virtual acupoint map was overlaid on the view of the volunteer's face and the positions of the points were saved.The angle between the volunteer and Google Glass was adjusted while still kept a comfortable and roughly-fixed distance.Assuming that the initial angle (x, y, z) = (0 • , 90 • , 90 • ), the volunteer moved his head to the angle of approximately (−30 • , 120 • , 90 • ).For the new angle, the pixel unit distance d between plus-sign-shaped markers and virtual acupoints were processed and recorded by using Matlab.The same process was conducted again with face tracking module added back in.Let d1 and d2 represented the average bias between plus-signshaped markers and virtual acupoints under the situation of without and with tracking, respectively.If d2 was much less than d1 in each trial, the position bias was significantly decreased and tracking algorithm is useful and effective.Table 1 summarized the average position bias for the same volunteer over 6 trials under the situation of without tracking and with tracking.The minimum value of d1 − d2 was 8.86 pixel units in the third time experiments.When position bias was more than 3 pixel unit, the jittering was obvious.In the tracking experiment, considering the minimum reduction of position bias was 8.86, which was much more than 3. Therefore, the tracking algorithm played an important role in maintaining the stability of AR and the proposed system had robust tracking result to adjust the geometrical relationship for each pair of virtual acupoints on different view angles of face.The second experiment was to test the accuracy for each virtual acupoint and its corresponding point was verified by E-ACUPEN.
In conducting the experiment, the E-ACUPEN was used to find the actual acupoints and the precise positions were marked using black cross marker.Google Glass position was fixed to face directly towards the volunteer, which indicated that the angle (x, y, z) = (0 • , 90 • , 90 • ).After the virtual map overlaid on the volunteer's face, the average pixel bias between each black cross marker and virtual acupoint was calculated, and the average error was represented by equation (5).
where xv is the virtual point's x-coordinate, and xa the actual point's x-coordinate.
Figure 4 shows the distribution of virtual and actual acupoints on a randomly selected volunteer's face.

DISCUSSION AND FUTURE WORK
The proposed wearable acupuncture assistant system generated a virtual acupuncture map and superimposed it on patient's face on the Google glass user's view, which provided a great help on learning the relative acupoints' positions.Moreover, since the Google Glass is a head mounted AR device, it also frees the trainee's hand for practicing the acupuncture skill.The localization results obtained by the ACUPEN were served as the ground truth for verifying the accuracy for each single virtual acupoint one-by-one.The results given in this paper, although preliminary, supports that the proposed system is a feasible study.
Although slight latency occurred due to the limited calculation power from google glass, it could be improved by offloading the calcula-tion intense task such as image processing to Android phone.The next stage of the work is to set up a client-server communication system between Google Glass and an Android phone in real time to allow multiple processors working in parallel.The whole system benefitted by letting the Android devices doing the calculation work, then the proposed work will be provided for testing by acupuncturists.

CONCLUSION
In this paper, we proposed an AR-based acupuncture assistant system running on Google Glass.Two experiments were carried out to verify the robustness of tracking and the rendering of the system.The experimental results showed that the AR-based system was capable of quantifying the traditional acupuncture practice and it could be applied on different objects while still preserving reasonably stability.

Figure 2 :
Figure 2: Virtual acupoint map on the view of frontal face in training mode.

Figure 3 :
Figure 3: The comparison between non-tracking and tracking.

Figure 4 :
Figure 4: Position distribution from one of the volunteer

Table 1 :
The comparison of average position bias d in the tracking experiment

Table 2 :
Table 2 summarizes the average accuracy bias for three volunteers.It is more intuitive to see the acupuncture map on faces and the proposed system has reasonably rendering virtual acupoints accuracy.The average accuracy bias verified by E-ACUPEN