Analysis, design and development of a framework focused on augmented reality applications: Medical rounds case

Augmented reality is a technology that opens doors to experience interactions with virtual and real-world elements. It provides an opportunity to acquire knowledge and practical skills through software. However, it is not an easy task to determine the qualities and features that an augmented reality framework must possess to lay the foundations to build a structure focused on augmented reality. Therefore, in this work, we propose a framework focused on augmented reality applications without a specific domain area. We conduct a review of different frameworks for augmented reality. In order to define the characteristics, components, and the overall structure of our framework, a review of the most relevant literature in the field was carried out. As a result, we designed a framework focused on augmented reality without a specific domain area. Finally, we conclude that the structures, components, and features for an augmented reality framework vary greatly depending on the requirements or characteristics used for each approach. Received on 04 February 2021; accepted on 22 February 2021; published on 26 February 2021


Introduction
Today, augmented reality (AR) is used in many areas of human interaction, such as medicine, industry, modeling, design, the Internet of Things, online learning, and entertainment, among others [1] [2] [3]. It is often associated with virtual reality (VR). However, they are two different terms. AR is considered to be the insertion of virtual information, such as images, text, video, audio, or 3D objects, into a real-time view of the real world [2] [4] [5]. One of the main benefits of AR is the generation of experiences through the visualization of a task, process, or activity. This generates relevant knowledge for the AR user [2]. Therefore, AR applications must be focused on recreating a pleasant visual interaction without being saturated with information [1]. Applications developed for AR devices vary in many ways, such as the programming language, structure, functionality, features, implementations, etc. Also, the processing and rendering abilities of the device are an important factor [2]. Therefore, it is a complicated task to define a model, architecture, or framework. Experience is an important factor in determining how to develop an effective AR application. AR applications must be intuitive, with interfaces that offer a useful AR experience to the user [6]. In this work, we focus on designing and developing a framework for AR applications, employing a review of the different frameworks, architectures, and models proposed in the literature. The framework is not directed to any specific domain area, but has a general focus. Then, with a specific domain area in mind, we instantiate the framework and develop all its components, focusing on that domain of application. This work is organized as follows. Section 2 describes related work on frameworks for AR. Section 3 describes the proposed AR framework. Section 4 describes an instance of the framework, focused on a specific domain area, that of the data and information needed in clinics and hospitals to conduct medical rounds. Section 5 presents some conclusions and outlines future work.

Related literature
To define the components for an augmented reality (AR) framework, we carried out a review of the different frameworks for AR. The frameworks reviewed differ greatly. Each framework was designed taking into consideration the needs of its project or the features of the specific problem to be solved. The purposes and technologies implemented also differ. Some of the papers describe architectural solutions and others include AR applications in their solutions.
In [7], it is proposed an architectural design as a solution for the problem of collaboration during maintenance, i.e., the aim is the improvement of onsite collaboration regarding maintenance. They argue that collaboration is an indispensable task and use AR applications only to archive this objective. They consider the context-awareness of an AR application as an important feature for discovering the details of the machine or equipment being maintained.
There are three main research areas in AR [8]. The first research area is context-awareness. It is described in different research papers [8] [9] [10] as those software techniques and methods that use information about the environment to "characterize augmented content" as messages, instructions, or alerts to the AR users. The second area is "Authoring", described as software techniques aiming to create augmented content through the information and data supplied by an expert or trained staff [8] [11] [12]. The last area is "Interaction-Analysis", described as software techniques and methods that analyze the status between the user and augmented content interactions with the purpose of providing relevant feedback to ultimately improve the interactions or make necessary changes in the AR content [8] [13] [14].
Other approaches, as in [12], describe a framework and a prototype to support industrial maintenance operations in real-time. As in [7], [12] use contextawareness as a feature of their AR applications, but they propose a solution based more on the AR applications. The proposed framework is segmented into three different parts, two platforms, and a database file. The first platform is based on the AR research area "Authoring" described in [8] as software techniques and methods whose objective is to create augmented content to display it through an AR application in the real environment. The authoring platform consists of a desktop interface to interact with the framework information to generate AR content. The second platform is the AR application used during the maintenance, which automatically generates AR content based on maintenance experts' inputs. Finally, the database file is used to communicate between both platforms.
The structure of the framework depends largely on the environment or application area. Some, e.g. [15] [16] and [17], have determined that AR applications can be decisive to motivate the interest of users. [15] propose a framework focused on exploiting the motivational impact generated by AR. They designed a base needed to create motivational AR applications, mainly focusing on improving student learning in the area of vocational education. However, it is not limited only to the training of educators. [17] also focus on AR applications for vocational education but unlike [15], focus mainly on the application itself and specific components for that purpose. [15] describe the modules that an AR application must have to efficiently support student motivation. It consists of three main parts: support applications, mainly used to manage accounts, learning analytics, and general management. The AR applications, in this case, are designed for a mobile or desktop with four different layers: user interfaces (interaction layer), activities (experience layer), student support layer, and assessment layer. Finally, Input, Sensing, and Registration, which completely depend on the AR device used.
Other works have described frameworks and architectural designs of solutions, e.g. [18] and [19]. However, there is not enough information about their solutions and implementations for comparison or use to guide a new solution. Also, in [20] [21] [22] AR is only taken as a small component of the framework and is not focused on.

AR Framework
We designed this framework based mainly on the components, capabilities and functions in [12] and [15]. In [12], a framework is described with three main components: Authoring platform, Database file, and Application platform. The first component is used by experts in charge of supplying information and data. The second component is the place where the first and third components store and retrieve information. The third component is used by staff users that need the expert's information to complete their tasks. We proposed a framework based on the structure of this component where there is a system for experts, a support component, a system to store and retrieve data, a storage component, and a system to use the data, the Authoring component. Also, we added another component to structure and organize the information to be used, the AR system. In [15], a framework for designing motivational AR learning applications is described. They propose a set of components needed for an application in AR. We base the AR component in our framework on the flow of the internal components in the AR Application in [15]. However, instead of using four layers, we only focus on the following three: Authentication, Profile Selection, and Data View. All the individual components are explained in detail in the following subsections.
We proposed the framework to guide the design and development of a whole environment to get, share, and view the provided information. Our aim is to create instances based on this framework while following its structure. It is not designed for a specific area or information type; its purpose is to be used in multiple areas. However, it can be focused on a specific area depending on the information with which it is supplied. We present an overview of the AR framework in Figure  1.
The framework promotes the design and development of four main components. The four main components are the framework's core; they are composed of different systems. Each system can only be part of one main component. We will go into detail with each of the four main components and the systems that compose them.

Storage Component
The storage component is the base component of the AR framework. All the data used and managed by the other three components' systems are stored here. Each component system will connect to it to get, modify, or store data. It is the only system in the entire framework that does not have other modules or subsystems. This way, any kind of technology and tools can be used, depending on the information and data provided.

Support Component
The support component system can be developed as a mobile application, desktop application, or web application. It is used by an administrator (expert or an authorized staff) to manage data and information that can perform some of the following tasks: user management, resource management, file management, profile views, reports, etc.
Its purpose is to be the point of management of the user data of the support, authoring, and AR systems, besides the information from the application area. It connects directly to the storage system and allows adding, modifying, and deleting information and data, with the appropriate permissions. All the user accounts for the authoring and AR components systems must be created by this system.

Expert
System. The expert system has a graphic user interface used to interact with the administration modules for users, resources, profiles, etc. Also, it is used to add anything needed as part of the data from the application area. It is possible to add more modules to fulfill any necessity or to manage specific data supplied to the framework. However, management modules are essential since through them we give a user access to specific data and components' systems in the framework.

Authoring Component
We designed the authoring component based on the characteristics and qualities that an application must have for creating content for AR [8] [11] [12]. We provide an example of the authoring system in Figure 1 (authoring component). Augmented content will depend on the data supplied to the framework. We present one of the more basic ways to generate augmented content through information profiles composed of the data in the storage system; the details will be provided in section 4.3.

Creation system.
The expert or trained staff, preferably accustomed to handling domain data, design the "layouts" used to create profiles in the creation system. Data can be textual, such as manuals and documentation, or multimedia files, such as videos, images, 3d models, etc. The "profiles", formed by layouts, are transformed into content used for the AR system in the framework AR component. The authoring system expert or user does not need to have experience in creating software as such. It will only be necessary for them to provide the information they want to be added to the AR system.

AR Component
This is the component system running on the AR device. It consists of a user interface for users that need to see the profiles created by the authoring component system using the previously supplied data from the storage component. Through this interface, the user must authenticate himself, select the desired profile or data, and then the display of that profile will be shown. Profiles can only be created and modified through the authoring system. Users in this system do not need to have experience in creating software as such either.
3.4.1. AR system (device system). This is the system running on the AR device. It is composed of three different subsystems or modules. In an AR application, the interaction experience, normally called "AR experience", is the main and most important feature [8] [10] [13] [14]. Through it, the user will experience the advantages of AR. The user interacts with the internal modules through the interface. In Figure 2 we show the composition and internal modules of the full AR system.
• Authentication: The authentication module is the first interaction between the user and the AR system. Through an interface, the user must enter their access credentials previously managed in the support system. Authentication credentials may be entered traditionally (username and password) or through voice recognition.
• Profile Selection: The "profile" selection module or the module to select the structure of the data defined in the authoring system. It mainly depends on the type of information or data supplied to the storage system. For example, consider maintenance tasks in the workshop of a car mechanic. The user could select profiles for cars or car parts. The profiles can be designed for maintenance or repair. On the other hand, in a hospital, medical personnel could select a patient, a room with patients, a medicine, information from other medical personnel, etc. It is suggested that the data selection be carried out through the input of information, recognition of barcodes or QR codes, or voice commands.
• Data View: The data view module takes all the data from the storage system and re-structures it according to what has been specified in the profile from Authoring. The interaction experience must be as pleasant as possible. The user's vision should not be overloaded with AR content. An important suggestion is to delimit one area in which texttype information is presented, and another for videos, images, etc.

AR Framework Instance
We developed a prototypical implementation of the four main components' systems in order to validate the capabilities of the presented framework. As the domain area, we used the data and information needed in clinics and hospitals to conduct medical rounds. "Medical rounds" are defined as the visits to hospital patients made by doctors, nurses or students with the purpose of assessing the state of health of those admitted, knowing their daily evolution, and making the necessary indications for their recovery and rehabilitation. The data used consists of test information simulating patients and their medical records. We describe the systems of each component of the framework below.

Storage System
In the storage component framework, there are no subsystems or components specified for its creation or implementation. It is the only framework's system with this kind of feature where the user can choose which tools, hardware, and software to use. In this work, we carry out two types of implementations of the storage system. In 4.1.1 subsection "Local" implementation and 4.1.2 subsection Amazon Cloud AWS implementation.

Local Implementation.
For the local implementation of the storage system, we decided to use a virtual machine instantiated on a server at LaNTI (National Laboratory of Information Technologies

Amazon AWS Implementation.
We also made an implementation of a storage system in a cloud environment, on the Amazon Web Services (AWS) cloud [24]. It was done this way since a local implementation of the system would not be accessible outside the local network where it is located. Without network administration permissions, it was not possible to allow access through an external network, such as the Internet. Therefore, it was necessary to carry out an implementation that allowed access from any network connected to the Internet that was using the AR device. We chose Amazon over other cloud providers because its Elastic Compute Cloud (EC2) service offers what is necessary for the storage system. Amazon is one of the companies that has been providing cloud services for the longest time; they are available over a wide range of regions and zones, and they have a wide repertoire of instances to cover almost any need.
We decided to use a free instance "Ubuntu Server 18.04 LTS" in 64 bits. Putty client was used to connect to the instance by ssh. It is important to know that the IP address in our instance is not static and Amazon may withdraw or change it without prior notice. To avoid this, we must request a static IP address. Finally, we performed the installation of the LAMP stack software and tools as in the local implementation.

Database
System. This is the database in the storage system which feeds the other systems of the framework. It uses version 10.4.11 of the MariaDB engine. The stored data in the database is the test data based on a fraction of the information a hospital uses from its medical staff and patients. The main purpose of this data is to be consumed by the other systems and define its functionality and behavior. In addition, it is used to maintain control of the users and roles. Such control is very important since, depending on the data managed, we can grant access, read, and write 5 Analysis, design and development of a framework focused on augmented reality applications: Medical rounds case EAI Endorsed Transactions on Creative Technologies 12 2020 -03 2021 | Volume 8 | Issue 26 | e3 permissions to users. The data consists of users and roles (in this case the medical staff), general patient information, physical examinations, medical histories, patient tests, radiographs, and profiles generated by the authoring system.

Support System
We designed the support system as a web application to be accessed through the Internet using a computer or mobile device. The Laravel framework is used to speed up the system development process. Laravel provides tools and structures necessary to form the foundation of the system. We implemented the system on an EC2 instance of AWS just like the storage system. This way, the application can be accessed from any computer or mobile device connected to the Internet. However, this does not mean that this is the only way to implement this system. If the user wants to implement the systems so as to be accessible only through a private network, it is possible to do it that way.
The support system allows access to different types of users with different permissions and privileges. Users of the system include Administrators, Doctors, Nurses, and Students. The Administrator is in charge of tasks like creating new users to manage the system. They also have all the privileges from the users below the administrator level. Doctors are the main "Experts" for this test instance: they can create and modify users with roles below them, such as nurses and students. They can also create and modify personal patient information, such as general information, physical examinations, history records, tests, and X-rays. In addition, they can remove patients from the system. Nurses have the permissions to create, modify, and delete patient information, as do doctors; they are the main responsible staff for monitoring, reviewing, and updating patient information. However, they can only create one type of user, students. The last type of user is student. Students do not have permission to create any type of system user. They cannot register new patients in the system or delete a patient. However, they do have permission to update patient information that was previously discharged by a doctor or nurse. In Figure 3 we show the main dashboard page for an administrative user in the ARSupport App.

Authoring System
We designed the authoring system to be a web application, as was the support system. The Laravel framework was used to set up the basis of the application. It was implemented in an EC2 instance in the AWS cloud. This system and the support system can be accessed through any device with a browser and Internet connection.
The application allows access to all manageable users of the support application. However, they cannot perform administrative tasks such as creating, modifying, or deleting any type of user. It is purely the means to generate and manage profiles that will be used by the AR System. The main purpose of the application is to allow users access to the AR system to create and manage profiles specifically designed for the use of the AR system. Once in the system, the user can consult, modify, and delete the profiles they have created. They can create new profiles without limit. There is a section in which the user can view other profiles created by other users of the system. We show in Figure 4 the profile creation page for an administrative user in the ARAuthoring App.

AR System
We developed the AR system for Microsoft's Hololens AR device. We used the video game engine Unity, version 2018.4.6f1, to form the elements and virtual scenarios, part of the interface and environment application. Additionally, we coded events, behaviors, and main performance in the C-Sharp programming language using the visual Studio 2019 tool. To manage the Hololens device, we used the Microsoft framework "MixedREalityToolKit" (MRTK) version "Foundation-v2.0.0-RC2".

Authentication.
Currently, it is not possible to destroy a scene that contains the gameObjects package of the MRTK framework. Therefore, we opted to leave the main scene and load scenes additively, while removing those scenes that have already fulfilled their objectives.
Once the AR application has started on the device, the main scene calls the scene to authenticate the user through a blank gameObject used only as a container for the calling script. The authentication scene is made up of a traditional login interface composed of gameObjects in which the user must enter credentials, as shown in Figure 5. Once the user has entered the credentials and sent them for review, the interface calls the verification script, which is connected to the storage system in which the identity of the user is verified. In the storage system, there are PHP scripts that look for the user's information in the database. If any match is found, it will return a true value, and, otherwise, a false value. The script displays an error credential message to the user in case of a false response or loads the next scene in case of a true response.

Profile Selection and Data View.
In this example, which is an instance for a medical round application, the user needs to select a patient to start. Therefore, profile selection and data view come together in a joint module because you must select a patient  first to load a profile. The new scene that has been loaded is the "Around Scene" since this scene must follow the medical staff in charge of carrying out the medical round. The interface of this scene is initially a gameObject with a message for the user, as shown in Figure 6. It can start in two different ways: one is to write or scan the QR code of a patient room. If the patient room is occupied by more than one patient, a list of patients is displayed, from which the patient to be reviewed is to be selected.
Selecting a patient from the list will load a new interface that shows a viewing environment for this patient's information, as shown in Figure 6. This new interface is composed of a little info tag with the most basic information and a photo of the selected patient. In the main interface, there would be a selection box where the user can attach the tag to the corner so that it does not hinder the field of vision. There's also a left panel with different buttons to show patient information, patient tests, X-rays, physical explorations, and medical history. On the right side, there's the information viewer frame where information is shown in four different windows. The user can choose between two options:  • Data Selection: The user can select the individual information that he wants to view in the middle frame in Figure 7. This separate information will be shown in the four different subframes. In each frame, the user can select a different window with the selected information from the selected patient.
• Profile Selection: The user can select a stored profile created in the Authoring App. This profile will show the different information windows from the patient information.
All the information shown in the different windows is extracted from the storage system through different scripts that search for the patient information and return that information to the AR system. There is a script in the storage system for every type of information in the directories "ExplorationDataRequest", "XRayDataRequest", "HistoryDataRequest", "TestsDataRequest", "PatientDataRequest" in addition to the authentication script.
In the left panel is the profiles section. If the user has created profiles through the authoring application, they will be shown in this part. Figure 8 shows a profile previously defined in the authoring application in this user's account.

Conclusions and future work
In this work, we presented a review of different frameworks for the use of augmented reality (AR) applications as the focus. We found that the structures, components, and features vary greatly depending on the requirements, focus, or characteristics that are to be used for each approach. All had the main objective of developing a framework for AR applications, one that could be directed towards any domain area where data and information need to be shown to users through AR devices, so as to quickly provide the user with information that was easy to view. As a result, the proposed framework consisted of a framework with four different components: storage, support, authoring, and AR. They work together to enable the user to manage the data, create "profiles" for the AR system, and display the data to the user.
Once the framework was finished, we made an instance to test the framework. The instance was focused on the domain area "Mexico Hospitals". The AR system focused on viewing patient data and information during medical rounds by the hospital staff. The instance consisted of the design and development of 4 different systems. The storage system is responsible for storing user data (medical personnel), patients and their medical information (history, physical examination, tests, radiographs) and the profiles for the AR system. The support system was in charge of managing users and domain information that will be used by the other systems. The authoring system was in charge of managing and viewing the profiles for the AR system. The AR system was responsible for displaying patient data and information to authorized users.
In conclusion, segmenting the framework into four main components gives us robustness. Since the possible domain areas are extensive and varied, the four components allow us to focus on certain activities and tasks on each of them. The storage component does not restrict or limit the tools, software, or implementation that can be used. We showed two of these implementations, locally and in a public cloud environment. The support component forms the roles and permissions that users possess and grant to other users, restricting access to information to unauthorized personnel and allowing only users with certain roles and permissions to access or modify information. The authoring component provides an ideal environment for creating and managing profiles that will display information in an order or sequence in the AR system. However, depending on the information that must be displayed, this component may require additional features or behavior. For this work, a simple design of data, such as text, images, videos, or multimedia files, has been proposed. The AR component is where the user will view the information and data of the domain area. The framework does not specify an AR device to be used. But the modules that it should contain are proposed using the data of the storage component managed by the other systems of the framework.
AR has been classified as an innovative technology with great potential for almost any area of human interaction. Its potential for intuitively addresses the users' needs should be explored and worked on. In the future, this framework could be improved so the AR component can be used in devices mainly or exclusively designed for AR. It may also be possible to reform the framework to cover even more domain areas with specific characteristics, adding or modifying components.