An e-tool for undergraduate surveying education: design and evaluation

We describe an innovative e-tool for Surveying Education, report the results of a pilot study with a group of undergraduate students, and discuss lessons learned and future work. The e-tool consists of two parts: a student’s component and an instructor’s component. The students’ component is a 2D Virtual Learning Environment that can be used by students to review surveying concepts and practices and get feedback on their understanding of the subject. The instructor’s component is a summative assessment tool that measures the individual student’s cognitive and practical abilities with accuracy. Results of the pilot study show that students perceived the application as easy to use, useful for reviewing class content, and effective at providing immediate and accurate feedback on their performance. A comparison between the grades obtained by manually grading the field exercise and the grades generated by the e-tool showed a significant disagreement between the two sets of data, with the electronically generated grades being much lower. The study revealed the limitations of the current tool due to its dimensionality (i.e. 2D) and suggested the need for a more realistic 3D learning environment.


Introduction
Surveying is a fundamental course in the Civil Engineering, Building Construction Management, Geomatics, Agriculture & Forestry, and Landscape Architecture curricula. One of the challenges of surveying education is the difficulty in assessing the individual student's cognitive and practical abilities. This difficulty is due to the fact that many surveying exercises are team efforts that require collaboration between at least two students. For instance, exercises that involve tape measurements, use of level to measure differential elevations, or theodolites to measure angles and distances are tasks that need to be completed by two or more students working closely together. It is challenging to evaluate the individual student's knowledge and performance, as it only takes one student to make a mistake in order for the team to get the wrong measurement. It is not uncommon for good students to get penalized for the mistake of another team-mate, or for mediocre students to get high grades because the other team members are knowledgeable and proficient at using the instruments.
The goal of the work reported in the paper was to enhance traditional surveying instruction methods with a unique approach: an effective virtual learning environment (VLE)/E-assessment tool. The prototype program described in the paper contains one educational module (chaining) and it is a first step toward the development of a surveying e-tool with five educational modules that will be integrated in introductory surveying courses. Students can use the e-tool to review concepts and practices and get formative feedback on their understanding of the subject; the instructor can use the program as a summative evaluation tool to generate a grade that truly reflects the student's performance. In the paper we report findings of a pilot study with 31 subjects. Evaluation results show that students reacted positively toward the program, were engaged with it, and found the software to be very valuable. Comments from the instructors show that the prototype program was perceived as a very effective assessment tool. However, the study revealed several limitations of the program due to its dimensionality (2D).
The paper is organized as follows. In Section 2 we define and discuss VLEs and e-assessment, and in Section 3 we describe the design and implementation of the prototype program. In Section 4 we report a pilot study with a group of undergraduate students enrolled in the Building Construction Management program at Purdue University and we analyze the results. In Section 5 we discuss findings and lessons learned, and in Section 6 we describe current and future work. Conclusive remarks are included in Section 7.

Virtual environments for engineering education
The pedagogical benefits of interactive VLEs have been examined by researchers in the areas of computer graphics, cognitive psychology, visual cognition, and educational psychology. In general, research findings show that VLEs can be more effective than traditional teaching tools [1][2][3]. Research also shows that VLEs are particularly suitable to mathematics and science education. VLEs present concepts in concrete terms and offer a valuable alternative to the conventional study of mathematics and science, which is based primarily on textual descriptions and 2D representations [4]. VR also supports learning in a nonlinear fashion, which has been shown to be effective in teaching students how to be critical and creative thinkers [5]. Computer simulations have been shown to be an effective approach to improve student learning and have the potential to help students develop more accurate conceptions [6][7][8].
Research shows that the use of simulation tools often reinforces learning and leads to performance improvements in a variety of disciplines. Therefore, recently, there has been significant progress in development of computer-based tutorial systems in many different areas.
Though progress has been less evident in engineering education [9] there are some noticeable examples of engineering virtual laboratories. For instance, Del Alamo, a professor of electrical engineering at MIT, created a webbased microelectronics laboratory for his students [10]. At Johns Hopkins University, Karweit [11] has simulated various engineering and science laboratories on the web.
In the area of surveying, Kuo et al. [12] have recently developed a virtual survey instrument (SimuSurvey) for visualizing and simulating surveying scenarios in a computer-generated VE, and studied the feasibility of intro-ducing SimuSurvey in regular surveyor training courses. Results of the study indicated improved student learning outcomes and positive attitude toward including Simu-Survey in regular surveyor training courses. At Leeds Metropolitan University, UK, Ellis et al. [13] have developed an undergraduate VR surveying application. The interactive software includes 360-degree panoramic images of sites and makes use of QuickTime VR technology. The application was evaluated with 192 undergraduate students; findings suggest that the interactive tool complements traditional learning approaches, maintains student interest, and reinforces understanding. At University of New Castle, UK, Mills and Barber [14] have implemented a virtual surveying field course which includes both a virtual field trip and a virtual interactive traverse learning tool (VITLT). The goal of the tool is to improve understanding of surveying methods for first-year students in the Geomatics degree. The application was evaluated by several Geomatics students; all subjects highlighted the potential of VITLT to help the learning and understanding of a traverse. However, the students did not see the e-learning tool as a replacement for a traverse observation as carried out on the field course, but suggested that it could be used as a preparation and revision tool.
Although some authors have documented that VLEs provide advantages over more traditional instructional methods [15], studies of VR projects are still relatively rare and a need exists for investigations of VLEs in the undergraduate classroom [5].

E-assessment
The term e-assessment is becoming widely used as a generic term to describe the use of computers within the assessment process. In general, e-assessment tools provide two forms of assessment: formative and summative. Formative assessment constitutes a learning experience in its own right and is concerned with the provision of developmental feedback to the learners such that students can gain from the feedback provided and adjust their learning style as appropriate [16]. Summative assessment is usually undertaken at the end of a period of learning in order to generate a grade that reflects the student's performance.
According to Howarth [16], e-assessment has many advantages over traditional paper-based assessment including: lower long-term costs, instant feedback to students, greater flexibility with respect to location and timing, improved reliability (machine marking is much more reliable than human marking), and enhanced question styles which incorporate interactivity and multimedia. Public and private sector experts have stated that computers, telecommunications, audio-or video-based media are critical enablers of learning, hence there is a need for assessment tools that measure those essential skills that cannot be captured by traditional tests [17]. Fogel [18] argues that e-assessments provide the essential feedback for true 21 st century education transformation in which student outcomes can be correlated to a cause-and-effect and in which there is continuous improvement of the e-Learning environment. The public-private coalition known as the 'Partnership for 21 st Century Skills' gives a vision of how students should be prepared to face the challenges of the 21 st century and underlines the benefits of using technology to give immediate and accurate feedback on student learning [17].
There are also disadvantages. E-assessment systems are expensive to establish and not suitable for every type of assessment (such as extended response questions). Educators need specific skills to create e-assessment resources, and producing e-assessment tools is a time-consuming process. Electronic testing has also been accused of bringing non-technology students to a disadvantage as students are required to use a computer to enter their answers [19,20].
Recently, several researchers have focused on development and evaluation of e-assessment tools for collegelevel learning. Doukas and Andreatos [21] have presented a computer-aided summative assessment system (e-Xaminer) to produce and deliver tests to the Hellenic Air Force Academy students and assess their performance. e-Xaminer uses meta-language concepts to generate tests based on parametrically designed questions. Examinations are delivered via a web-based interface and the system grades the answers submitted by each student. e-Xaminer also allows for implementation of question parameterization and counter cheating measures. The researchers conducted a pilot study that compared paper-and-pencil examinations versus the electronic examinations in digital electronics, computer science, microprocessors, and computer network courses. Results showed that the deviation between the manually graded tests and the electronically graded ones was less than 1% and over 90% of the students thought that the electronic test was equally difficult and preferable to the traditional one. In addition, students expected their automatically assigned marks to better reflect their performance.
Perry et al. [22] report a project whose goal was to introduce and evaluate a hybrid formative/summative e-assessment tool in an introductory course in Chemical Engineering. The e-assessment tool was created using Respondus [23] and the e-tests were delivered by WebCT4. Answers from a questionnaire completed by tutors and students showed that over 80% of the students found the feedback provided by the e-assessment tool to be very useful and helpful in determining the areas of learning that needed improvement. Tutors noted that the e-test saved about a day's work and had the main advantage of allowing students to take the test from home.
Andreatos and Michalareas [24] describe a Matlabbased e-assessment application for an introductory course in analog electronic design. The application included a student interface and an instructor interface. Students designed a transistor amplifier and provided their answers through their interface, and the instructor could automatically evaluate the student answers qualitatively and quantitatively.
Moscinski [25] reports examples of using Model-based tools for summative e-assessment. The e-assessment tools were tested in both theoretically oriented courses on control systems, as well as software and technology-oriented courses on computer networks and Internet technologies.
The questionnaire-based analysis demonstrated the popularity and efficiency of the e-assessment tools and methods both among students and teachers.

The surveying e-tool
To date, we have developed a prototype 2D e-tool that includes one educational module. The application consists of two components: (i) a 2D virtual learning environment (VLE) that is used by the students to review concepts and procedures and perform surveying exercises; and (ii) an evaluation engine that tracks the student's interactions with the program and outputs performance reports.
(i) The educational content of the student VLE focuses on chaining. The goal of this first educational module is to help students visualize and apply the concepts of chaining in various scenarios including horizontal plane, steep slope, rough terrain, error of standardization of steel tape, error due to temperature, and error due to both temperature and standardization. The VLE includes reference documentation on surveying methods and the students learn and practice how to measure the horizontal distance between two points using the proper techniques and instruments. Students are required to use virtual instruments such as steel tape, plumb bobs, hand levels, tension meter, and pins, and are expected to measure the horizontal distance precisely and accurately. Screenshots of the student VLE are shown in Figure 1; a video demonstration of the program can be viewed at: http:// www2.tech.purdue.edu/cgt/i3/VELS/. A detailed description of the VLE can be found in [26]. (ii) The evaluation engine tracks the student's interactions such as (a) the student ability to select the correct tools; (b) the student ability to set up at the correct point of interest; (c) the student ability to hold the tape horizontally, therefore the level has to be perfectly plumb; (d) the student ability to exert the correct amount of tension on the tape, so that the tape can read the horizontal distance; (e) the reading on the tape as a record of the student's measurements; (f) the student decision to delete or retain a specific reading (this is used to evaluate the student interpretation of the results); (g) the time spent on each task; and (h) the number An e-tool for undergraduate surveying education: design and evaluation EAI European Alliance for Innovation of correct and incorrect answers. The evaluation engine outputs two types of reports: a summary report that provides formative feedback to the student ( Figure 2) and a detailed performance report for the instructor in the form of an excel spreadsheet. The instructor uses this report to generate the final grade.

Pilot study
The objectives of the study were: (i) to determine whether there is a significant deviation between the grades obtained by manually grading the chaining field exercise and the grades generated by the e-assessment tool; (ii) to collect feedback from the students on the usefulness and usability of the tool; and (iii) to collect feedback from the surveying instructors on the perceived effectiveness and accuracy of the program.

Subjects
The pool of subjects included 31 male undergraduate students and 2 faculty with experience in surveying education. The students were enrolled in a Construction Surveying Fundamentals course in the College of Technology at Purdue University. The subjects who volunteered to use the e-tool were students who needed additional credits to improve their grades in the class.

Procedure
The goal of the exercise presented to the students was to measure the horizontal distance between two points with the required precision and accuracy. The subjects performed the chaining exercise in two settings: (i) in the field and (ii) in the surveying laboratory using the e-tool. The instructors graded the exercise in both settings.
Setting (1). The students measured the horizontal distance between two points marked on the ground with the help of a colleague. Students used a steel tape, plumb bobs, tension meter, hand levels, and hand clamps, and had to ensure that the tape was held horizontally at the two points and the correct amount of tension was exerted in order for the tape to be correctly stretched between the two points. The students recorded their measurements, adjusted for temperature and tape standard error, and reported the measurements in a log book. The instructors timed the exercises and compared the recorded values to the correct values. The students were graded based on how close their measurement was to the true value.
Setting (2). The students were first given guidelines on how to use the program; they were then provided with a set of directions and assumptions for the chaining exercise. The goal of the exercise was to measure the horizontal distance between points A and B with precision and accuracy-the tool presents six possible points. The following assumptions were to be considered: the terrain is a rough terrain, the temperature is 86 F, and the error in the tape is 1/100th per cent short, i.e. when the tape measures 100 feet it is in reality 99.99 feet. The students were instructed to use e-assessment tool to measure the average distance between A and B. Once the average value was determined, the students had to adjust for the tape error and the temperature error using the correction formulas. Each individual student used the e-assessment tool, completed the chaining exercise, and received formative feedback from the program under the supervision of the experimenter. The instructors generated the

Analysis of results
The students' average e-grade (i.e. the grade obtained with the e-assessment tool) was 65%, whereas the student average m-grade (i.e. the manually generated grade resulting from the field exercise) was 75%. The same number of students who achieved an 'A' in the field test earned the same grade using the e-assessment tool. One out of four students was able to earn a 'B' in the field exercise and achieved the same grade using the e-assessment tool. None of the students earned an 'F' as m-grade while nine students earned an 'F' as e-grade. A weighted kappa measure of agreement, a paired t-test, and a sign test were performed in order to determine any correlation between the students' grades obtained by manually grading the chaining field exercise and the grades generated by the e-assessment tool. All three tests show that there is low agreement between the two sets of grades. Based on the weighted kappa value, the agreement between m-grades and e-grades appeared to be very weak.
The 'Paired t-test' was used to test whether the difference between the e-grades and the m-grades within each student was zero or not. We computed the differences between the e-grades and m-grades of each student and if a standardized mean of differences was too large (or too small), then we could conclude that e-grades and m-grades are different. In this test, since the p value (Pr > |t|) is very small, we concluded that e-grades and m-grades are different.
One drawback of the t-test method is that it requires normality of data; in our case this assumption was not met. Therefore we used a Sign test, which is a non-parametric method that does not require such normality of data. The Sign test counts the number of cases where the m-grades are higher than the e-grades and the number of cases where e-grades are higher than m-grades. If there is no difference between e-grades and m-grades, the two numbers would be very similar; and if some difference exists, then either one of two numbers is larger than the other. In this test, the p value (Pr |M|) was also very small, so we could conclude that e-grades and m-grades are different and specifically, m-grades tend to have greater value than e-grades. Details of the kappa measure of agreement, paired t-test, and sign test can be found in [27].

Students' observations
The students were asked to provide feedback on the usability and usefulness of the e-assessment tool. Seventy-six per cent of the students thought that the e-assessment tool was a good learning tool, as it helped them visualize fundamental steps and procedures. Seventy-six per cent of the students thought it was very helpful in terms of capturing the essence of the chaining exercise and 40% felt that it replicated the field exercise with accuracy. However, 60% of the students commented that the e-tool cannot replace the actual field experience. Sixty per cent thought it was a good practice tool and some of them recommended that it should be used in the classroom for review and practice. Twenty-eight per cent thought it was easy to use, while 8% felt it was difficult at first. Eight per cent of the students observed that the e-assessment tool allowed them to think ahead about every step they needed to make.

Instructors' observations
The instructors commented that with the e-tool they were able to calculate the individual student grades based on a very thorough report of their performance. Students were assessed based on their ability to select the correct tools the first time, ability to select correct procedures the first time, making more than two readings in order to eliminate the random error generated by the instruments errors, and making the correct judgment by deleting the erroneous and outlier measurements if the deviation was larger than the allowable instrument errors. In the field exercise it was not possible to track all these factors. For instance, students selected the required tools and performed the measuring procedures with a colleague, hence it was not possible to analyze the individual student performance.

Discussion and lessons learned
The disagreement between the m-grades and e-grades and the observation that the m-grades are generally higher than the e-grades suggest that this difference is due to the ability of the e-assessment tool to track the individual student's performance with higher level of accuracy.
The results of the pilot study are promising. Students found the e-assessment program a useful tool for learning and for providing formative feedback on their level of understanding of chaining concepts and procedures. Instructors commented that it is a very effective summative assessment tool that allows educators to calculate a grade that truly reflects the individual student's performance. However, faculty also pointed out that the dimensionality of the current tool (i.e. 2D) presents the following limitations: (i) The 2D environment cannot replicate field scenarios with high level of fidelity, as in real life users have to use 3D equipment in a 3D environment. With 2D illustrations and/or animations, it is not possible to effectively simulate how surveying instruments operate. (ii) The 2D illustration/animation of the instrument is a very simplified representation that requires abstraction from the user and does not replicate accurately how the instrument looks and functions in real life.
(iii) Illustrating the chaining problem using a series of 2D images provides the students with a partial solution. For instance, (a) in the 2D environment students cannot be challenged for alignment. When faced with placing intermediary points, the intermediary points should align with the starting and ending points. The 2D illustration implies the solution to the student. Screenshots 1 and 3 shown in Figure 3 below show a top view, guiding the student throughout the steps required for alignment of intermediate points with the end points. (b) In the 2D environment students cannot be tested on the technique of breaking down the chain. This technique requires the students to break the distance into increments in order to hold the tape at a horizontal distance between the two measured points. The illustration in Figure 3screenshot 4 suggests the solution as the students can see on the cross-section view the rough terrain and the representation of the user in the model. (c) In the case of a rough terrain students cannot view the starting points and end points due to the 'curves' of the terrain. Therefore, they have to think about their choice of points in order to satisfy the alignment and breaking down the chain criteria. In order to represent this scenario in a 2D environment, we provide the students with a cross-section of the terrain, thus giving them a partial answer to the problem. Figure 3 shown below presents the four screen shots the student needs to follow in order to adjust all the variables as far as alignment and to keep the tape horizontal.
In order to overcome these limitations, we are developing a 3D version of the tool that includes realistic 3D virtual terrains and surveying instruments. The 3D terrains allow for representation of realistic field scenarios without providing a partial solution; the 3D instruments require less abstraction from the user as they look, operate, and produce results comparable to the physical ones.
6. The 3D tool: initial development and future work

Technical implementation
The 3D tool is being developed using the highest end in 3D interactive animation. We use Autodesk Maya software to model and texture the virtual instruments and to animate their functionality; interactivity with the 3D components is programmed in C# using the Unity game development platform (Unity 2010). The 3D application supports import of Digital Elevation Model (DEM) data and allows for real-time generation of realistic 3D terrains (represented as polygonal meshes) based on these data. The program is designed to run on hardware and software infrastructure that is already widely deployed in universities. Students will be able to use it on low-end personal computers (PC/MAC) with a low-end graphics card. Different strategies are being used in order to optimize performance. Geometric complexity of the 3D models is kept at a minimum, while retaining visual quality, to ensure client hardware can run the application at interactive rates. Normal maps, a technique for simulating complex geometric detail, are being used to add fine detail to objects without adding extra geometry. Level of detail is also being employed to find an accurate balance between performance and visual quality on the client machine; light maps are used to provide high-quality lighting for static geometry without impacting performance. Figures 4  and 5 show two screenshots of the 3D tool.

Educational content
The 3D tool has an open architecture that supports flexible customization. Educators will be able to easily modify existing content and add new educational modules to fit the needs of the course. In addition to chaining, we are currently working on development of three educational modules that are specifically designed to address common students' learning challenges. The modules are in based in part on [28,29].
Module 1: differential levelling. At the end of this module students will have a working knowledge of the procedures that have to be followed in order to determine the elevations of a relief and points of interests. Levelling concepts/exercises will include: the peg test-to determine the instrument error; closed-loop levelling; open-loop levelling-example of determining the elevation for a road; contour lines creation (currently, a major challenge to the students); calculating and determining quantities for excavations and backfill (currently another major challenge to the students). Students will be able to determine the acceptable level of accuracy of the taken measurements, distribute the error if acceptable, and perform all the math checks required to detect recording errors.
Module 2: triangulations and coordinate calculations. This module illustrates the proper techniques and the math skills needed in order to determine point coordinates and apply vectors concepts. At the end of the module, students will be able to: calculate horizontal angles and vertical angles; find a meridian/reference line; find the correct orientation and directions concepts of azimuth and bearings; calculate the coordinates based on bearings and distances; perform traverse calculations; compute angular errors; determine closure error; distribute error for closure and balance the angles and distances.
Module 3: current technologies in surveying. In this module the student will become familiar with the instruments, their functions, limitations, and how to perform best practices in order to reduce error and increase accuracy. Students will learn: how to use GPS equipment to perform tasks in module (4); how to use Robot equipment to perform tasks in module (4); CORS-Continuous Operating Reference Station, ability to refine the measurements using the data collected from the CORS station; how to use Rover instrument to work in the field triangulating between the satellites systems and the CORS; how to use the Barcode level to determine the elevation to the thousands of the unit; how to determine the heights of the instruments to set up grade using automatic machine graders.

Conclusion
This paper presents the development of an e-tool for surveying instruction, reports the findings of a user study with 31 undergraduate students, highlights lessons learned, and describes ongoing and future work. The results of the pilot study are promising. Students found the program useful for learning and providing formative feedback on their level of understanding of chaining concepts and procedures, and instructors commented that the software is an effective summative assessment tool. However, the study revealed several limitations of the current application due to its dimensionality, i.e. 2D. Our ongoing work focuses on developing a 3D version of the application that overcomes current limitations by including realistic 3D virtual terrains and instruments. Future work will involve extending the content of the application to include three additional teaching modules (Differential Levelling, Triangulations and Coordinate calculations, and Current technologies in Surveying), and evaluating the 3D e-tool with a larger sample size.