EVALUATING THE USE OF VIRTUAL REALITY FOR MAINTAINABILITY-FOCUSED DESIGN REVIEWS

SUMMARY: The Operation and Maintenance (O&M) phase can account for as much as 80% of the overall lifecycle cost of a project. The inputs from O&M practitioners are often not incorporated during the design phase leading to maintainability issues during the O&M phase, which results in significant costs and hinders building performance. Traditionally, design communication with practitioners is conducted using 2D drawings and in some more advanced projects, with Building Information Models (BIMs). Virtual Reality (VR) has the potential to facilitate maintainability-focused design input from O&M practitioners, but this application has not been studied in detail. This paper addresses the gap by examining the usability of VR for providing access-related maintainability inputs. A quasi-experimental approach was adopted to evaluate the difference in access-related inputs provided by sixteen O&M practitioners, once using 2D drawings and BIM and again using VR. We assessed the variation in time for O&M practitioners to provide the inputs and the perceived quality of the inputs using the two design communication methods. We also examined the effect of practitioners' years of experience on providing access-related maintainability inputs using VR. The results suggest that there was no statistically significant difference in the access-related maintainability inputs provided using the two design communication methods, which demonstrates that O&M practitioners could provide similar access-related inputs using VR and traditional means of communication using 2D drawings and 3D models. The results also show that O&M practitioners were able to provide inputs significantly faster using VR. The O&M practitioners also reported that VR facilitated ease of use and high confidence in their inputs. Furthermore, the practitioners' years of experience did not lead to any variation in the access-related inputs provided and the time efficiency of providing the inputs. While the small sample size limits the generalizability of the results, the study acts as a proof of concept on the usability of VR for improving O&M practitioner input on the maintainability of building designs. maintainability inputs from O&M practitioners over the traditional means of using 2D drawings and 3D models. The findings are limited by the sample size of the participants and the complexity of the design problem. Future research will focus on increasing the complexity of the design problem by integrating other systems' impact and evaluating different use cases for VR adoption.


INTRODUCTION
The Operation and Maintenance (O&M) phase has a significant impact on the financial outlay of a building and can contribute up to 50-80% of the project lifecycle cost (Assaf et al., 1995;Blanchard et al., 1995;Lee et al., 2012;Sapp, 2017). The rework on defective components detected during the O&M phase can cost up to 5% of the project's construction cost (Boukamp and Akinci, 2004). The frequency of O&M issues is increasing with the complexity of the buildings and leading to higher operation costs (Lai, 2010). Addressing O&M-related issues is vital for long-term project success from the perspective of building owners and investors.
The O&M issues encountered in a building can often be attributed to decisions made during the design phase (Khalek et al., 2019a). The lack of maintainability considerations in the design phase can increase the buildings' operations and maintenance costs (Ballast, 2013). Erroneous design decisions are generally caused by a lack of engagement of O&M practitioners during the design phase (Chong and Low, 2006;De Silva et al., 2016). The communication and collaborative exchange between designers and O&M practitioners are uncommon in the construction industry because of traditionally siloed procurement methods. Moreover, facility management teams are generally not appointed during the design phase (Erdener, 2005). However, with the advent of new multi-party agreements like Integrated Project Delivery (IPD), collaborative exchanges between designers and O&M practitioners are becoming more common (AIA, 2007). The increase in collaborative exchanges between stakeholders provides an avenue for integrating maintainability considerations in the design phase that can lead to more sustainable and cost-efficient buildings during the O&M phase (Ganisen et al., 2015;Wood, 2011).
The typical approach for engaging O&M practitioners in the design phase is by conducting maintainability-focused design reviews (Liu et al., 2018). Such reviews aim to obtain the O&M practitioners' buy-in of the design and ensure the operability and maintainability of building systems. Traditionally, two-dimensional (2D) drawings are used for communicating and gathering inputs during the design review sessions (Gould and Joyce, 2009). With the increasing use of Building Information Models (BIM) in the construction industry, intelligent threedimensional (3D) models are also becoming available to support design, construction, project delivery, and facility management (Hardin and McCool, 2015). Despite their increased usage, viewing virtual BIM content for identifying maintainability concerns can be challenging, and maintainability issues can easily be hidden within the 3D model (Liu and R.A. Issa, 2014). Additionally, it has long been recognized that 2D drawings present difficulties with accuracy and mental visualization compared to 3D alternatives (Carvajal, 2005;Cory, 2001). Generally, frequent transitions between 2D drawings and 3D models are required to fully understand an issue, identify its impact, and provide feedback (Mehrbod et al., 2017). This process of interpreting 2D drawings and 3D models into mental models can introduce inefficiencies and opportunities for mistakes due to misinterpretation (Dadi et al., 2014). It has been reported that about 26% of the issues encountered during design reviews result from inefficiencies introduced by using 2D drawings for design communication (Kim et al., 2018). These limitations with 2D drawings and 3D models warrant exploring alternate modes of design communication for maintainabilityfocused design reviews.
Virtual Reality (VR) is the term used to describe a 3D, computer-generated immersive environment that is interactive and explorable by a person (Chang, 2016). VR aims to enhance the sense of presence by improving interactivity through stereoscopic visualization, tracking head, body, hand gestures, and allowing for manipulation of scene elements (Paes and Irizarry, 2018). Using an immersive platform like VR for design review sessions has been shown to increase the user's concentration on the review task and awareness of issues under analysis. In addition, when using VR, reviewers would not get distracted, discouraged, or disinterested, thus decreasing decision-making time significantly (Majumdar et al., 2006). Other benefits include enhanced collaboration among stakeholders (Bassanino et al., 2010;Fernando et al., 2013), better spatial understanding of virtual prototypes in comparison to 2D drawings and 3D models (Paes et al., 2017;Schnabel and Kvan, 2003), the anticipation of design decisions and identification of issues otherwise not possible (Bassanino et al., 2010;Dunston et al., 2011), and prediction of human-building interactions that can provide designers with reliable user behavior data (Adi and Roberts, 2014;Bassanino et al., 2010;Heydarian et al., 2015;Kuliga et al., 2015).
The aforementioned benefits of VR are primarily derived from user tests conducted during design-developmentfocused reviews. These reviews are typically attended by architects and engineers, who are either the author of the design or familiar with the design progress (Liu et al., 2018). In contrast, during maintainability-focused design reviews, participants are primarily O&M practitioners, owner's representatives, and end-users who generally have little or no prior exposure to the design. Furthermore, it has been reported that O&M practitioners usually have just 1-2 days at the most to review all the project's documentation (Kim et al., 2018). Consequently, they have difficulty conducting a thorough review of the project's documentation due to insufficient time. These constraints of maintainability-focused design reviews support the exploration of using an immersive platform like VR with its documented benefits of providing improved spatial understanding and reducing decision-making time. However, the usability of VR to support maintainability-focused design reviews with O&M practitioners has not been studied in detail.
This study aims to address the research gap by assessing the usability of VR for O&M practitioners to provide maintainability-focused design inputs during design review sessions. Specifically, this paper addresses the following research questions: 1. How does the use of VR affect the maintainability-focused design inputs provided by O&M practitioners, time required for providing inputs, and perceived quality of the inputs provided? 2. How does an O&M practitioner's years of experience affect the maintainability-focused design inputs provided using VR and the time required for providing those inputs?
These research questions are addressed by executing a quasi-experimental procedure with O&M practitioners. The participants in the study were asked to provide access-related maintainability inputs once using traditional means of 2D drawings and 3D models and again using VR. The access-related inputs provided using the two design communication methods and the corresponding time taken for providing the inputs were compared to determine any statistically significant differences. The participants' perceived quality of the inputs provided and years of experience were identified through pre-task and post-task questionnaires. The following sections outline the research approach and findings in further detail.

Designing with Maintainability Considerations
A design that considers maintainability ensures that a failed piece of equipment can be restored effectively and safely within an expected period and according to the prescribed O&M procedures (Barbarosoglu and Arditi, 2019). Maintainability requires considering the physical design characteristics of components placed in a space, such as visual access, ergonomics, and maintenance access (Stapelberg, 2009). It has been reported that 20% of maintainability issues result from decisions made during the design phase (Chanter and Swallow, 2007). The lack of maintainability considerations in the design phase can significantly increase buildings' O&M costs (Ballast, 2013).
Lack of consideration for maintenance access to building components has been recognized as the leading cause of O&M issues in buildings (Barbarosoglu and Arditi, 2019). Access-related issues restrict O&M practitioners from making specific movements and behaviors required to maintain a component. Poor maintenance access affects the status of building components, jeopardizes the safety of maintenance personnel, causes rework, and requires extra work and cost (Khalek et al., 2019b). The common cause for variation between the anticipated time required and actual time spent on maintenance activities has been attributed to access-related issues (Barbarosoglu and Arditi, 2019). Consequently, it is vital to consider the maintenance access for building components while designing with maintainability considerations.
The current methods for incorporating maintainability considerations in design suggest the use of checklists for guiding the designers to create more suitable designs in general and for specific systems (Hassanain et al., 2015;Ikpo, 2009). However, studies have shown that such checklists can lead to cognitive overload, resulting in overdesigning (Tjiparuro and Thompson, 2004). Some studies have suggested general maintainability criteria such as ensuring enough space to allow access and reach to components (Ganisen et al., 2015), reducing the general complexity of systems included (Zhu et al., 2018), making the design more easily adjustable in the future if needed (Ismail and Ibrahim Mohamad, 2015), and utilizing longer-lasting components (Ganisen et al., 2015). However, these recommendations are often neglected under the conventional design review approach because the design changes often and reviewing each change according to maintainability criteria creates an extra burden for the designers (Cavka et al., 2015). The inadequacies of the current methods have led researchers to explore new avenues, such as using BIM to integrate maintainability considerations into the design process.

Building Information Modeling
Building Information Modeling involves developing intelligent 3D models that encompass information related to the intrinsic properties of modeled objects stored in an attached database (Hardin and McCool, 2015). BIMs can contain information such as precise geometry, cost estimates, material inventories, and project schedules, which can be referred throughout the process of design, construction, operation, and maintenance (Azhar et al., 2008;Sacks et al., 2018). While the use of BIM has been increasing during the design and construction phase, researchers are just beginning to realize its benefits in the O&M phase (Khalek et al., 2019a).
BIM has been used to help maintenance personnel locate and assess work orders in facilities (Lu et al., 2018) and optimize space utilization and travel time in facilities (Kumar and Cheng, 2015). Additionally, studies have demonstrated the value of BIM for improving the information handover process and increasing the accessibility and accuracy of facility management data (Love et al., 2015). BIM-based frameworks have been developed to support safe maintenance and repair practices during the O&M phase (Wetzel and Thabet, 2015). These studies show the potential for using BIM in the O&M phase.
However, few studies have incorporated BIM in the design phase for O&M purposes (Roper and Payant, 2014). The traditional maintainability-focused design review is often conducted using 2D drawings, the accuracy and efficiency of which are highly dependent on the designers' experience (Zhou et al., 2010). Similarly, dealing efficiently with 3D models can be challenging for O&M practitioners. They often lack the means or expertise to leverage the BIM for facility management during the operations phase (Cavka et al., 2017). Furthermore, maintainability issues can easily be hidden within the model (Liu and R.A. Issa, 2014). Studies have proposed using maintainability checking algorithms and model checker add-ins for design authoring software to accommodate maintainability requirements in the design phase (Barbarosoglu and Arditi, 2019;Liu and R.A. Issa, 2014). However, the adoption of such tools remains limited due to the customizations required for analyzing the 3D model from O&M practitioners' perspectives (Cavka et al., 2018). These limitations with the use of 2D drawings and 3D models for incorporating maintainability-focused design inputs from O&M practitioners highlight the need to explore alternate means for design communication.

Virtual Reality
Virtual Reality (VR) creates an immersive virtual environment that allows the users to interact in real-time with digital 3D objects with varied spatial orientations (Warwick et al., 1993). In recent years, VR has been used to tackle various design and construction problems combined with BIM. The use of VR has been investigated for project planning , construction education (Rekapalli and Martinez, 2007), and safety training (Sacks et al., 2013). VR has also been used to simulate and monitor site progress (Adjei-Kumi and Retik, 1997). The applications of VR for facility management have focused on improving the productivity of workers performing O&M tasks (Carreira et al., 2018), enhancing the effectiveness of design meetings between workers and managers Park and Kim, 2013) and developing a multiuser shared environment for virtual walkthroughs .
The main driving force for developing VR applications has been architectural design and marketing (Bouchlaghem et al., 2005). The designers often use VR to communicate design ideas with the clients in a more direct manner by generating walkthrough models (Ormerod and Aouad, 1997). VR has also shown a strong potential for use in the design review process (Kuliga et al., 2015). It shortened the review duration from 8 hours to 3 hours in a case study that compared VR with traditional methods for design communication. The increased concentration on review tasks and awareness of issues under analysis has been cited as the reasoning behind reduced decisionmaking time (Majumdar et al., 2006). VR has been shown to enhance collaboration among stakeholders and enable the identification of design issues that otherwise could be overlooked (Paes and Irizarry, 2018). VR enables a better understanding of the space by allowing users to visualize and immerse themselves in the designs and understand both the qualitative and quantitative nature of the space (Bouchlaghem et al., 2005). Using VR, users can evaluate the proportion and scale of building components using intuitive and interactive modeling environments (Kurmann, 1995).
Despite these well-documented potentials of VR, its usability for maintainability-focused design reviews has not been studied in enough detail. Specifically, the extent to which actual O&M practitioners can utilize VR to provide effective maintainability-focused design inputs compared to traditional 2D drawings and 3D models has not been evaluated. This paper addresses the knowledge gap by employing a quasi-experimental research approach which is detailed in section 3.

METHODOLOGY
This study uses a quasi-experimental research approach to determine the influence of VR on maintainabilityfocused design inputs provided by O&M practitioners. Specifically, the access-related maintainability inputs provided, time required for providing inputs, and perceived quality of the inputs provided by O&M practitioners using VR are compared to inputs provided with traditional 2D drawings and 3D models. This quasi-experimental methodology was adapted from similar studies that investigated the use of augmented reality for conducting design reviews and completing construction tasks in comparison to 2D drawings (Chalhoub et al., 2021a(Chalhoub et al., , 2021bChalhoub and Ayer, 2018;Hartless et al., 2019;Lee et al., 2020).
A convenience sampling technique was used, given the proof-of-concept nature of the study. Participants in the study were electricians, plumbers, and steamfitters from the Campus Energy Center (CEC) located at the University of British Columbia (UBC) in Vancouver, Canada. All the participants provided access-related inputs using two different visualization approaches, VR and traditional 2D drawings and 3D models. While using 2D drawings and 3D models to provide design inputs, the participants could instruct the research coordinator to navigate and section the BIM for visualizing an object in a 3D view. The overall research methodology is outlined in Fig. 1 and further detailed in section 4, section 5, and section 6.

Identifying common maintainability issues
The leading cause for O&M issues encountered in a building has been attributed to a lack of maintenance access to building components (Barbarosoglu and Arditi, 2019). Studies have categorized access-related issues into four groups: horizontal reach, vertical reach, obstruction, and ergonomic factors (Khalek et al., 2019a(Khalek et al., , 2019b. Horizontal and vertical reach represent access-related issues for building components that are difficult to reach on a horizontal and vertical axis, respectively. Access-related issues categorized under obstruction include components that block other components' range of motion during maintenance. Ergonomic factors refer to access-related issues caused due to limited physical maneuverability around a component during maintenance (Khalek et al., 2019b).
To verify these findings and subsequently develop the scenarios for testing, the researchers conducted a tour of the Center for Interactive Research on Sustainability (CIRS) building on the UBC Vancouver campus, accompanied by trades and facility managers. The 4-storey LEED Platinum-certified CIRS building has a gross area of 5,675 m 2 (61,085 ft 2 ) and complex mechanical rooms to support its sustainability features and systems. During the tour, the O&M practitioners highlighted the frequently encountered issues while performing scheduled or unscheduled maintenance on equipment in the mechanical room. Access for performing maintenance activities on equipment was highlighted as a frequent issue. The left image in Fig. 2 shows an example of access-related issues associated with horizontal reach and obstruction for a red pump located on the floor, as highlighted by the O&M practitioners. The right image in Fig. 2 shows an example of access-related issues associated with vertical reach and ergonomic factors for a piece of red equipment hung to the ceiling. The categorization of maintainability issues observed in the building was consistent with the literature, which informed the development of the test scenarios.
FIG. 2: Examples of maintenance access-related issues as highlighted by O&M practitioners in the CIRS building

Developing test scenarios
The experimental scenarios for testing were created using BIM of the mechanical room from the CIRS building that the researchers toured. Autodesk Revit was used for modifying the model. Two scenarios were created based on the groups of access-related issues that were observed. In both scenarios, a piece of equipment was positioned such that it had access-related issues in its original location. The participants were asked to determine the minimum distance by which they should move the equipment to make it more maintainable. The participants were informed that the equipment required bi-weekly maintenance and a work duration of 20-30 minutes to set a precedent for working conditions.
Scenario 1: The first scenario was developed based on the categories of access-related issues of horizontal reach and obstruction. An expansion tank was placed on the floor with its center at 2 feet 6 inches (76.2 cm) from the wall. A component of the expansion tank that required maintenance was located at 9 inches (22.9 cm) from the wall. The participants were asked to determine the minimum distance by which the equipment should be relocated away from the wall to make it maintainable for the future. The 2D plan, along with the sectioned 3D view from Autodesk Revit provided to the participants for the first scenario, is shown in Fig. 3.

FIG. 3: 2D plan and the 3D view from Autodesk Revit for the first scenario
Scenario 2: The second scenario was developed based on the categories of access-related issues of vertical reach and ergonomic factors. A piece of equipment that required regular maintenance was placed at the height of 8 feet (2.44 m) from the floor. The participants were asked to determine the minimum distance by which the equipment should be lowered to make it more maintainable. The participants were informed that the working conditions do not allow using a ladder or a stool. The 2D section view and the customizable 3D view from Autodesk Revit provided to the participants are shown in Fig. 4.

Developing VR environment and design artifacts
The researchers used Enscape (2021), a plugin for Autodesk Revit, to create a VR environment for testing. Enscape creates a real-time link with the BIM in Autodesk Revit. Any changes made to the BIM, such as moving a piece of equipment, are visible instantly in the VR environment. The researchers used HTC Vive Pro, a head-mounted display (HMD) device with two hand controllers, to immerse the participants in the VR environment. The HMD and hand controllers were connected wirelessly to the computer and allowed the participants to move around freely. Enscape calibrates the height of the participant in VR based on the location of the HMD. The location of the two hand controllers acts as a virtual representation of the participant's hands in VR. The setup enabled the participants to view the objects in VR at room scale from the perspective of their own height.
The 2D drawings in plan and section view for the two scenarios were printed using the BIM and provided to the participants. Additionally, the participants were provided access to the 3D model in Autodesk Revit. The participants could instruct the research coordinator to navigate and section the 3D model as required.

Pre-experiment
The participants signed an informed consent form as per the University's Research Ethics Board before the experiment. The informed consent enabled the researchers to use and analyze the data collected during the session, including multi-angle video and audio recordings. The participants were given a brief explanation about the research activities followed by a questionnaire on their background and perception of VR. The pre-task questionnaire inquired about the participants' age, years of experience, experience using VR, and frequency of using 2D drawings for their daily work activities.
Following the questionnaire, the participants were given a brief 5-10-minute demo of VR in which they were placed in a sample living room unrelated to the study. The participants could familiarize themselves with the VR environment during the demo and get acquainted with the use of hand controllers. The participants were also introduced to a think-aloud protocol where they were asked to state their thoughts while performing the research activities. This protocol helped the researchers to understand the participant's reasoning and underlying thought process.

During Experiment
The participants used VR and traditional 2D drawings with 3D models to provide access-related maintainability inputs. The researchers aimed to compare their performance while using the two design communication methods. A counterbalanced experimental design was used to make the comparison, adapted from a similar study for electrical conduits construction with mixed reality (Chalhoub and Ayer, 2018). The participants were allocated to two groups randomly. The participants in both groups provided inputs for the two scenarios alternately using the VR and 2D drawings with 3D model, as shown in Table 1. The setup ensured that the participants did not provide inputs for the same scenario using the two design communication methods consecutively. Furthermore, the setup helped avoid learning effects and order-induced errors, which could happen if all the participants started with the same design communication method. The participants from a group were asked to provide inputs using their allocated design communication method. The setup of the room where the experiment was conducted is shown in Fig. 5.

Fig. 5. Room setup for VR inputs and participants providing inputs for the two scenarios
When using VR for providing inputs, the participants were able to freely move around in the space and use their hand controllers to reach objects in the VR environment. The participants were asked to provide their inputs to the research coordinator verbally. The coordinator made changes to the BIM based on their inputs, and the changes were visible in real-time to the participants in VR. The participants verbally informed the research coordinator once they were satisfied with the new location of the equipment, and the task was marked as complete. The BIM with changes suggested by the participant was saved as a new file for comparison with the BIM before the suggested changes.
When the participants provided inputs using 2D drawings, the coordinator presented the 3D view of the BIM on the TV screen and performed any navigation, sectioning, or distance measuring requested by the participant. The BIM with changes suggested by the participant was saved as a new file to document the inputs provided using the 2D drawings and 3D model.

Post-experiment
After completing the task, the participants were given a new questionnaire about their overall experience and perception of VR after using it for providing access-related maintainability inputs. The questions inquired how confident the participant felt about maintainability inputs provided using VR. Furthermore, the questionnaire elicited responses about the consistency of the VR environment with real-world experience. The post-task questionnaire also inquired about the perceived quality of inputs that participants could provide using VR. The post-task questionnaire aimed to identify any shift in perception of the participants related to the use of VR for providing design inputs.

ANALYSIS
Three types of data were collected during the session: the pre-task and post-task questionnaires about their perceptions, distance data from the BIMs based on the access-related inputs provided to the coordinator, and audiovideo recordings documenting the reasoning behind the inputs suggested by the participants.
The participants answered a printed version of the questionnaire. The questionnaire responses were imported into a spreadsheet, and multiple-choice questions were assigned numerical values for subsequent analysis. The minimum distance for moving equipment suggested by the participants was calculated based on deviation from the original model and added to a spreadsheet. The audio-video recordings were transcribed to understand the participants' reasoning behind an input provided during the experiment. The findings related to the performance and perception of the two design communication methods are presented in section 7.

Participants
Sixteen O&M practitioners participated in this study, which included electricians, plumbers, and steamfitters. Out of 16 participants, 10 participants (62.5%) had over ten years of experience, 3 participants (18.75%) had experience between 6 to 10 years, and 3 participants (18.75%) had experience between 1 to 5 years. All the participants in the study were male, and their ages ranged between 30 years to 61 years. All the participants had little to no experience in using VR inside or outside of work. Twelve participants used 2D drawings daily for their work, and the other four participants used them with varying frequency based on their work tasks.
The participants in this study were actual O&M practitioners from the UBC Building Operations team and working at the Campus Energy Center (CEC) in the UBC Vancouver campus. Prior experimental studies that focused on testing VR usability were primarily conducted with university students as participants (Adi and Roberts, 2014; Heydarian et al., 2015;Kuliga et al., 2015). Furthermore, prior VR studies that recruited actual practitioners for usability testing of VR were limited to less than ten participants in most cases (Carreira et al., 2018;Paes and Irizarry, 2018). While the relatively small sample size of sixteen actual O&M practitioners may limit the generalizability of the results, this study is intended as a proof of concept on the usability of VR for improving O&M practitioner input on the maintainability of building designs. Subsequently, the results and analysis of the access-related inputs provided by the practitioners are outlined in section 7.2.

Analysis of Access-related Inputs
The participants used 2D drawings with 3D models and VR to provide access-related maintainability inputs for the equipment in the two scenarios. The access-related input of minimum distance for moving the equipment suggested by the participants using 2D drawings with 3D models and VR was compared for each of the two scenarios. A paired statistical test was used for the comparison. To select an appropriate statistical test, all data sets were subjected to the Shapiro-Wilk test of normality. The null hypothesis for the Shapiro-Wilk test is that the data are normally distributed, and the alternative hypothesis is that data are not normally distributed . If the p-value from the Shapiro-Wilk test is less than 0.05, then the null hypothesis can be rejected, suggesting that data are not normally distributed. A paired t-test is adequate for comparing a normally distributed data set; otherwise, the Mann-Whitney test is required (McCrum-Gardner, 2008). Table 2 shows the results of the Shapiro-Wilk test for the data set in terms of the p-values for the distance suggested by the participants using the two design communication methods in both scenarios. All the data sets except for distance suggested using the 2D drawings with 3D model in the second scenario had p-values less than 0.05, suggesting that the null hypothesis of the normally distributed data should be rejected.
Following this analysis, given that the data were not normally distributed, the paired Mann-Whitney test was used for comparing the minimum distance suggested by the participants using the two design communication methods in both scenarios. The null hypothesis for the Mann-Whitney test is that the distribution of the two data sets is equal. The alternative hypothesis is that the distribution of the two data sets is not equal. Suppose the p-value is less than 0.05, the null hypothesis that the distribution of the two sets being equal can be rejected. Alternatively, if the p-value is greater than 0.05, the null hypothesis cannot be rejected, suggesting that there is no statistically significant difference between the two data sets. Table 3 shows a comparison of the mean distance for moving the equipment as suggested by participants by using the two design communication methods in both scenarios, based on the Mann-Whitney test. The mean difference in the distances suggested using VR when compared to 2D drawings with 3D models was 2.125 cm for Scenario 1 (p-value of 0.589581) and 9.437 cm for Scenario 2 (p-value of 0.101642). However, these differences are not statistically significant, as the p-value for both scenarios was greater than 0.05. Subsequently, it can be stated that the familiarity of using 2D drawings did not lead to statistically significant differences in the access-related distance input provided by the participants for both scenarios. In other words, the participants were able to provide similar access-related maintainability inputs using VR and traditional means of 2D drawings and 3D models.
The observations from the audio and video recordings highlighted the differences in the decision-making approach adopted by the participants. When using VR, the participants walked around the equipment to see the available space. The participants used their hand controllers to physically reach a VR component and consequentially provided their inputs for moving the equipment. Using 2D drawings, they relied on their experience and mental visualization of the space for making decisions. The participants mentally visualized known measurements, such as the height of a door frame and used it as a benchmark for providing their input.
The participants visualized heights mostly without looking at the 3D model but preferred referring to the 3D model when visualizing the space around a piece of equipment. The sectioning and measuring tools in the 3D model were used most frequently by the participants during the process of providing inputs. The need for mental visualization and measurement was reduced in VR because the components were visible to the participants at room-scale and in relation to their height. The reduced need for mental visualization led to differences in the time required for providing the inputs using the two design communication methods, as outlined in section 7.3.

Time Analysis
The researchers compared the time required by the participants to provide inputs using the two design communication methods in both scenarios by using paired statistical analysis. The Shapiro-Wilk normality test was used to check if the data sets were normally distributed. Table 4 shows the p-values from the Shapiro-Wilk test for the time required to provide inputs in both scenarios. Both data sets for the first scenario were normally distributed (p-value > 0.05), suggesting that paired t-test could be used for comparing the time required for providing inputs in the first scenario. In the second scenario, the data set for the time required to provide inputs using VR was not normally distributed (p-value < 0.05). Hence, the paired Mann-Whitney test was used for comparing the time required for providing inputs in the second scenario. Table 5 shows the time comparison of using VR and 2D drawings with 3D models in the two scenarios. The results suggest that using VR was comparatively faster than using 2D drawings with 3D models for providing inputs in both scenarios and the difference was statistically significant (p-value < 0.05). VR was 36% more timeefficient than using 2D drawings with 3D models for providing access-related maintainability inputs in the first scenario that required the participants to suggest the minimum distance for moving the equipment away from the wall. In the second scenario, VR was 33% more time-efficient than using 2D drawings with 3D models for providing access-related maintainability inputs related to height.
The observations from the audio and video recordings suggested that the increased time efficiency for providing inputs using VR resulted from the real-time sense of space that VR provided. The participants could make decisions faster by looking at components at room-scale and physically moving around in the VR environment. In comparison, participants had to spend time looking at 2D drawings to identify distances and view the 3D model sections to understand the space entirely. This transitioning required for building a mental visualization of the space was time-consuming. The results are noteworthy given the participants' years of experience and their familiarity with using 2D drawings. All the participants had little to no experience using VR before the experiment and were still faster at providing inputs using VR.

Demographic Effect
The researchers investigated whether the participants' years of experience had any influence on their performance for providing inputs using VR. Younger generations are shown to be significantly better at using technology because of their exposure from a young age (Marc, 2001). Alternatively, some studies suggest training is required for younger generations, the same as older generations when using new technologies (Kirschner and Bruyckere, 2017;Margaryan et al., 2011). A study that used augmented reality for the electrical point layout task reported varied performance based on the participants' years of experience . There were two potential effects of interest for the researchers: the effect of experience on the access-related inputs provided using VR and the effect of experience on time taken to provide inputs using VR.
To identify these effects of interest, the performance of participants who had over ten years of experience was compared with participants who had between 1-5 years and 6-10 years of experience. The maintenance accessrelated distance inputs and time required to provide the inputs were used as the metrics for comparison. Mann-Whitney test was used to make a comparison between the different groups. Table 6 shows the comparison of access-related inputs provided by the groups with different years of experience when using VR. The difference in the maintenance access-related distance inputs provided by the participants with over ten years of experience when compared to participants with 1-5 years and 6-10 years of experience was not statistically significant (p-value was greater than 0.05 for each comparison). Expressly, the results suggest no statistically significant difference in the access-related inputs provided by the participants irrespective of their experience. The years of experience of the participant do not seem to affect the access-related inputs provided using VR. In other words, participants with varying years of experience can provide similar access-related maintainability inputs using VR.
The participants were significantly faster at providing inputs using VR than 2D drawings and 3D models in both scenarios. The researchers investigated whether the years of experience of the participant had any effect on the time required for providing inputs using VR. Table 7 shows the results of the Mann-Whitney test used for this comparison. The results suggest that there is a difference of 12.37 sec in Scenario 1 (p-value of 0.370629) and 13 sec in Scenario 2 (p-value of 0.272727) between the groups with 1-5 years of experience and more than 10 years of experience. However, these differences are not statistically significant as their p-values are greater than 0.05 for all the comparisons, irrespective of the years of experience. In other words, there is a high uniformity in the efficiency of using VR to provide access-related maintainability inputs, irrespective of the years of experience of the participants.

Quality of Decisions
The participants were asked to compare the quality of decisions they made using 2D drawings with 3D models and VR as a part of the post-task questionnaire. The questions aimed to understand the participant's perspective on the ease of use, the accuracy of their design inputs, and confidence in their inputs suggested using VR for both scenarios. 11 out of 16 participants answered the questions related to the quality of inputs, and the results are shown in Table 8 for both scenarios. The results suggest a strong consensus among the participants about the ease of use of VR for providing accessrelated maintainability inputs. 81% of total participants agreed that VR was easier to use than 2D drawings in the first scenario, and 91% of the participants agreed in the second scenario. This strong consensus was in contrast with the participant's perspective on the ease of use of VR when asked the same question before performing the task as a part of the pre-task questionnaire. Only 38% of the participants felt that VR would be easier to use than 2D plans before performing the task. This substantial increase in participants' perspective is noteworthy, given the short time they spent using VR.
The participants did not show any disagreement with the capability of VR for making more accurate design decisions in comparison to 2D drawings in both scenarios. The participants suggested using VR in conjunction with 2D drawings to make the most accurate decisions in the comments of the post-task questionnaire. In terms of the participants' confidence for using inputs provided with VR for design changes, almost all the participants reported high confidence in their inputs. The participants provided their inputs related to confidence on a Likert scale of 1-5. A confidence rating was established based on the weighted average of the inputs from all participants. The participants had a confidence rating of 4.27 out of 5 in the first scenario. For the second scenario, when the participants interacted with equipment at a height, they were substantially more confident in their inputs (confidence rating of 4.36 out of 5). The ability to see objects at room-scale in VR, without the need for mental visualization of dimensions, gave the participants higher confidence in their inputs.

Questionnaire Analysis
The researchers used pre-task and post-task questionnaires to understand the perceptions of the participants related to VR. The pre-task questionnaires elicited responses related to the participants' background and perceptions about VR usability. The post-task questionnaires assessed their perceptions after using VR. The two questionnaires were used to identify any shift in perception of the participants and gauge VR usability. Sample questions from the pretask and post-task questionnaire answered by the 16 participants are shown in Table 9.
The pre-task questionnaire results suggest that a small fraction of participants (26%) felt that VR could completely replace paper plans and showed a low willingness (31%) to move away from the traditional way of using 2D drawings for design communication. In contrast, after using VR for performing the tasks, a substantial majority (81%) felt that they could effectively make design decisions without the need for 2D drawings. A participant highlighted this observation in the audio recording by suggesting, "I was a bit skeptical initially, but after trying VR, I can see its value for checking space and positioning of objects or equipment for people to be able to work in the area/space provided."  A considerable share of the participants (44%) suggested that they would only use VR instead of 2D drawings for making design decisions. Most participants suggested using VR in combination with 2D drawings for making design decisions in the post-task questionnaire comments. In terms of the experience of using VR, a significant portion of the participants (69%) felt that adjusting to the VR environment was comfortable. Additionally, a substantial majority (75%) felt that the VR experience was consistent with real-world experience, where the interactions with the VR environment seemed natural. A large majority of participants (88%) also suggested that VR would be helpful for inexperienced individuals to make design decisions. The ability to view objects at roomscale in VR can be beneficial for inexperienced individuals to understand and make judgments about a given space without relying on mental visualization and experience.

LIMITATIONS
This study is presented as a proof-of-concept with an experimental approach for investigating the usability of VR for incorporating design inputs from O&M practitioners during maintainability-focused design review sessions.
Given the relatively small sample size of the participants, the results may not be generalizable to the whole industry. However, this research showcases an experimental approach for larger studies, and the significance of the observed data may improve based on a larger sample size.
Additionally, the participants were asked to provide inputs without consideration for other equipment or systems in the mechanical room. Moving a piece of given equipment in a mechanical room may have downstream impacts on the systems connected to the equipment. Future studies can account for the impact of other systems in the room on the inputs provided by the participants.

CONCLUSION
In this study, the usability of VR for O&M practitioners to provide design inputs during maintainability-focused design reviews was evaluated. An experiment was conducted to compare the access-related maintainability inputs provided with VR and traditional 2D drawings and 3D models. The variation in the time required to provide the inputs and the perceived quality of the inputs provided using the two design communication methods was evaluated. The effect of practitioners' years of experience on the access-related inputs provided using VR and the time required for providing those inputs was examined. Additionally, the practitioners' perceptions about VR before and after using VR for the experimental tasks were assessed.
The results suggest no statistically significant difference in the access-related maintainability inputs provided using the two design communication methods. In other words, the practitioners can provide similar access-related inputs using VR and traditional means of 2D drawings and 3D models. However, while using VR, the practitioners were at least 33% faster in providing their inputs. This finding is noteworthy given the practitioners' years of experience and their familiarity with using 2D drawings. All the participants had little to no experience using VR before the experiment and were still faster at providing inputs using VR. The increased efficiency is attributable to the ability of VR to represent objects at room-scale leading to a reduced need for mental visualization of space.
The practitioners' years of experience had little impact on the access-related maintainability inputs or time required to provide those inputs while using VR. There was a strong consensus among the participants on the ease of use of VR and the quality of inputs they could provide using VR. The participants felt highly confident about the inputs provided using VR. Additionally, there was no disagreement among the participants on the capability of VR for providing more accurate inputs compared to 2D drawings and 3D models. The participants noted that using VR in conjunction with traditional design communication methods can lead to the most suitable inputs. Furthermore, participants suggested that inexperienced individuals could benefit the most by using VR to provide design inputs during maintainability-focused design reviews.
This study contributes to the body of knowledge by empirically demonstrating the usability of VR for incorporating access-related maintainability inputs from O&M practitioners over the traditional means of using 2D drawings and 3D models. The findings are limited by the sample size of the participants and the complexity of the design problem. Future research will focus on increasing the complexity of the design problem by integrating other systems' impact and evaluating different use cases for VR adoption.