INTEROPERABLE MIXED REALITY FOR FACILITY MANAGEMENT: A CYBER-PHYSICAL PERSPECTIVE

SUMMARY : The management of building commissioning requires specialists from different organizations and with different skills. Collaboration processes involves several actors and decision-making at different levels. As building commissioning has already been described as systems-of-systems (SoS), the research reported in this paper claims that this definition can be extended into cyber-physical system-of-systems (CPSoS), requiring identification and support of both human-machine and machine-machine interactions in a hybrid environment. These requirements give rise to several challenges, such as capturing information about the existing facility, visualizing, comparing


INTRODUCTION
Facility Management (FM) is an organizational function that integrates people, places, and processes within the built environment, to improve people's quality of life and the productivity of the corporate core business (IFMA, 2022).The focus of FM is the achievement of the facility's performance objectives, which are defined according to overall corporate strategic objectives.Thus, the organizational function of FM consists of a series of schedules and actions operated at the three canonical levels of management: strategic, tactical, and operational.FM involves a broad set of skills.It includes both the technological and operational dimensions of construction and finance, up to the management of contracts and services, and the real-time management of operations inside and outside the facility, including emergency scenarios (Roper andPayant 2014, Rondeau et al, 2012).The organizational function of FM involves a number of actors, engaged in complementary areas of competence, called upon to coordinate synchronously or asynchronously, carrying out their activities both on-site and remotely.The transience of such actors and the operational dynamism, which can scarcely be circumscribed to pre-determined operational patterns, are the factors that generate complex patterns in managing facility processes.For these reasons, FM is an organizational and operational area that challenges the managers' planning and controlling capacity.
Recent technological developments in cyber-physical systems hold the potential to change the focus of the FM paradigm, by introducing a clear reflection on its operational complexity.In fact, the cyber-physical view introduces a systemic perspective to FM, by conceiving actors (whether technical systems or human operators) as entities that dynamically and collectively rearrange their operations to achieve the desired function.Cyber-physical models offer the possibility to qualify such dynamics in terms of adaptivity, opportunistic coupling, emergent qualities, and distributed control (Luhman, 2011, Barišić et al, 2022, Mishra et al, 2022).However, this is not an easy task.It involves rethinking the overall ergonomics of the FM practice, including previously neglected technical dimensions such as Information Technology, Mechatronics and Artificial Intelligence, and the extension of the organizational modelling to the entire universe of human cognitive capabilities.
By introducing the concept of structural coupling (Maturana and Varela, 1980), the cyber-physical perspective brings the reflection on ergonomics back to issues of systemic interoperability through coupling interfaces.This means that, at any level of the organizational hierarchy, whatever the nature of the interacting systems (human or cyber), interoperability concerns issues of process coupling through well-defined operational interfaces.This systematization opens new perspectives.For instance, when mixed reality technologies are used to support site inspections that involve multiple actors and cyber systems, the ergonomics analysis must be extended to the human-human and human-machine interactions, conceived also in terms of information ergonomics, knowledge ergonomics and operational ergonomics.This means that the cognitive dimensions introduced by the technology and the affordance of virtualized interfaces should be analyzed in a situated cognition context (Clancey, 1997, Ataizi, 2012).Situated cognition is the study of human and artificial learning that takes place when some entity is active and present in both physical and virtual worlds.Learning occurs in a situated activity that has social, cultural, and physical contexts that act as a whole (Raikov and Pirani, 2022b).The situated cognition perspective is steadily penetrating the so-called socio-technical perspective of Industry 4.0, which introduces the concepts of integration and interoperability of technical systems conceived as Cyber-Physical systems, including the human-machine interface (Beier et al, 2020, Borgia, 2014, Sony and Naik, 2020, Annaswamy et al, 2023).
The analysis developed in this paper discusses the complexity of the outlined interoperability issues in the FM context through an analysis of a specific FM process, so-called Building Commissioning (BC).Building commissioning (Grondzik, 2009) is a means of ensuring that a building owner gets the expected and deserved facility quality.Although the concept of commissioning is straightforward, the building commissioning process can be complex, it can involve numerous and continuously evolving players, and it may cover the full-time span of the building delivery process (Vaccarini et al, 2022).The Building Commissioning association (BCxA, 2022) identifies three main phases in the BC process: a planning phase, which may contain steps for initial assessment and technical insights, an implementation phase, and a final hand-off or sustaining phase in which operation and maintenance procedures are implemented.
Through an in-depth analysis of two relatively simple technical solutions aimed at improving BC processes, the paper indirectly discusses the issues of FM cyber-physical interoperability.This will be analyzed according to two different dimensions: • Systemic Integration and Interoperability (knowledge and expertise dimensions): as the integration of data and processes takes on new relevance, new issues relating to the physical dimension of data, its lifecycle with respect to that of the ICT components, and their affordance in terms of accessibility, indexing, and computing become significantly more relevant; • Collaborative working and decision-making (social and situated cognitive dimensions): in the cyberphysical perspective, decision-making is itself a hybrid human and artificial, collective, and social process, in which shared artificial arrangements act as enhancers, facilitators and social prostheses in decision-making.Our approach stems from the situated cognition hypothesis, viz.that cognition arises from the purposeful interaction between agents and their context, made of physical objects and social relationships (Clancey, 1993, Clancey, 1997, Roth and Jornet, 2013).By enabling the actors to cooperate in a mixed reality setting, the presented study extends the situated cognition hypothesis to interaction with an augmented environment that combines both physical as well as virtual objects, and where social relationships can be either local (i.e., on the same construction site), or remote, (i.e., geographically displaced but interconnected through the Internet).In this context, knowledge interoperability plays a major role.To keep the decision-making and operational processes flexible and general enough, great attention is placed on the interaction between the processes and the knowledge level as well as the knowledge and the physical levels (Perko, 2021, Raikov andPirani, 2022a).
To sum up, this article aims to discuss and exemplify some of the issues related to the cyber-physical approach to FM.To this end, the article proposes an analysis limited to a single FM process, so-called Building Commissioning (BC).Through an in-depth analysis of some technical solutions to improve BC processes, the article indirectly discusses the problem of cyber-physical reorganization of FM from two complementary and partly overlapping perspectives: on one hand systemic integration and interoperability; on the other hand decision-making and situated cognition.Technically, the paper contributes to the cyber-physical approach in building commissioning by answering the main research question about how it is possible to create an open and shared facility commissioning experience while integrating specialties that are scattered in space and time.In addition, two technical sub-questions will be investigated.The first one concerns what interoperable framework can steer the complex processes involved in facility commissioning to converge toward good decisions.The second one investigates what suitable technology platform could support members of different teams to collaborate in an epistemic framework and to enrich the shared information model progressively.
Section no. 2 includes an analysis of related works.Section no. 3 investigates the concept and implementation of interoperability in facility commissioning.Section no. 4 reports on the technical development that was showcased through the on-site tests described in section no. 5. Conclusions are provided in section no. 6.

RELATED WORKS
The integration of cyber and physical worlds is the cornerstone of Industry 4.0, which can be of great inspiration for the construction industry in general and FM in particular.Industry 4.0 suggests applying principles and technologies from IoT (Internet of Things) to the manufacturing industry, and concerns three block categories from the socio-technical perspective, that have been called human, technology, and organization (Beier et al, 2020).The "human" category refers to effects of and on humans that are directly or indirectly affected by Industry 4.0.The "technology" category refers to effects of or on technical systems or technical concepts that are relevant in the context of Industry 4.0.The "organization" category refers to effects on both processes and the structural organization of an enterprise.The authors of the paper (Beier et al, 2020), have associated ten key features with every category, as a result of how often they bumped into those features while going through a literature review.
Examples of features that are included in all three categories are interconnectedness, communication, and cyberphysical systems.In the human category, the human-machine interaction and collaboration features are very relevant.In the two categories of technology and organization, both the integration and flexibility features are included.These findings suggest that technology is supposed to enable and support communication among humans, machines and products (Monostori, 2014), as well as collaboration and interfaces enabling interaction between humans and machines, such as AR and MR systems (Baur and Wee, 2015) and mobile devices making information management more flexible (Schuh et al, 2015).Interconnectedness is a key feature of the cyberphysical nature of Industry 4.0, which is meant as a network of machines, workers, and systems (Schmidt et al, 2015).
The efforts mentioned above are expected to lead towards the full realization of Industry 4.0, which will foster intra-as well as inter-organization integration in the form of a socio-technical system.More specifically, three types of integration mechanisms are differentiated, which are horizontal, vertical, and end-to-end integration (Sony and Naik, 2020).According to the socio-technical systems theory, a production system can be understood from the complex interactions among the various parts of the system.It would consist of the abstract interaction of people, processes, and technologies, which interact with one another to work out products and services.As a consequence, the introduction of new technology must ponder the behavioural issues caused by sociological changes in the workplace.Socio-technical thinking recommends that such thinking must be systemic, and it calls for consideration of both the social and technical factors within the organization, in the surrounding environment and at the product lifecycle management level.Once again, connectivity, interaction and human-machine cooperation issues require a rethink of relationships between humans and machines.Furthermore, each facet of the integration design must be investigated with the environment existing around the integration system, such as economic, financial, regulatory frameworks and stakeholders.
Getting back to the classification mentioned above, horizontal integration requires cooperation between various organizations in the supply chain in order to create value for customers (Shamim et al, 2016).New innovative business models must be worked out to create a competitive advantage (Ivanov et al, 2016).However, in the construction industry, most of the research concerns data interoperability and follows the information exchange principles required in level 2 as per standard ISO 19650 (ISO, 2018) to go after full multi-party collaboration facing challenges due to interoperability in a BIM environment (Jin et al, 2019).BIM level 2 should allow information to be shared and exchanged across the multiple stages of a construction project.The latest advances in this field mainly concern data exchange among different software tools that pave the way for digital workflows and connect all players involved.The "Association Francophone des Utilisateurs de Linux" (AFUL) defined interoperability as the uninterrupted ability of different systems or product interfaces to communicate freely without restrictions (Aful, 2015).Software and systems usually communicate through common data formats and communication protocols built upon open standards.Within the AEC industry, Green Building XML (gbXML) and Industry Foundation Classes (IFC) are widely used schemas adopted in various domains, which have unique data structures influencing how exported data from BIM models are translated into other software (Panteli et al, 2020).Broader research concerning the interoperability issue involves data sharing, semantics, and ontology, which accommodates topics such as collaboration, cloud computing and interoperability in the same research category (Jin et al, 2019).The data inconsistency and semantic interoperability issues have been studied to enable multi-party collaboration in BIM-driven projects to perform a cost estimate and a quantity take-off (Niknam and Karshenas, 2015).The uptake of BIM is encouraging interdisciplinary collaboration among project team members through visualization and information sharing, as shown in the implementation of the integrated project delivery (IPD) method over the conventional design-bid-build (DBB) one.Interoperability gaps were found between the semantic levels of BIM-based data forms and GIS, too, including spatial data, temporal data, and informational data.This implies that improved architecture for supporting such data interoperability should be developed (Panteli et al, 2020, Wong et al, 2018).The evolution of interoperability is supposed to lead to a BIM level 3 as per ISO 19650, which has not been defined yet by the standard, but is supposed to enable full collaboration and integration in a cloud-based environment, including asset life cycle management (Panteli et al, 2020).
Indeed, asset lifecycle management accounts for 85% of the entire life cycle cost of an asset (Wong et al, 2018).
In this field, the other type of integration that is called end-to-end integration is required.The product is supposed to be smart and capable of storing information about itself, in terms of manufacturing, preceding operation, current state, assembly information, maintenance, etc. (Foidl et al, 2016).The main goal of this approach is to add value across all stages of the product lifecycle, and related technology must focus on cradle-to-grave thinking for products and services (Sony and Naik, 2020).Some concerns in this field have been raised in relation to the purpose of construction asset maintenance, mainly due to the elevated operation and maintenance costs, which represent between 50% and 70% of the total annual facility operating costs.This was a first step towards the awareness that building maintenance decisions require the analysis and integration of different types of information and knowledge, such as maintenance records, work orders, causes, and knock-on effects of failures.As a consequence, a well-integrated data system is becoming important for FM companies to manage the huge amount of staff and facilities data, and to accommodate the constant changes occurring in the facility.A couple of urgent needs have been stressed by literature in FM.The first one concerns data interoperability between BIM technologies and computer-aided FM (CAFM) technologies adopted in the operational stage (Wong et al, 2018).The second one was raised as a direct consequence of the automation of data capture, maintenance and as-built model updating, which generates improper data management caused by the lack of standardized frameworks for the implementation of BIM in existing buildings (Panteli et al, 2020).Unfortunately, the former suffers the limitations of IFC and COBie in delivering some of the data entities, types and parameters required for FM, which compels the integration of omitted properties and relationships related to FM on a shared ontology, so as to provide a sound semantic base.
In other words, the dynamically extensible schema provided by IFC provides the support at data level but not at the semantic level (Sacks et al, 2018).An attempt to carry out a well-executed interoperability plan for exchanging data between BIM and an FM tool chose the IFC format as the source and one of the commercial CAFM tools as the destination (Rogage and Greenwood, 2020).The tests successfully resolved technical interoperability, so that data can be transferred between applications, despite the fact that accuracy and local organization still remain an open challenge.The results prompted the use of a hybrid solution for data transfer, on a case-by-case basis, between IFC and an operation and maintenance tool.This allowed automated mapping between corresponding data fields where the IFC model mapping allowed it, whereas manual mapping, i.e. establishing manually the semantic correspondence between fields, was adopted where it did not.To the best of the authors' knowledge, no product lifecycle management has been applied in construction that conforms to the definition of end-to-end integration as reported above.
In addition, the full realization of the cyber-physical approach in FM and building commissioning requires that even vertical integration be implemented.In other words, this type of integration is the creation of flexible and reconfigurable manufacturing systems within an organization through digitalization.As such, the department subgoals will also have to be strategically aligned with the main goal through a vertical integration strategy (Nauta et al, 2002).This creates a need for smart infrastructures, which include technology for data collection, analysis, decision-making, self-regulation, networking, reporting, controlling, organizing, etc. (Sony and Naik, 2020).In this respect, a current trend for the post-construction phase is the specialty field of renovation projects.Adopting digital approaches combined with BIM could facilitate the execution of relevant tasks, modernize, and make the whole renovation project more efficient.
To this purpose, Ammari and Hammad (2014) and Gheisari et al (2014) developed, respectively, an integrative BIM-based system where real-time data can be collected to support the inspection of building information through a mixed reality (MR) application, and a BIM tool integrated with mobile augmented reality technology to allow facility managers to experience an intuitive natural interaction with their mobile interfaces and access required information efficiently.Several definitions of Mixed Reality have been suggested in the scientific literature.The one complying with the role given in this paper is the MR environment where virtual objects are rendered so that they are indistinguishable from the physical world, and users can interact with both virtual and real objects in realtime, and these objects can interact with each other (Flaviàn et al, 2019).Such "environment awareness" implies that not only virtual objects can act in the real environment, but real objects can also modify the virtual environment.Although MR has already been widely used in architecture as a means to overcome the problems of visualization, limits have been encountered in the process of uploading all data relevant to maintenance from BIM models into MR devices (Kim et al, 2019).The implementation of projects in FM still faces coordination and collaboration challenges.In order to be able to collect and share FM-enabled data seamlessly throughout the project life cycle, the owner and FM team, designers, general contractors, subcontractors, and BIM consultants must all collaborate at a high level (Pishdad-Bozorgi et al, 2018).Some drawbacks caused by the current practice are that designers and contractors too often duplicate modelling efforts and lose efficiency; in addition, the general contractor must often use sub-models that can be generated by subcontractors only.
The construction sector is currently borrowing and adapting System-of-systems (SoS) approaches from Industry 4.0 that pursue model-based system engineering, in order not to reinvent the wheel, but rather to start from consolidated methodological experience (Axelsson et al, 2019).Construction and renovation projects, though, are complex socio-technical problems that can only be solved through effective teamwork and cooperation.Thus, sufficient consideration for the deep systemic intertwining between human and artificial agents is a key factor in introducing true innovation to the FM discipline.The cyber-physical production systems emerged as SoS in recent years (Yilma et al, 2021), and this motivated a new definition of Cyber-Physical System-of-Systems (CPSoS) that was recently the object of scrutiny by researchers and practitioners (Engell et al, 2015, Bondavalli et al, 2016, Ferrer et al, 2018, Bonci et al, 2018a, Bonci et al, 2018b).In the following sections, we embrace this perspective and define the building commissioning discipline as CPSoS.
Finally, the cyber-physical approach to FM presented in this paper abides by the situated cognition hypothesis.
Then, new approaches in CPS and digital-human interaction must consider the growing interest in pragmatic constructivist approaches that consider cognition and cognitive loads as the result of a circular and continuous active permeation between the human, the artificial tools, and the environment (Raikov and Pirani, 2022b).In the AEC sector, most of the work is performed on-site and humans are deeply intertwined with the physicality of the workplace.Nevertheless, several researchers are exploring the added value brought by augmented and mixed reality solutions to the AEC industry both as learning tools as well as frameworks for collaborative working (Akanmu et al, 2020, Ayer et al, 2014, Delgato et al, 2020).

CYBER-PHYSICAL INTEROPERABILITY IN FM
Referring to the definitions of Bondavalli et al ( 2016), building commissioning can be described as a Cyber-Physical System-of-systems (CPSoS).The set of actors involved in a building commissioning context is depicted in Figure 1, where different human actors using different user interfaces interact with the building elements and a set of autonomous Technical Systems (TS) of the building (e.g., Heating, Ventilation and Air Conditioning, Lighting, Fire safety, etc…) by means of a set of services.The CPS concept envisions the introduction of artificial intelligence that in some cases allows automation to arrive at the agent level, implementing proactive-interactive entities that participate as actors in the process.Even if the constituent systems may, or may not have been designed to work together, at a certain point, and through some means, they are coupled in the real process, so that they can interoperate.Therefore, many of the challenges associated with SoS engineering in building commissioning concern achieving appropriate interoperability to ensure useful emergent properties (de C Henshaw, 2016).Indeed, this is the innovation view that the CPSoS approach brings to FM: in order to achieve the desired performance levels, systemic interactions must be identified and reconsidered for all operational scenarios.The main consequence of this approach is that key processes, actors and main interactions (i.e.their interfaces) must be identified and modelled.The building commissioning collaboration workflow for the design phase, presented in (Vaccarini et al, 2022) and depicted in Figure 2 as a building process modelling (BPMN) diagram, was assumed as a reference in the remainder of this paper.In Figure 2, the stakeholders fall into three main categories: the client, the FM office, and the design team.The former is usually the owner and is the person responsible for expressing the project needs and the available budget.Also, they have the final word when it comes to accepting or rejecting an appointed design solution among several candidates.The FM office usually denotes a group of people managing the overall building commissioning project and coordinating several technical specialists on behalf of the client.
The FM office interacts and cooperates with the design team in order to help the latter come up with candidate design solutions that may satisfy all the needs identified by the client, with the available resources.The process of selecting the best available candidate scenario may require several iterations between the FM office and the design team, while choosing the final appointed solution may require one or more iterations, if relevant changes in the requirements are introduced.A key aspect for ensuring a successful accomplishment of the building commissioning process is the prior identification of any flaws in the candidate design models, possibly blocking them before they reach the next stages.It is well known that fixing design flaws during production is more expensive and less effective.This implies the introduction of additional iterations in the building commissioning process, with the aim of identifying and solving such flaws as early as possible.On the other hand, each new iteration delays the project schedule, thus potentially increasing the cost of the design phase.It is not trivial to identify the right number of iterations required by the entire building commissioning process.For this reason, ensuring effective and seamless interoperability of all the cyber-physical components along the entire building commissioning process is of paramount importance.
In the Building Commissioning process scenario in Figure 2, interoperability is a clear effect of a process over data played by agents (be they human or artificial).To support such a deep interaction between humans and machines in whatever combination it may occur, a methodological twist is required.A novel interpretation of the actions conducted by humans or artificial agents from the whole stack of operations in a vertical direction (knowledge extraction and information processing from data), and throughout the business processing in a horizontal direction (operational collaboration and enrichment through collaboration) must be developed.
The approach developed in this paper applies the handy classification of Nilsson (2019), which recognizes four types of interoperability that are inherently hierarchical: • information interoperability is the ability of systems to exchange and use information and it is a requirement for any of them; • semantic interoperability is a step above information interoperability as it ensures that designers and engineers reach a shared understanding and level of knowledge about the meaning of each piece of information and how it relates to the others; • dynamic interoperability is the ability to achieve interoperability at run-time, in real operating conditions (real-time use and adaptation), and is a prerequisite for operational interoperability in complex environments; operational interoperability is the ability of systems to work together to achieve some shared goal (purposeful goal attainment).The systemic integration and interoperability requirement is met by implementing the first three types of integration of Nilsson's hierarchy, while the collaborative working and decision-making requirement corresponds to reaching the operational interoperability level.
The information and semantic side of interoperability are achieved if there is an intersection between the outcomes of the commissioning process.The existence of this intersection depends on the convergence of the observation over data.Two matters are promptly implied.The first one is the hierarchy between data, information, and knowledge, which must be clarified.The second one is the role of the observer in a communication channel of information.
Information Interoperability is the one that transforms data into knowledge.According to Shannon's model, information is the data that generates a decrease in an agent's uncertainty about a particular state of affairs.Hence, information does not have proper existence without the active intervention of intelligent agents.It can also be expressed by the famous statement from H. Maturana: "Everything is said by an observer" (Maturana and Varela, 1980).The agent's active role recurs from information to knowledge.Knowledge refers to the codified experience of agents.The experience is the source of the information for solving problems.By "codified" we mean that the knowledge has been formulated, recorded, and made ready for use (Stefik, 2014).
In this mechanism resides the second argument previously mentioned, in which the observer plays a fundamental and necessary role in the nature and existence of information and knowledge.
However, it also states that the presence of data is necessary for two observers to share information and knowledge in a purposeful way.In Figure 3 this concept is depicted as a set diagram.Information and semantic interoperability can emerge through information processing and interoperation when data is shared between two observers.Note also that data is the only necessarily invariant entity for any kind of interoperation to happen.Dynamic interoperability affects both lower levels.At information level, the dynamic nature of the information systems that manage a building's data and the unpredictability of their technical evolution over long periods, rise issues of data consistency and access.In facility commissioning dynamic interoperability at information level can be reached by either assuming that the building to be commissioned contains physical objects that embed data into themselves or by attaching smart devices capable of carrying pieces of data to existing building elements.An effective way to persistently embed data into building elements is to install embedded tags, and this plunges the Internet of Things technologies into the domain of facility commissioning.This is in line with the "intelligent product" concept that has been well-developed for at least 20 years in an industrial context (Derigent, 2022).The data is kept tightly coupled with the physical object ready to be retrieved and used for interoperability purposes along the many processes crossed during the entire life cycle of the building itself.
As an example, we could think of the life cycle of information relating to any technical element, e.g., a solar panel (Figure 4).The product information originates within the manufacturing company's information system and it is organized according to the purposes of the company's operations.It includes, for example, both aspects relating to technical specifications and aspects relating to the production process.This information is accessed through the internal channels of the company's information system.When, as consequence of a design phase, the product is purchased and installed in a building, it stops being a product and takes on a new role as a building component.Thus, we can think of the solar panel object as leaving the production process in order to enter the building construction process.This transition implies a change in the nature of the information level pertaining to the physical object.In addition to the performance aspects, information related to the building construction process is added, while some information relating to the solar panel production process may disappear.In the operation phase, this information changes again, for example, maintenance data is added.At the end of life, when dismantling the component, information about the correct ways of waste disposal or recycling eventually written at the production phase is recovered.
The semantic level requires adaptation, possibly at runtime, of agents' knowledge schemas to provide interpretations of states of affairs that evolve over time.This is indeed the human level of interoperability that can hardly be reached by full autonomous systems.In modeling a real CPSoS, therefore, the possibility of enhancing decision-making by means of situated information, as well as of complementing semantic alignment limits by the cyber component by means of any suppletive actions by the human component must be considered.Dynamic interoperability at the semantic level can be made effective if data is pushed as far as possible down into the physical and local dimensions, near the objects that constitute the working environment.Primary data are embedded into the physical dimension of the cyber-physical systems.Embedded data are the seeds that enable the growth of information and knowledge achieved by the knowledge structures that are stored on the "cyber" or "bits" side.The outcome of this enrichment and extraction produces situated and contextual information that is at the basis of the humans' situated cognition processes.In other words, enhancing contextual awareness, while at the same time avoiding physical information access and communication barriers, is the basis of effective decisionmaking processes.Section 5 provides a detailed example of how this could be achieved in a real operating condition.
The last type of interoperability is operational interoperability that exploits processes and decision procedures shared at a dynamic interoperability level in order to enable human and automated agents to cooperate towards their common goals.
The autonomous subsystems of SoS can exchange information in two different manners.The first is a typical communication over some channel with a transmission between two entities.The second, is a bio-inspired paradigm of information communication that uses the environment as the channel.In this case the receiving entity actively scavenges information from data embedded in the environment of the action.The data was left in the environment by the sender of the information.In the literature this communication and information processing is defined as stigmergic (Bondavalli et al, 2016, Marsh andOnof, 2008).When dealing with the problem of a collective decision working towards a common goal and acting asynchronously at different times, the stigmergic channels can help to close missing links in a control loop and can have a decisive influence on the system-level behaviour of an SoS and the appearance of emergent phenomena.
In the context of BC, the kind of purposeful collaboration that one wishes to achieve is the collective analysis of candidate solutions.Following the situated cognition approach, our hypothesis is that this goal can only be reached by creating a Cyber-Physical System-of-Systems, representing the hybrid reality made up of the human and digital artifacts that pertain to the design options under scrutiny.Human and artefacts in hybrid reality can activate stigmergic processes which consist of spreading data as information seeds in their shared environment (physical or digital).The mutual processing of this information usually grows into knowledge that depends on the situated actions that the entities perform in the shared environment.While for humans it is quite natural to localize themselves in the construction site of the building to be commissioned, for artificial agents and tools this is not a trivial task, especially because the working conditions of the applicative domain most of the time assume that the operators are working indoors, without direct access to GPS satellites that may localize them as observers.Thus, there are two main purposes at the core of the Cyber-Physical perspective of an interoperable framework for facility commissioning.First, operators must immerse themselves in the hybrid reality in order to fully understand the implications and requirements of any candidate design options.Secondly, in order for the former to happen, humans and machines must localize themselves in real-time in a shared space containing both the digital and human building components pertaining to the building commissioning task.
To this aim, a clear epistemic structure must be developed, i.e. stable informative structures must be added to the operators' environment in order to reduce the cognitive complexity of their tasks, e.g.maximize search, inference, memory load, and so on (Chandrasekharan and Stewart, 2007).In the case of building commissioning, BIM and its enhanced informational views that are achieved by interoperating with XR (in particular, mixed reality tools), IoT (e.g.mobile devices), and small sensors (e.g.RFID) can be used.The interoperation between these heterogeneous sets of tools allows humans and digital actors to localize themselves inside and around the facility to be commissioned, as well as to record and exchange rich tokens of situated information that in turn contribute to building a contextualized knowledge base.The richness, as usually intended for example in a database and knowledge creation context, is given by high interconnection between different attributes and contents, rather than big amounts of tabular data with fixed key-value properties.In this sense, the BIM structure tends to be rich and multidimensional, proving to be a perfect token of knowledge and information to be shared by human and digital agents while collaborating.This is already evident when using the most basic representations for BIM (e.g.IFC files), but it is even more evident when further layers of information are added for interoperability purposes (e.g., OWL ontologies and linked datasets).Once rendered interoperable, such tokens of information constitute a valuable epistemic framework within which human actors can cooperate, working towards a purposeful and collaborative analysis of candidate design options.

THE TECHNOLOGICAL FRAMEWORK
As explained in the previous section, the Cyber-Physical perspective on FM is based on two major requirements, the first one pertains to the realization of a framework allowing full systemic integration and interoperability while the second one is focused on enabling effective collaborative working and decision-making.It has also been explained that while the former requirement can be realized by implementing information, semantic, and dynamic interoperability, the latter can be conceived as operational interoperability, following Nilsson's classification.In the next sections, we first present a data model at the core of the information and semantic types of interoperability.Then, we present the technological stack supporting all the tasks along the full stack of Nilsson's interoperability levels.

A data model for information and semantic interoperability
The information and semantic interoperability data model is the core element of a common data environment (CDE) where the complex cyber-physical processes involved in a global facility commissioning task can be represented and carried out.A common methodology for effectively representing knowledge derived from pieces of information is to structure a knowledge graph of some sort whose vertices are the entities of the domain under analysis and whose edges are the meaningful relationships among the latter.The meanings behind an entity are indeed encoded as the paths in the knowledge graphs departing from that entity towards the others.Knowledge graphs can be encoded using two types of technologies: on one side we have those where the graph structure is rigid and determined apriori, during a stage called semantic negotiation or knowledge elicitation, possibly involving several actors and organizations.Examples of such technologies are relational databases and RESTful application programming interfaces (APIs).On the other hand, we have technologies where the graph structure can evolve during the system lifetime, allowing one to discover and represent arbitrary relationships between entities and the meaning of the stored entities may evolve during time because the structure of the allowed relationships are not determined apriori.Examples of such technologies are graph databases or RDF triplestores.
The current implementation of the prototype used in the tests reported in this paper is based on ArangoDB, a stateof-the-art graph database engine to represent the system knowledge.A fragment of the data model of the database is provided as a UML entity-relationship diagram and depicted in Figure 5.In this respect, entity types are mapped onto so-called ArangoDB document collections, while relationships among entity types corresponds to so-called edge collections in ArangoDB.Instances of entities and relationships are represented as documents in the corresponding collections.The GLTF file contains a 3D object model while the JSON file contains a tree of objects, together with their relevant properties, a link between the globally unique identifier (GUID) of the object as it appears in the IFC file and an identifier of the 3D model contained in the GLTF file.Instances of Annotations, instead, wrap audio files recording the comments of On-Site Users expressing their opinions about specific portions of the IFC model they are assessing.In an Annotation, the IFC object is referred to through its GUID, the latter being detected when the user points to a virtual object in the MR Headset App.The Physical Objects entity type is a registry of objects that are present in the facility to be managed and information has been embedded into them, e.g. by means of NFC tags.An instance of such physical objects is identified by the serial code of the NFC tag they use, the name of the supplier and the product itself, but it also carries a property set that may differ for each such physical object.Such a property set could be represented by a JSON structure, i.e. a hierarchical structure mapping keys to values that are easily serialized to a text string and are nowadays used as a standard exchange format for datasets with arbitrary structures.Instances of Building Element type correspond one-to-one to instances of physical objects and link each of the latter to a project, assigning to each physical object a list of IFC GUIDs with which such physical objects are represented in the available design options.The main difference between instances of Physical Object type and instances of Building Element type is that the former involves information carried by the object themselves, e.g.embedded in NFC tags, while the latter is the representation of the former in the graph database underlying the technological solution presented.Finally, instances of NFC Reading type store the serial identifier of NFC tags that are scanned using the Mobile NFC Scanner, as well as the timestamp of the scan operation.When sensed by the Mobile NFC Scanner, such serial identifiers are compared against the known serial codes of instances of Building Element type, and if they match, the MR App can deduce the position of the observer relative to the GUID of the sensed NFC Tag.
The aforementioned data model supports the embodiment of information through the Physical Object, Building Element and NFC Reading entity types.By means of them, a link between the real world and the virtual world is established, creating the hybrid reality shared by human and digital actors: while the physical objects in the real world describe the facility as-is, in the current operating state, the virtual objects describe how to reach the target profile of the overall building commissioning task.An NFC tag, in principle, can be attached to any physical object acting as a building component in the facility, and in this way, its entire life cycle must be traced along the recommissioning task (doors, windows, walls…).Once a real-world object can be localized, either indoor or outdoor, its coordinates can be associated with the attached NFC tag and each NFC Reading entity type contributes to localizing the user while interacting through the Mixed Reality tools.The purpose of this workflow is to enhance the user's spatio-temporal and logical perceptions through visualization and in some cases even mediated interaction of not-yet-existing objects (i.e.holograms) with already-existing objects (i.e. the real world) in the facility to be commissioned.
The workflow thus realizes an environment surrounding the user with objects that are already present in the building or that will be present there in the future.Following the situated cognition hypothesis, such purposeful interaction between the user and the surrounding environment is key to reaching a deep understanding (cognition) of the actions required in order to realize the intended facility recommissioning goal, and of their implications for the owners and the users of the building.To be noticed that any NFC tags located in the building can be used to reach a number of other goals of interest for Facility Managers, other than geo-localize real-world as well as virtual-world objects, as represented in Figure 6.This data model is able to support information and semantic interoperability.As an example, we may consider a wall of the facility.In the case of a building recommissioning project, any wall may be part of the structure of the building, they may undergo different construction processes since some of them may be bearing walls whereas other ones may be partitions.Also, the number of layers of a wall may affect energy performance.This type of information can be recorded in the construction company's databases, but it may also be stored in some NFC tags attached to the walls during their production.If a wall is prefabricated, the NFC tag can be installed by the manufacturer.The same NFC tag can be included in the IFC model of the building in its correct location.In this scenario, when a physical object is sensed through the attached NFC tag, an instance of NFC Reading (see the class "NFC Reading" in Figure 5) is created in the database and a visual marker is displayed on the screen of the Mobile NFC Scanner.Next, the MR App installed in the headset looks up the identifier of the Building Element corresponding to the most recent NFC Reading in the database and uses that identifier in order to rotate the model such that the hologram with the given identifier is found in the position where the tag is displayed.Repeating this procedure twice can finalize the alignment process.This example is representative of how interoperability issues can be overcome by "planting" seeds of information in the environment.In this case, the only piece of information that needs to be stored in the NFC tag is an identifier of the tag itself.Through the tag, it is possible to look up the GUID of the building product embedding it in the IFC file and the exact position (i.e.3D coordinates) of the NFC tag in the IFC file.To be noticed that NFC tags embedded in the construction element can be used to catch its 3D coordinates and communicate them to the MR App by using a QR-code recognized by the MR App in the headset.
The same NFC tag can be used to store useful pieces of information for different purposes: construction, management, recommissioning, and observer localization, as depicted in Figure 6.This information is stored in the database as instances of the class "Annotation" of the data model (see Figure 5).In this way, NFC tags become actual stigmergic channels embedding information in the environment and easing interoperability and data exchange among the toolsets of remote and on-site users.To be noticed that in this case the duplication of data is not redundant.Indeed, the two contents have a completely different lifecycle: the data entered in the NFC tag shares the lifecycle of the technical element; whereas the data contained in the database shares the lifecycle of the information system.In the broader sense of interoperability defined in this paper, MR is the technology that links the two data.

Towards dynamic and operational interoperability: architecture and technology
The architecture of the technological platform prototype is depicted as a UML diagram in Figure 7.The Remote Users have at their disposal tools to be used on a desktop computer, while On-Site Users can work on the field using wearable devices such as an MR headset and mobile devices (e.g.tablets, smartphones …).Being constantly connected to the Internet, such tools can synchronize among themselves using JSON web services implementing a RESTful API.Through them, the tools have access to the most updated version of the BIM models for a given facility and can update it with information coming from the field.The components are deployed onto separate containers, thus realizing a microservice-based architecture.Such an architectural pattern has several known advantages.First, being separate, every container can secure the managed data from intruders or non-authorized users.Secondly, the architecture itself is more scalable and cloud-oriented, enabling the rapid migration of services from a single Virtual Private Server to a geographically distributed pool of container pods.Thirdly, each service implementation can be replaced and updated, as long as the service interface is respected by the new implementation.Fourthly, containerized services ensure higher availability to the clients, since they can be easily restarted even in the presence of critical errors, thus reducing the disconnection time and allowing the clients to reach the so-called "five-nines" availability target, i.e. to be up and responding for 99.999% of the daytime.Such features enable the implemented technology to transition from being a prototype in a controlled test environment to being an actual business service in a production environment.These features (microservices, containerization, and high availability) are key enablers for operational interoperability, supporting an effective collaboration among teams that cooperate and enrich BIM models along their facility commissioning tasks.
Several services are used in the architecture: Storage, a Database, a Mobile RFID Scanner, and an IFC-to-3D conversion service.The Storage is responsible for hosting files (e.g.IFC, GLTF, JSON …) as they are uploaded by the user or generated by other services in the architecture.The Database hosts the entities and the relationships described in Section 4.1.The Mobile RFID Scanner uses Web NFC API to enable every mobile device with an NFC antenna (e.g.smartphones, tablets,…) to become an RFID Scanner that can be used to rotate and align the 3D holograms appearing in the MR App used by the On-Site User.The IFC-to-3D conversion service wraps the xbim toolkit library to provide an online conversion service taking an IFC file as input and returning a pair of files, viz. a GLTF describing the 3D geometry of objects present in the IFC files, and a JSON tree describing the relationships among such objects as well as linking the 3D object identifier with the original object GUID as given in the IFC file.
The MR App is developed using Unity3D 2019 with Microsoft Visual Studio 2019 and Mixed Reality Toolkit (MRTK).The app was then deployed on Microsoft Hololens, allowing the On-Site User to wear it and see 3D holograms superimposed on the real building.A requirement of the app is that a Wi-Fi network should allow the Hololens to be continuously connected to the Internet.In this way, the MR App can act as a client using the provided services to download the 3D model of the design options to be assessed on-site, as well as existing annotations added during previous on-site surveys, if any.After wearing the MR headset and logging in using a username and password, the On-Site User can select the current facility commissioning project and the design option to be assessed.After selecting the design option, the 3D model previously generated by the IFC-to-3D conversion service is downloaded in the form of a GLTF file and rendered on the MR headset.The JSON file is downloaded as well in order to allow the app to list objects or classes of objects to be hidden or displayed (e.g.walls, windows, furniture…).
This technological platform can be used to manage facility commissioning.First of all, the FM office creates a recommissioning project in the platform itself, collecting the requirements elicited by analysing the project needs as expressed by the client.The recommissioning project is then shared with the design team, which in turn is required to produce and upload design options that should meet the given (technical as well as non-technical) requirements.Then, the design team usually works with a wide range of BIM authoring tools to produce several design options.The platform is compatible with all the BIM authoring tools that can export the produced design options as IFC files.For every IFC file, a new design option can be created in the platform, within the given recommissioning project, and the IFC file is uploaded into the latest produced design option.Once the IFC dataset is imported, the platform triggers an automatic conversion service extracting two different pieces of information from the IFC file: on one hand, a 3D model representation is saved as a GLTF object, while on the other hand, a JSON document represents the graph of objects contained in the IFC model (walls, windows, doors, furniture, ...) together with their properties and relevant IFC metadata (e.g. the GUID identifying each object).After this stage, the FM office or any appointed technical specialist can go on-site, wear the MR headset and in turn exploit Wi-Fi and Internet connectivity to select the design option that they want to assess directly on-site.
As soon as the 3D model has been downloaded, and before displaying it, the MR App waits for the On-Site User to scan RFID tags using the Mobile RFID Scanner.The scan operation will display a target image on the Mobile RFID Scanner display, and such a target is recognized by the MR App, which in turn connects to the remote web service to get the GUID of the latest scanned RFID tag.By repeating this procedure twice, the MR App uses an AR engine, in our case the VuforiaTM engine, to store the coordinates of the 3D objects corresponding to the last two scanned GUIDs, to compare them with the position of the MR headset relative to such objects, and to compute the rotation angle able to align the model with respect to the position of the observer.Through this process, the platform aligns the virtual objects with respect to the real-world objects around the On-Site User, enabling the desired situated cognitive interaction between the user and the platform.Then, the On-Site user can point to virtual objects and enrich the BIM by providing additional information through their voice, through state-of-the-art speech recognition and natural language processing (NLP) techniques.
Summarizing, in the present implementation we have developed and tested on real buildings features such as registration of holograms with the reality, localization of the observer in the hybrid reality visualization and analysis of candidate design options using mixed reality, human-machine interfaces that use gestures and voice, and desktop user interface for remote operation on the platform.These features are key elements of the proposed cyber-physical approach allowing all the actors to independently and asynchronously operate on the building to achieve a common building commissioning or FM task goal.

The pilot building
The implementation of the interoperability levels mentioned in Section 3 has been showcased through experimental tests concerning the demonstration of Mixed Reality technology that has been carried out within the EU-funded project ENCORE "ENergy aware BIM Cloud Platform in a COst-effective Building REnovation Context" (CORDIS, 2023).It is a three-level demonstrator house, which is located in Extremadura (Spain) and is managed by Junta de Extremadura (Figure 8a).It is an existing building, and the performed demonstration simulated the management of a building commissioning project.Within the development of the ENCORE project, a design team worked out two renovation options for the building that would be assessed on-site by means of an MR App (Carbonari et al, 2022a).Both renovation options were developed in order to improve the building's energy performance and they differed in terms of types and number of designed renovation actions.More specifically, the first option included the installation of a new air supply system, an internal ceiling, a new partition, and some internal probes to track air quality inside the building.The second renovation option included the installation of electric window openers, replacement of sun shading systems, resizing of some widows on the north-facing façade, and installation of an external coating.Figure 8b shows the IFC model of one of the renovation options developed by the design team.In this view, a portion of the envelope and some building components have been hidden to display the positions of two of the NFC tags (as defined in the data model in Figure 5) included in the virtual model, which have been marked inside a red box on the same figure.
For the purpose of the tests reported in this paper, the data embedded in every NFC tag includes the NFC tag's identifier.As described in subsection 4.1, this number links the real tag with its virtual representation in the IFC model and it brings further implicit information about their positions in the virtual model and the corresponding locations in the physical facility.Thus, data stored in the virtual model about any building components can be associated with any physical objects in the field of view of the operator wearing the MR headset and displayed aside.This carries data to existing building components and facilitates dynamic interoperability at the information level.One of the advantages is that any updates of data in the virtual model (e.g. from the planning through the implementation and hands-off phases) would be immediately mirrored in the physical representation.In addition, NFC tags can embed some static data, e.g.name of the manufacturer and product ID number at the date of production, if this is deemed useful by the management team.
The tests that will be described in the next two sub-sections were carried out by eight volunteers.They were required to have a technical degree, in order to simulate the behaviour of experts assessing a BC solution on-site.However, another basic requirement was that none of them have expertise on the use of MR technology to be tested.They were all employed at the Junta de Extremadura institution, which is the institution supplying the testing facility (Figure 8a).Three of them were younger than 30 years old, while the remaining five were aged between 30 and 50.They all had a Master's Degree.Four of them worked as either an architect or an energy expert; two of them as a building engineer; one as a developer of technical systems for buildings; the remaining one worked as a developer of research projects.Due to that, a training session was led by the researchers from Università Politecnica delle Marche prior to the execution of the experimental session (Carbonari et al, 2022a, Carbonari et al, 2022b), in order to enable them to use the MR App developed for this purpose.Then, every volunteer was asked to perform two surveys.The first survey started with the alignment of the model through the use of tags and under the supervision of an expert researcher from Università Politecnica delle Marche, as is reported in Section 5.2.In the second phase of the first survey, they were required to autonomously investigate every renovation action included in the first BC design solution (see subsection 5.3), i.e. (i) installation of a mechanical air supply system; (ii) installation of a new ceiling; (iii) installation of a new partition on the ground and first floors of north-facing rooms; (iv) installation of an indoor air quality system.In the last phase they assessed each of the aforementioned actions by recording their feedback through the audio-recording feature of the MR App.The second session of the survey required no model alignment because the user had already been located in the building after the first session.They started with the investigation of the second BC design solution, including three renovation actions, i.e. (i) installation of electronic window openers and sun shading systems on the south-facing envelope; (ii) resizing of north-facing windows; (iii) installation of external coating on the northand south-facing envelopes.Then, the second and last phase concerned assessing each of the aforementioned actions by recording their feedback through the audio-recording feature of the MR App.

Systemic integration and interoperability
In this building commissioning process, a member of the design team was in charge of uploading the renovation option solutions to the web server, by means of the web GUI depicted in Figure 9, which implements some of the processes shown in Figure 2. He/she is one of the instances of the "user" entity depicted in the model of Figure 5 and can be categorized as a human actor that makes the digital artifacts that pertain to the design actions under scrutiny available, which is the precondition to enable the realization of an immersive collaborative working environment.In the same figure, a snapshot is shown of the first-person view of the on-site user while downloading one of the virtual models available on the platform.Thanks to the MR App, the virtual model of one of the design options along with the information embedded or linked with it, was displayed on-site and made manageable by any user or observer.So far, the virtual model was not aligned with the real building and the information connected with the virtual model had no link with the real environment.
For this reason, once the on-site user locates at least two of the NFC tags installed in the building like on the wall in the background of Figure 10a, he/she can create one or more instances of the NFC Reading entity, which stores the serial identifiers of the scanned NFC tags in the web server through the Mobile App.The result of this action is shown in Figure 10b, where the Mobile App displays a picture on its screen as soon as the reading has been accomplished.As a result, the MR App that runs in MicrosoftTM HoloLens receives an acknowledgment from the web server and gets the position of every read tag stored in the virtual model.Then, it localizes the tag in the real building by exploiting the capabilities of AR engines integrated into the App, which in this case is called Vuforia.Finally, it performs the alignment procedure (Ozlu, 2018, Macario et al, 2022) that comes out with the virtual model aligned over the real building; and the virtual tag on the real tag as a consequence, which is showcased by the red boxes of the first-person view visible in Figure 10c.The final result was sensed by the on-site user as shown by the first-person view of Figure 10d.In this view, the virtual model is well superimposed over the real facility.
In other words, a link between the virtual model and the real facility has been made.As a consequence, the embodiment of information was accomplished, and the observer was integrated within the hybrid environment made of both virtual and real instances.From now on, an immersive experience is in progress, thanks to the link enabled by the presence of virtual and real NFC tags.Although NFC tags may be used to store technical data about the building component, the main feature enabled by their coupling with their respective building components and their positions in the virtual model, is the alignment of the virtual model and the triggering of the assessment phase of the displayed renovation option.

Collaborative working and decision-making
The environment created through the processes described in section 5.2 triggers an on-site experience that generates a collaborative working environment, eventually supporting decision-making.More specifically, an observer can interact with both virtual and real components, retrieve related information, and be aware of the relationships established among those entities.As long as the on-site operator selects and queries virtual components (ref. Figure 11a) and combines such information with what he/she can sense from the real building, a situated cognition in a hybrid reality scenario is taking place.Technically, the existing information can be displayed on-site using the virtual menu and windows popping up while navigating across the virtual model.Figure 11b depicts an example of such a visualization, which brings the available information of the selected component out.
In addition, the analyses performed at this stage to assess the design option subject of the building commissioning process, take advantage of the coupling between virtual and real parts.This is represented by the activity diagram included in Figure 12 by means of the "Real building" node, which is an external node in the overall framework and the sub-systems it is made of (i.e.web server, remote use, HoloLens, on-site user, mobile App).Nevertheless, it exploits the link established with the virtual model as already detailed in section 5.2.At this stage, an on-site user can participate in the assessment process, thanks to the recording feature, that is made available by the MR App, as shown in Figure 13a, which is a snapshot of the first-person view of the observer right after he/she has selected a component and annotated it through an audio file.The on-site user can also play and be aware of other audio files previously associated with any components.In other words, the observer can embed further data useful for decision-making in the environment and link them to any specific component, in fact keeping related information up-to-date and accessible through several means, i.e. both on-site through wearable MR tools and tags, and remotely via the Web GUI.Such information enriches the model stored in the web service and helps the facility commissioning process to converge to a final good decision.A double navigation path is available in the Web GUI to retrieve the feedback made by on-site operators on any BC design solution.The first path requires that a user opens the embedded IFC viewer to display the enriched IFC model, first.Then, he/she can select any building component in the model, check whether an audio file has been associated with that component, and play that audio file from the viewer.The second option is that the user opens the full list of audio files associated with a model, and, then, he/she checks which building component has been associated with any audio file.From the list, the user not only can play any audio file, but also can open an instance of the IFC viewer displaying the IFC model and the selected interested component highlighted (e.g.blue-coloured) in the active window (see Figure 13b).It is worth emphasizing that in this context interoperability is not enabled through seamless data sharing, rather it is enabled by embedding data in the physical environment and virtual objects.This scenario triggers the situated cognition hypothesis realizing a purposeful interaction between agents and their context.This process is made possible by the mediation of MR.
Overall, the continuous circular and active permeation between human agents, artificial tools and the environment, brings about a collaborative environment.Thanks to this collaborative working environment enabled by the technical framework built upon BIM and mixed reality, the overall system puts in place a decision-making process that is concluded by the remote user (ref.Figure 12).It is worth noticing that the representation provided in Figure 12 sums up the overall interoperable framework.It highlights that several interactions take place either between human agents and building components (e.g.instantiation of NFC readings by the on-site operator through the MR App and interaction with a tag embedded in a building component) or human agents and machines (e.g. an on-site user records his/her feedback while pointing at a virtual artifact of a building component in the immersive collaborative environment).As soon as the on-site user downloads a renovation option, it conveys pieces of data to building elements of a collaborative environment.This information can be displayed close to the component it refers to, as soon as the model has been aligned, thus ensuring dynamic interoperability at the information level.Furthermore, the immersive visualization facilitates agents' interpretation of state of affairs towards semantic interoperability.Despite being asynchronous, this collaborative environment enables the collective analysis of BC design solutions, facilitated by systemic interfaces and situated cognition.This affirms the feasibility of an epistemic collaboration framework among the actors involved, that can support even dynamic and operational interoperability.

CONCLUSIONS
Building commissioning projects pursue several types of objectives and involve a broad set of skills that face technological, operational and financial challenges.This generates complexity and requires the adoption of a cyber-physical paradigm that includes both the realization of interoperability in a systemic perspective and the development of processes that implement interacting functionalities among human and artificial agents.
This paper adopts the socio-technical perspective praised by Industry 4.0 and develops a methodology and prototypical technical framework supporting information, semantic, dynamic and operational interoperability.The main result is the realization of some of the key elements of a cyber-physical approach that can allow all the actors involved in building commissioning to independently and asynchronously operate to achieve a common goal.It produced two main outcomes.The first one is a data model enabling information and semantic interoperability.
The second one is a technology stack that supports dynamic interoperability and collaborative working.Thanks to human-machine interaction, BIM models can be enriched continuously, and their components can be linked to relevant information and included in a hybrid reality scenario, eventually facilitating decision-making.The coupling between BIM and mixed reality is necessary to set up systemic interfaces, such as the link between the virtual and physical parts, that create the hybrid reality, as well as to accomplish embodiment of information in physical objects.This facilitates the observer's immersive experience.This approach has been tested successfully in the case of a hypothetical building commissioning project for a full-scale real-life demonstrator building located in Caceres (Spain).Future research may concern the development and testing of a paradigm enabling full operational interoperability for collaborative working and decision-making.

Figure 1 :
Figure 1: The building commissioning problem as a Cyber-Physical System of Systems (CPSoS) made of human actors, building elements (BE), technical systems (TS), and service support operations.

Figure 2 :
Figure 2: The facility commissioning reference process.

Figure 3 :
Figure 3: Emergence of interoperability from data seeds.

Figure 4 :
Figure 4: Life cycle of the information about a technical component.In our context the actors involved have a broad common goal represented by building commissioning.Within this framework, a number of specific decisions must be made regarding different design solutions.The developed information model and the technology platform supports the epistemic collaboration of the different actors by situating the information within the operating context, thus guaranteeing semantic interoperability, and operational interoperability to achieve a successful completion.

Figure 5 :
Figure 5: The UML Entity-Relationship diagram documenting the information and semantic interoperability data model.The main entity types involved in the proposed information model are Users, Projects, Design Options, Conversions, Annotations, Building Elements, Physical Objects, and Near Field Communication (NFC) Readings.Instances of the User type store user-related data allowing an individual to access the platform, by means of standard authentication and authorization procedures (e.g.first and last names, usernames and passwords).Any instance of the Users entity type assigns one or more roles to each system user, among the available ones we mention On-Site User and Remote User.The former represents an expert or any person who is allowed to use the Mixed Reality App as well as add Annotations when surveying the facility site.Remote Users, instead, are supposed to operate mostly through the desktop interface and to be more involved in activities such as developing new Design Options or analysing the Annotations collected during the on-site surveys.Instances of Project type store information about the facility to be commissioned as well as the overall commissioning project (e.g. the name of the project, the facility owner, milestones).Instances of Design Options enclose IFC files developed by the design team using their favourite authoring software tools.An instance of Conversion stores a GLTF file together with an automatically compiled JSON file starting from the IFC dataset uploaded by the user in a Design Option.The GLTF file contains a 3D object model while the JSON file contains a tree of objects, together with their

Figure 6 :
Figure 6: An example of how a situated data seed can grow in different directions in the cyber-physical perspective on FM.

Figure 7 :
Figure 7: The UML Component diagram documenting the architecture of the technological platform.

Figure 8 :
Figure 8: (a) View of the demonstrator building and (b) close view of the IFC model of the tested renovation option, where some building elements were hidden to bring two of the NFC tags to the forefront.

Figure 9 :
Figure 9: Screenshot of the Web GUI where the renovation options have been uploaded by the design team and screenshot of the MR App while downloading one of the renovation options for on-site use.

Figure 10 :
Figure 10: (a) Two of the NFC tags coupled with the inner surface of the building's envelopes; (b) one of the onsite users reading NFC tags by means of the Mobile App and (c) first-person view of the on-site user with automatic detection of the tags by the MR App; (d) first-person view of the virtual model aligned over the real facility (d).

Figure 11 :
Figure 11: Interaction between an on-site user and the hybrid environment; (b) retrieval of information concerning one of the virtual components.

Figure 12 :
Figure 12: Activity diagram concerning the integration, interoperability and collaborative working experience.

Figure 13 :
Figure 13: (a) Once a component is selected, the on-site user can record his/her opinion and associate an audio file with it; (b) the enriched model can be queried by remote users through the web GUI.