USE OF KEYPHRASE EXTRACTION SOFTWARE FOR CREATION OF AN AEC/FM THESAURUS

Branka Kosovac, PhD candidate,
University of British Columbia;
email: branka@civil.ubc.ca

Dana J. Vanier, PhD,
National Research Council Canada;
email: Dana.Vanier@nrc.ca

Thomas M. Froese, PhD,
University of British Columbia;
email: tfroese@civil.ubc.ca

SUMMARY: The paper describes a method used to collect terms needed for the development of a thesaurus in the roofing domain. This work is part of a larger effort to investigate the potential of thesauri as an aid in product modeling and as a tool for information management in model-based systems. Extractor, a software module that extracts keyphrases from documents, was used for collecting candidate thesaurus terms from Internet sources. The principal advantage of the Internet as a source of candidate terms is that it reflects the language that is actually used in communications concerning buildings and that it covers the widest range of different views on the domain. The advantage of using Extractor or similar software is that it allows processing huge text corpora available on the Internet while eliminating irrelevant terms. The methodology used was found to be highly useful, although it was not sufficient by itself for constructing a thesaurus for the architecture, engineering, construction and facilities management industries, as considerable human intervention was required. Some possibilities for customizing the software and for partially automating a thesaurus construction process are suggested.

KEYWORDS: thesauri, Internet, automatic indexing software, thesaurus construction.

1. INTRODUCTION

Extractor (Extractor 2000) is a machine-learning-based software module, developed by the Interactive Information Group of the National Research Council Canada, that scans an electronic document and extracts keyphrases best describing the document's subject matter. In this project, the authors used Extractor 2.0 as a support tool for collecting and selecting terms to be included in the proposed thesaurus.

In the process described, Extractor was used for a specific task and under given circumstances. Time and resource constraints did not allow full exploitation of Extractor's capabilities. These constraints also precluded testing corpora (document collections) large enough to provide statistically valid results. Therefore, the work described is of explorative nature and cannot be considered as a study that evaluates the performance of Extractor. However, the patterns noticed in the analysis of the results can point to possible use of the software for related purposes and suggest possibilities for further research and development.

1.1 Problem description and general approach

• extract selected terms from the TC/CS to form a sectorial thesaurus,
• update the sectorial thesaurus to reflect current technology and terminology, and
• develop a micro-hierarchy of narrower concepts, if required.

• collect terms relevant to the field,
• select terms to be included in the thesaurus according to its intended purpose,
• check the terms against the existing TC/CS thesaurus,
• organize the terms into hierarchies following the TC/CS guidelines.

• terminological sources in standardized form: existing thesauri, dictionaries, glossaries, classification schedules, encyclopaedias, lexicons, journal indices, back-of-the-book indices, term lists, treatises on terminology of a subject field;
• literature scanning;
• question (i.e., user query) scanning;
• users', subject experts', and compilers' knowledge (Aitchinson and Gilchrist, 1987).

1.2 Use of information technology in thesaurus construction

Since the mid-eighties, there have been considerable research efforts related to automatic extraction of semantic relationships and automatic construction of structured thesauri, e.g. Chodorow et al. (1985), Fox et al (1988). Despite some claims of successful R&D projects, such as MindNet (Richardson et al., 1998), wide use of full automation in thesaurus construction is still unrealistic. One simple but important reason is that the procedure requires special corpora -- in almost all projects machine-readable dictionaries have been used. As a comprehensive, authoritative and up-to-date source for a certain subject domain is rarely available, it can be expected that computers will still be used only as support tools in this part of the process for a time to come.

1.3 Use of thesauri in product model-based systems

Thesauri address the general task of mapping related concepts, a task that arises frequently in model-based systems. They provide formalisms and approaches for mapping concepts, such as the addition of relative specificity of relationships (e.g., one term might be described as related to, but more narrowly defined than, another term). Thesauri can be used to map not only related words and phrases, but to map a wider range of representational elements such as semantic models, heterogeneous forms of computer-based data, etc. Roles in which thesauri may be useful in model-based systems include the following:

• Basic terminology: Data models must use specific, unambiguous terms for each concept represented in the model. Real world language, and even the terminology used within the set of all AEC/FM computer applications, is much less precise, and a wide variety of terms may commonly be used to refer to the same concept. Thesauri can help map the terminology used by different applications and users to the terminology used within a shared product model.
• Schema: Information sharing in distributed model-based systems relies on common, standard data models (schemas). In practice, however, many different schema exist that must be mapped to each other. Examples include the mapping of application schemas to common shared schemas, accommodation of different views and perspectives representing various disciplines or users, and the generation or interpretation of new semantics from a project database. While research and development is progressing in the area of mechanisms for mapping schema (such as the Express-X language), several aspects of this mapping relate to thesauri issues and techniques.
• Heterogeneous representations: Within distributed model-based AEC/FM systems, information about a single physical component of a building may be represented in numerous forms, such as textual and numerical properties of a software object, data values dynamically calculated on-demand, 2D and 3D geometry in one or more formats (e.g., surface models, solid models, etc.), digital multi-media information such as photographs, hyperlinks to external references, and so on. Techniques must be developed to inter-relate all of these data forms at varying levels of granularity and to manage the heterogeneous data sets. Again, while this is beyond the scope of traditional thesauri applications, the underlying requirements are very similar and thesauri-based approaches are likely to provide useful contributions to the required capabilities.
• Further work in this research direction, including the use of the developed thesaurus, will be reported in subsequent papers.

2. PROJECT

2.1 Approach

On the other hand, the TC/CS has a thoroughly elaborated structure and well defined inter-term relationships that facilitate the addition of new terms, given that their exact meaning is known. Furthermore, it is available in electronic form on the World Wide Web ( http://www.cisti.nrc.ca/irc/thesaurus/. ) thus allowing easy searching for known terms. For these reasons the "bottom up" approach suggested earlier seemed to be the most appropriate approach for this work.

As one of the primary intended uses of the proposed thesaurus is indexing and retrieval of Internet/intranet sources, the most useful source of terms would be corpora available on the Internet. The Extractor 2.0 documentation pointed to the suitability of the software for performing "literature scanning" of Internet sources as it integrates HTML and e-mail filters and permits processing of large corpora by extracting only relevant terms.

2.2 Preparatory tests

First, the query initially used to test harvesting corpora using automatically generated queries that combine synonyms of the top term (summum genus) and its immediate narrower terms was as follows:

("low-slope roof*" OR "flat roof*") AND ("built up" OR BUR OR "multi ply" OR "single ply")

(where BUR is an abbreviation for built-up roofs). This query did not provide optimum recall and precision within some search services as processing a sample large enough to compensate for the deficiencies was unrealistic. The query was thus modified to the following:

("flat roof*" OR "low slope") AND ("built up" OR BUR OR roofing OR membrane*)

This better reflected the content and the language of the relevant documents. Although the new query did not eliminate all the "noise", nor did it ensure absolute recall, the retrieved sets of documents seemed acceptable for the purpose.

Second, the initial strategy was to process documents using the lowest setting for the number of keyphrases, identify terms that should be added as stop phrases (i.e., terms such as "roofs" or "roofing" that appeared as keyphrases in most of the documents preventing extraction of more specific terms), cluster documents based on the rest of the keyphrases, process them with the highest setting for the number of keyphrases, and repeat the process until the desired level of specificity was achieved. However, the inability to frequently customize the software by adding new stop phrases and the labourious task of processing the same documents more than once made this strategy unfeasible. Using the new release with the maximum setting for the number of keyphrases derived and simply removing the most frequent terms proved to yield satisfactory results.

It was observed that long documents, which tended to abound with very specific terms, yielded only very general terms in the Extractor output. Although keyphrases derived by Extractor reflected well the subject of the documents, they did not include specific terms that would be more useful for the purpose. Efforts to automatically divide long texts into meaningful sub-documents did not prove feasible with most of the Web documents. It was, therefore, done only on a small number of scientific papers that tended to be well organized and have a better HTML structure, thus allowing easy division by searching for heading tags.

2.3 Methodology

2.3.1 Corpora

• Documents retrieved by general search services

("flat roof*" OR low-slope) AND (built-up OR BUR OR roofing OR membrane*)

The services were searched in alphabetical order, taking care to avoid duplicates. Where relevant documents from a certain site were grouped together, only the first one was used in order to avoid language of one author and frequent appearance of the same corporate names and trademarks.

• IRC Roofing Resources collection (Roofing Resources 1998)

• Relevant documents retrieved at FacilitiesNet (FacilitiesNet 1998)

• Collection of selected articles

Messages from newsgroups archives were not processed separately but a small number of this type of document was included in AltaVista hits.

2.3.2 Procedure

• relevance factor number (provided by Extractor),
• document from which the term was extracted,
• and for each document:
• collection ID,
• size of the file.

2.3.3 Criteria

After the identification of exact matches, terms from the list were also searched for occurrence as:

• [PH] phrases from the thesaurus
• [Q] qualifiers in the thesaurus.
• [~] close matches, meaning terms having the same stem.

• [A] acronyms;
• [N] proper names;
• [*] phrases ill defined by Extractor (meaning that they could not stand alone either as terms in the thesaurus--which includes only nouns and noun phrases--or as meaningful keywords for a document--e.g. "requiring", "single", or "install"); and
• [$] matches that have the same form but different meaning in the thesaurus.

3. RESULTS AND DISCUSSION

The results from the general search services were then compared with those from selected collections. There were no significant differences noted in the relevance of extracted keywords that would justify the laborious task of searching, evaluating, and selecting sources. The quantity and diversity of documents that can be easily retrieved by general search services can successfully compensate for the quality of selection. As the first 20 documents from the compiled collection described in Section 2.3.1 did not bring new terms to the list, this collection was not further processed and it is not included in the final results. However, the documents have been saved for later comparison of scholarly and natural language terms and for exploration of specific sub-areas.

The final list consists of 1054 terms (2423 occurrences) extracted from 176 documents. Almost half of the terms extracted were single words (49 %). They accounted for almost all of the top 4% most frequent terms with only two exceptions that would normally be included in stop-phrases. The usual practice of treating such terms separately was not followed as most of the terms were also identified as single word terms in the TC/CS.

A huge number of single occurrences of a term (78%) can be explained by the insufficient size of the sample. In order to extract relevant terms from this group, they were searched for component words and stems that could also be found in other terms. Terms found in this way were ranked higher in the list as more relevant. Rough scanning of the remaining single-occurrence terms found very few terms that were relevant to the field. For that reason, these terms were excluded from further processing. It is important to note, however, that this group contained a substantially smaller percentage of terms marked as ill-defined, significantly contributing to the overall average of only 1% ill-defined terms.

3.1 Comparison with the existing thesaurus

FIG.1: List of terms with more than two occurrences compared with TC/CS Fig. 2 shows the breakdown of terms with more than two occurrences that had no matches or close-matches found in the existing thesaurus -- 12% (or 3% of all the terms extracted) were proper names, 8% (2% of all the terms) were acronyms, and 19% (or 5% of all the terms) were marked as ill defined.

Figure 2: Breakdown of terms not found in TC/CS The majority of mismatches, however, do not indicate irrelevance of the terms but more often the outdatedness of the TC/CS. The frequent occurrence of the term "membranes" for example, and the phrases containing "membranes" that are not found in TC/CS reflects changes in the field of low slope roofing and its terminology. The noise-making terms, i.e. extracted terms that do not belong to the low-slope roofing domain, come from specific kinds of documents, mostly glossaries and book catalogues. Such documents can be easily excluded from the beginning by modifying the initial query.

The relatively high coincidence of terms may, on the other hand, indicate a lack of more specific terms that would be required for developing a microthesaurus. Whether this is the case can be established only later in the process.

The matches were found in all semantic classes and in various hierarchies, showing a broad coverage of the domain. Completeness of coverage, however, is yet another problem that cannot be properly evaluated at this point but only after organizing terms into hierarchies (Petersen 1990).

3.2 Comparison with the glossary

4. CONCLUSIONS AND RECOMMENDATIONS

4.1 Observations on performance of Extractor

The number of phrases marked as ill-defined (see the explanation in Section 2.3.3) was 1% of all the extracted terms. Since the total automation of the thesaurus constructing process is not considered and since ill-defined phrases cause no serious consequences, this percentage can be considered negligible. Therefore, increasing the number of keyphrases even above the maximum that was possible in version 2.0 (in order to retrieve more specific terms) would probably be safe. In most cases the lack of more specific terms in the Extractor output will not represent a deficiency; terms too specific to be the subject of a document are rarely included in a thesaurus. In addition, their presence might make one of the most important decisions in thesaurus constructing -- where to stop -- even more difficult. However, if constructing a microthesaurus, or if for any other reason more specific terms are needed, these terms may be obtained by processing larger corpora or by narrowing the searches for the Internet documents to be processed.

4.2 Observations on the Internet as a source of thesaurus terms

4.3 Implications for the further work on the thesaurus

• terms derived by Extractor from new sets of documents retrieved by more specific terms,
• terms manually extracted from documents already retrieved and judged as relevant to the subfield according to keyphrases derived by Extractor.

4.4 Possibilities for automating the process

• automatic retrieval of relevant documents from the Internet and their processing with Extractor,
• periodical processing of the list by Extractor for finding terms with significantly high occurrence and making them stop-phrases,
• ranking terms by frequency of occurrence,
• exclusion of terms that occur only once in large corpora,
• automatic exclusion of geographic names ,
• grouping of terms containing the same words or stems,
• grouping of terms by co-occurrence in documents,
• suggesting inter-term relationships by co-occurrence and syntax.

5. FINAL NOTES

Since the time of the study, several new versions of Extractor have been released. Anyone interested in implementing the method described in this paper should review new features of the software by visiting the Extractor Web site or by contacting the software's authors.

Original data and full results of the study are available from the authors of this paper.

6. ACKNOWLEDGEMENTS

7. REFERENCES

Biegel, S. (1989) . Roofing Materials, Encyclopedia of Architecture, Design, Engineering & Construction. Vol. 4, American Institute of Architects, 314-319.

Chodorow, M., Byrd R. and Heidorn G. (1985) . Extracting semantic hierarchies from a large on-line dictionary, Proceedings of the 23^rd Annual Meeting of the ACL, 299-304.

Extractor (2000) . National Research Council of Canada, Interactive Information Group, Ottawa, Canada. Available from: http://extractor.iit.nrc.ca. [Accessed February 7, 2000]

FacilitiesNet (1998) . Trade Press Publishing, Milwaukee, WI, USA. Available from: http://www.facilitiesnet.com. [Accessed November 27, 1999]

Fox, E.A., Nutter J. T., Ahlswede T., Evens M. and Markowitz J. (1988)  . Building a large thesaurus for information retrieval, 2nd Conference on Applied Natural Language Processing, Association for Computational Linguistics, (Ballard B., ed.), Bell Communications Research, Morristown, NJ, 101 -108.

Gilchrist, A. (1971) . The Thesaurus in Retrieval, Aslib, London.

Kosovac, B (1998) . Internet/Intranet and Thesauri, Canadian Institute for Scientific and Technical Information, Internal Report, National Research Council Canada, Ottawa, Canada. Available from: http://www.nrc.ca/irc/thesaurus/roofing/report_b.html [Accessed November 27, 1999]

Lancaster, F.W. (1986) . Vocabulary Control for Information Retrieval, Information Resources Press, Arlington, VA, USA.

Petersen, T. (1990) . Developing a new thesaurus for art and architecture, Library Trends, Vol. 38, No. 4, 644-658.

Richardson, S. D., Dolan W. B. and Vanderwende L. (1998)  . MindNet: acquiring and structuring semantic information from text. Microsoft Research Technical Publications (MSR-TR-98-23). Available from ftp://ftp.research.microsoft.com/pub/tr/tr-98-23.doc [Accessed January 31, 2000]

Roofing Resources (1998) . National Research Council of Canada, Institute for Research in Construction, Ottawa, Canada. Available from: http://www.nrc.ca/irc/roofing/roofing.html. [Accessed November 27, 1999]

TC/CS (1978) . Canadian Thesaurus of Construction Science and Technology, Department of Industry, Trade and Commerce, Government of Canada, Ottawa. Available from: http://www.cisti.nrc.ca/irc/thesaurus/. [Accessed November 27, 1999]

Vanier D.J. (1994). Canadian thesaurus of construction science and technology: A hypercard stack, Proceedings of the Joint CIB Workshops on Computers and Information in Construction (Montreal, Que., Canada), (CIB Proceedings, Vol. 165), 559-564.

APPENDIX A