- Open Access
The MERG Suite: Tools for discovering competencies and associated learning resources
Source Code for Biology and Medicinevolume 3, Article number: 7 (2008)
As the demands for competency-based education grow, the need for standards-based tools to allow for publishing and discovery of competency-based learning content is more pressing. This project focused on developing federated discovery services for competency-based medical e-learning content.
We built a tool suite for authoring and discovery of medical e-learning metadata. The end-user usability of the tool suite was evaluated through a web-based survey.
The suite, implemented as an open-source system, was evaluated to identify areas for improvement.
The MERG suite is a starting point for organizations implementing competency-based e-learning resources.
Competencies in healthcare
Since the Accreditation Council for Graduate Medical Education (ACGME) published the Outcomes Project and the American Board of Medical Specialties (ABMS) endorsed the general or core competencies described in the ACGME project, medical educators across the continuum have sharpened their focus on development of competency-based education . Despite the challenges associated with implementing competency-based education , many specialty societies, boards, residency directors, and associations of medical educators are defining competencies for their specialty or discipline to guide the choice of learning activities and the assessments that learners must complete.
Concurrently, many educators are developing digital repositories of learning resources. These repositories provide rich sources of information but may be difficult to find and access, and the resources within are not always clearly associated with a formal system of competency-based education.
E-learning technology offers a way to deliver education and to assess learning and knowledge application by geographically dispersed and time-constrained physicians. Just as the adoption of electronic health records and their interoperability is being enhanced via adherence to emerging standards , implementation of effective e-learning may be enhanced by adoption of emerging e-learning standards . The health care community is only beginning to make use of standards for e-learning. The standards organization taking the lead in adapting e-learning standards to health care is MedBiquitous , an ANSI-accredited organization which is devoted to advancing healthcare education through technology standards that promote professional competence, collaboration, and improved patient care.
E-learning content creation can be expensive and require considerable investments of time by authors and developers. Early e-learning systems were tightly bound to the presentation, often hard coding the content into the presentation software or more recently dynamically generating learning content from databases.
Historically when Learning Management Systems (LMSs) were used to manage the delivery of e-learning, the learning activities were built on a specific LMS, and if the LMS changed, migration of content could be difficult and costly. In a world of rapidly evolving technology and advancements in learning platforms, consumers demanded the flexibility to migrate between LMSs without sacrificing investments in course development. Further learners and educators were challenged to locate and access relevant learning materials across LMSs or the Web.
The Sharable Content Object Reference Model (SCORM) emerged to address the lack of interoperability, reusability, accessibility, and durability. SCORM is a collection of standards and specifications for packaging, describing, and running e-learning content, as well as tracking user progress through content. In addition, SCORM allows for separation of content from navigation and deployment . One component of the SCORM model, the IEEE Learning Object Metadata (LOM) standard, standardizes the metadata about learning objects in an XML format. Healthcare LOM is an extension of this standard to meet the requirements of healthcare education and was designed by the MedBiquitous Learning Objects Working Group , a collaboration of over 30 individuals from academia, government and industry.
Work has been underway in the education sector to develop specifications for competencies and competency frameworks as well. The IEEE recently approved the Reusable Competency Definition (RCD) standard . This XML standard defines a data model for describing, referencing, and exchanging competency definitions, primarily for online learning. Reusable Competency Definitions provide a way to represent the key characteristics of a competency and enable interoperability among learning systems that use competency information.
The MERG suite
Leveraging e-learning standards, the investigators developed and implemented a prototype search tool called the Medical Education Resources Gateway (MERG). MERG allows users to: a) search the metadata records of competencies and e-learning content, and b) traverse linkages between selected competencies and associated content.
MERG provides a number of tools and services that make competency-linked health-related learning content easier to locate. Content creators and maintainers can easily author metadata, create public repositories to publish the metadata, and register their repositories with MERG. They can also associate metadata with competency frameworks. The suite monitors these registered repositories, harvests the metadata and competency associations as they change, creates metadata indices, analyzes the relationships and provides various search interfaces to discover and navigate the content and competencies based on the inter-relationships of these learning elements. The MERG suite, implemented on Java and Python platforms, is a portable collection of tools for content authors, system administrators, software developers and end users. All the tools are released as open source, enabling user customization.
Architecture of MERG Suite
MERG leverages the standards described in Table 1. Online learning modules use the SCORM model for interoperable learning content. The content is described using MedBiquitous Healthcare Learning Object Metadata (Healthcare LOM). Competencies are individually defined using the IEEE Reusable Competency Definition. Hierarchical relationships among the competencies are defined using the Simple Reusable Competency Map. MedBiquitous developed an XML schema to define an intermediate data structure for the specification of the associations between content and competencies. Figure 1 shows an instance of association through this schema. The information architecture of MERG is further described in previous publications .
Figure 2 shows the flow of information within MERG. The XML metadata created by the authoring tools is placed in a web-exposed directory along with the competency map, competency definitions, and competency associations file. These directories are periodically visited by the harvester, which supplies them to the indexer. The generated index is made available for search through web, feed and web service interfaces for both interactive and automated clients. The technical architecture is detailed here. The evolution of the suite is discussed below since the evaluation has impacted the later implementation. The earlier implementation is made available in the source package since its approach potentially has value and can be combined with the current approach once better tools become available.
Technical choices and evolution of MERG
XML database approach
MERG leverages several open source products in its implementation. The original design of MERG envisioned it as an application centred on an XML database since that would have provided both storage as well as search capabilities for the metadata in its native form without the need to maintain any further transformed data representations in synchrony. We used eXist , an open source Java implemented XML database that provided XQuery  capabilities for this purpose. XQuery is a W3C standard for querying across collections of XML documents.
Classical text retrieval approach
We later moved away from that design since the need for emphasis on proper ranking of the search results became evident. We used traditional text search techniques on these mirrors in place of XQuery. Apache Lucene , an open source Java toolkit for developing search engines was used for building the search components. The results were ranked using standard TF/IDF (Term Frequency/Inverse Document Frequency) ranking. This is a commonly used approach which ranks documents by the frequency of occurrence of search terms and their uniqueness in the document collection.
Lucene allows for considerable control over search strategies in an elegant way. We currently do not have enough data for rationally arguing for alternate strategies for this metadata corpus but opportunities exist in the future for the evolution of the product. Moving to Lucene also increased the search performance, but competency-based searching has not yet been implemented. We are now using the file system as mirror storage for the repositories.
There are currently ongoing open source efforts towards providing full text search support for eXist  and the results might be relevant to our implementation. Once these options become mature, the project may be able merge both the search approaches for a more natural design.
We used Karrigell , an open source pure Python based web server for Python based designs and used GlassFish , an open source Java Application Server for the later Java based designs. The early implementation was implemented in Python while the final implementation was on the Java platform with the exception of the simple harvester script, which is implemented Python invoking the GNU Wget  which provides the mirroring functionality.
The RSS and ATOM features of our application were enabled by the library provided by the ROME project .
Addressing various user profiles
The audience of MERG is a diverse group. It consists of authors who create competency-based content and metadata, system administrators who create the local metadata repositories, repository maintainers who update them, content consumers who search the aggregated repositories and software developers who may build on top of the existing infrastructure. Each of these user classes have distinct technical backgrounds and perform very different tasks. Addressing these multiple perspectives is one of the challenges of MERG. The modularity of MERG allows it to direct specific sub-components of the system to the user profiles that directly interact with them. Each of the user profiles, the tasks they perform, the sub-components the users interact with and some of the design decision decisions that went in to address those are described below.
To support authors in creating metadata, the MERG suite includes Hermes, a metadata authoring tool for the Healthcare LOM. Hermes provides a graphical user interface (GUI) for creating and editing metadata, removing the need to manually manage the XML files through a text or a generic XML editor. MERG features a simple sample network-based look-up service to query Medical Subject Headings (MeSH) terms, which are loaded into Hermes. Healthcare LOM supports the use of MeSH Terms as structured keywords. This service feature is a sample stub and is intended to be replaced by similar but more pertinent custom look-up services to vocabulary subsets specific to the needs of organizations. Hermes provides editors for the learning object and asset metadata files. Hermes is a Java web application and can be easily deployed to most desktop environments. Figure 6 shows the user interface of the Hermes metadata editor. The LOM and Asset metadata editors are shown within the Multi Document Interface.
To guide the system administrators who setup the repositories, the MERG suite provides a specification for the repository structure and an interface for registering repositories with MERG. It has a harvester component that visits registered repositories at a specified frequency to update its local copy of repository content. The repository structure and harvester functionality are intentionally kept as simple as possible. Individuals with skills to set up a Web server should have the skills to set up a metadata repository that can be harvested by MERG.
To simplify the task of repository maintainers, MERG repositories are specified with a simple structure intended for use by non-technical audiences, namely content and metadata authors. The repository is maintained by placing all the metadata files into a flat Web exposed directory. Metadata files conform to the Healthcare LOM standard and are assigned filenames with '.xml' extensions. The server's directory listing feature can be used to create a directory of the files if such a feature is available. Alternatively, the listing can be manually created, although this is not usually recommended since more effort or additional tools may be required to synchronize the listing with others in the repository. The metadata files need not be specially named or arranged, but are to be placed in the same directory. Competency definitions, maps, and competency-metadata associations are placed in the Web-exposed directory with the metadata files. MERG's Indexer is able to recognize the file types and processes them accordingly.
To assist the software developers in developing extensions and other value added services, MERG uses the W3C SOAP (Simple Object Access Protocol) 1.2 standard  to provide a remote interface to query the MERG search engine for metadata. This allows software developers to incorporate the end user functions of MERG into their applications without worrying about web user interface changes that may potentially occur in future.
Through a simple web interface, MERG enables users to query across multiple metadata repositories registered with MERG. The users navigate between related competencies and learning objects. Figure 3 shows the MERG search interface after a search has been performed and a LOM result from the search has been selected. The selected LOM record has a single competency associated with it. Figure 4 shows a selected competency record after a search. The selected competency record has no parent or child competencies.
Users wishing to be alerted to new content received on the MERG repository may provide specified search criteria and monitor search results using syndication feeds. The users will be presented feed URLs for each search they perform. They may then bookmark these in their feed reader, as many feed readers support user notifications when the feed content changes. MERG supports the latest standards for both RSS and ATOM, thereby allowing a very broad range of feed readers to be used. Figure 7 shows a LOM feed for the search term "contraception" that the user dragged and dropped into the browser's (Firefox) bookmarks toolbar.
A use case for the MERG Suite
A single coherent hypothetical use case scenario where various MERG Suite components come into play is described as follows.
Dr. X is a gastroenterologist and a medical educator who has prepared an online course for internists in his institution, Springfield Medical Center (SMC), on recent trends in hepatology. SMC uses a SCORM-compliant Learning Management System to deliver its content. The CME co-ordinator, Ms. Y, manages the various courses that either have been purchased or developed in house. Since Dr. X's content has been developed in house and the author has no special knowledge of the workings of Learning Management Systems, Ms. Y helps him by creating the metadata for his content and placing his course in the learning management system. Ms. Y herself has no special technical skills and is only modestly familiar with XML and certainly not with all the rules associated with authoring valid metadata. She uses the Hermes metadata editor to create the metadata for Dr. X's content through a friendly user interface.
A few months later, Dr. X's content has been well received. Through the word of mouth, physicians from other local institutions have requested access to his course. Dr. X and Ms. Y see potential to offer this course to the broader community. Dr. X would like to update the content from time to time as new advances are made in the subject area and Ms. Y sets him up with Hermes to edit the metadata. Ms. Y asks Mr. Z, an SMC network administrator, to set up a metadata and content repository so that SMC's courses can be searched by the broader community. She also asks Mr. Z to work with her to associate the content to a new competency-framework for gastroenterology that is being used by the residency program. Mr. Z is the web master for SMC web site. He downloads the competency map and competency definitions form a national gastroenterology association site. He finds that he simply has to place the metadata and competency files in a folder and enable directory browsing on the folder. He registers the URL of this repository with MERG search engine. MERG harvests and indexes the content and the competency associations. The content itself is placed in a password protected folder.
Dr. W, an internist from rural Pennsylvania, searches MERG for Hepatology content. He finds Dr. X's course and contacts Ms. Y at SMC. He registers for the course and Ms. Y authorizes Dr. W to view the content. Dr. W completes the course and subscribes to the RSS feed from the Hepatology search using his web browser. He will be notified as new Hepatology CME content is registered into MERG.
Ms. Y mentions MERG to her EMR provider, Z-Med. Z-Med looks into MERG and finds that it provides a web services interface to its search. Z-Med has been looking to integrate CME opportunities into its electronic health record system. Z-Med comes up with a plan to integrate links to CME content in its physician portal.
In order to test the usability of the system, we developed metadata for two content domains:
The course, Introduction to Biomedical Informatics, was developed by one of the authors (WRH) and is taught in the biomedical informatics graduate program at Oregon Health & Science University (OHSU) and the AMIA-OHSU 10 × 10 Program .
Content from Contraception Online  was made available by another author (MF) and offers a variety of learning resources for healthcare providers and patients on a range of topics related to contraception. Resources included didactic slide presentations, interactive cases, monographs, and patient education handouts.
MERG Suite underwent internal iterative developmental evaluation towards the design of Hermes and the server side tools. The feasibility of the prototypes were tested using the above two real world metadata collections. Since the web search component of MERG was expected to service most users, we conducted a survey of the potential users of the systems. After receiving approval from the OHSU Institutional Review Board, we administered the survey using Survey Monkey . Participating medical educators were recruited using several email lists (e.g., the MED-ED email list of the Association of American Medical Colleges; the DR-ED email list of Michigan State University; and the MedBiquitous Competencies, Learning Objects, and Virtual Patient Working Groups). The survey consisted of two parts. The first part focused on the use of competencies in medical education. The second part of the survey focused on experience with MERG suite and was administered after participants had an opportunity to apply the tool to one of the two content areas – medical informatics or contraceptive topics in women's health.
The survey instrument included questions about participants' educational roles and responsibilities, their use and the availability of competencies for the discipline(s) in which they teach, and their views on the value of a tool that would enable searching for e-learning content by using competency search terms. Each participant was asked to use our prototype system with its candidate competencies and content and to provide feedback to guide further development.
A total of 42 individuals took part in our survey. As seen in Tables 2, 3, 4, the participants serve in various medical education roles, with responsibilities for educating a diverse mix of learners, including physicians (the most frequently represented target audience, graduate students, and nurses and allied health professionals. With respect to physician education, over three-quarters had responsibilities in undergraduate medical education, two-thirds for graduate medical education, and slightly less than half for continuing medical education. However, all were frequently involved in developing learning materials. Nearly 80% (30/38) stated that competencies were available to them, with most citing the ACGME competencies, rather than discipline-specific competencies. As shown in Table 5, most of survey participants had used competencies recently.
Table 6 shows their perceptions about the competencies. While over 80% described comfort with using competencies to develop educational materials, over 40% reported that significant barriers exist to using competencies for that purpose. Furthermore over 90% felt that linking learning resources to competencies would be useful and 88% would use a tool that facilitated such linkage. Finally over half believed that access to such a tool may change how they design instruction.
With respect to the barriers in using competencies, respondents elaborated in the free-text comments:
Available competency sets do not align with curriculum goals.
There are inadequate numbers of competency sets.
The faculty does not yet understand the concept of competency-based education.
Available competency sets are not practical or are vague or unproven.
Educators have trouble choosing between competing competency sets.
There is difficulty in adopting competency sets within traditional curricula.
Creating competency-based assessment tools is difficult.
Competency sets add to information overload.
Although participants expressed enthusiasm for the MERG-type general search tool, their reaction to our prototype system was mixed. As shown in Table 7, nearly half of the participants indicated that the MERG search tool was easy to use. However, assessments of the usefulness of accessing learning resources were mixed as indicated by the responses to three additional questions. Over three-quarters (24/31, 77.4%) of participants reported that the search result pages did not show sufficient information, and about half (16/31,51.6%) reported that the search function was useful and a similar proportion (17/31,54.8%) reported that the links between competencies and content were useful.
Although our intent was to assess respondents use of our search system linking competencies to relevant learning objects using limited data sets for illustrative purposes, survey participants in their comments focused more on the narrowness of available learning objects rather than on any challenges encountered in the use of the search system. This may be that in our instructions we did not communicate the limited nature of learning objects available for testing and thus did not manage expectations and/or the search tool itself was intuitive for most users and the challenges encountered related more to their inability to identify a wide variety of resources. The feedback we gathered from the study guided us in updating the web user interface. Figure 5 shows the updated user interface for MERG. The current user interface is more fully featured and designed as a single page view with a tabbed interface. It is also more responsive and incorporates some cosmetic improvements.
Our study engaged a group of medical educators that understood and endorsed the competency-based approach to learning; although they did indicate that the competency sets specific to their teaching are currently limited. While they agreed in principle with the approach taken through development of the MERG search tool, the participants focused on the comparatively scant content generated from their searches. In retrospect, participants should have been alerted to the narrowness of the current content knowledge-base which was assembled only for usability evaluation purposes. More specific instruction should have been provided to participants explaining our interests in testing the functionality of the search tool and factors related to ease of use, rather than investigations related to the comprehensive nature of search returns.
Notwithstanding the issue of search return volume, the results from this assessment of MERG suite functionality indicate that overall the professionals who tested it found it easy to use and were in agreement with the usefulness of the approach. With the addition of a wider range of content and linked repositories, quite different testing results would likely be realized and greater insights into functionalities of interest might be revealed.
The future utility of MERG will depend on the availability of standardized, competencies that are linked to content.
Barriers to adoption
While the important role of competencies is now recognized in the educational community and standards for authoring competencies are available, only a small portion of content is linked to competencies. Tools such as the MERG suite may lower the barrier of entry in developing and distributing competency-linked content from a technical perspective by providing tools that shield the users from the details of the specification and by offering guidance for setting up the repositories. However, additional research is needed with respect to effective use of competencies. Experiences in domains such as U.S. Public Health Service, which appear to be at the forefront in designing and applying competencies for health professional education, may inform future development in this area.
MERG has been designed as a modular application with loose coupling between components. While this approach has distinct advantages, it does pose challenges in deployment. Although not a substantial service at present, our aim is that the MERG tool kit serve as an evolutionary platform for implementing such services in the future.
Availability and requirements
Project name: MERG Suite
Project home page: http://ir.ohsu.edu/MergSuite/
Operating system(s): Windows 98 or higher, Unix-based OS.
Programming language: Java, Python, Groovy
Other requirements: Java 1.6 or higher, Python 2.5 or higher
License: GNU GPL.
Any restrictions to use by non-academics: None
Batalden P, et al: General competencies and accreditation in graduate medical education. Health Affairs. 2002, 21: 103-10.1377/hlthaff.21.5.103.
Harden RM: Learning outcomes and instructional objectives: is there a difference?. Medical Teacher. 2002, 24: 151-155. 10.1080/0142159022020687.
Brailer DJ: Interoperability: the key to the future health care system. Health Affairs. 2005, 24: 19-21.
Fallon C, Brown S: E-Learning Standards: A Guide to Purchasing, Developing, and Deploying Standards-Conformant E-Learning. 2003
MedBiquitous Consortium. [http://www.medbiq.org]
SCORM®. – Sharable Content Object Reference Model. [http://www.adlnet.gov/scorm/index.aspx]
Learning Objects Working Group. [http://www.medbiq.org/working_groups/learning_objects/index.html]
IEEE Reusable Competency Definition or Educational Objective – Information Model. [http://www.ieeeltsc.org/working-groups/wg20Comp/wg20rcdfolder/]
Hersh WR, et al: Adopting e-Learning Standards in Health Care: Competency-based Learning in the Medical Informatics Domain. AMIA Annu Symp Proc. 2006, 334: 8.
eXist XML database. [http://exist.sourceforge.net/]
Apache Lucene. [http://lucene.apache.org/]
eXist eXtension to full text search. [http://exist.sourceforge.net/soc-ft.html]
Karrigell Web Server. [http://karrigell.sourceforge.net/]
GlassFish Open Source Application Server. [https://glassfish.dev.java.net/]
GNU Wget. [http://www.gnu.org/software/wget/]
Direct Web Remoting Library. [http://getahead.org/dwr]
ROME Project. [https://rome.dev.java.net/]
Apache Velocity. [http://velocity.apache.org/]
W3C SOAP Specification. [http://www.w3.org/TR/soap/]
Hersh W, Williamson J: Educating 10,000 informaticians by 2010: the AMIA 10 × 10 program. International journal of medical informatics. 76: 377-82. 10.1016/j.ijmedinf.2007.01.003.
Contraception Online. [http://www.contraceptiononline.org/]
Survey Monkey. [http://www.surveymonkey.com/]
ADL SCORM. [http://www.adlnet.gov/scorm/index.aspx]
MedBiquitous Healthcare Learning Object Metadata. [http://www.medbiq.org/working_groups/learning_objects/index.html]
IEEE Reusable Competency Definition. [http://www.ieeeltsc.org/working-groups/wg20Comp/wg20rcdfolder/]
Proposed Draft Standard for Learning Technology – Simple Reusable Competency Map. [http://www.ostyn.com/standardswork/competency/ReusableCompMapProp20051117.pdf]
MedBiquitous Competencies Working Group. [http://www.medbiq.org/working_groups/competencies/index.html]
This work was funded in part by Grant 1G08 LM008235 of the National Library of Medicine
The authors declare that they have no competing interests.
RTB is a doctoral candidate in the Biomedical Informatics PhD program at Oregon Health & Science University. RTB designed and programmed the systems.
WRH is Professor and Chair of the Department of Medical Informatics and Clinical Epidemiology and provided leadership as Principal Investigator of the project. He also provided part of the metadata used for testing the systems from a course taught by him as well as designed and disseminated the survey used for evaluation.
VS is the Deputy Director of MedBiquitous. VS provided the expertise for working with standards from MedBiquitous.
MF is the Senior Associate Dean for Continuing Medical Education at Baylor College of Medicine and Director for the Center for Collaborative and Interactive Technologies. MF assisted with women's health portion of the metadata content that was used to test the readiness of the system for real world metadata and contributed to the survey design
PSG is the Executive Director of MedBiquitous and provided leadership as a Co-Investigator on the project.
Ravi Teja Bhupatiraju, William R Hersh contributed equally to this work.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.