Evaluation of Expert Recommender Systems

Tim Reichling & Volker Wulf, University of Siegen

Summary

Expert recommender systems (ERS) hold a great potential and this paper does an evaluation of some such systems.

Details

Knowledge management (KM) has moved from repository based approaches to social networking based strategies. ERS is a major application for KM systems. Tradeoffs exist between automating profile builder and yet respecting privacy. Existing ERS use text matching algorithms, consider structured data can match different sources of personal data. Authors had developed an ERS called ExpertFinding (EF) for a European national industrial association for which they have performed this evaluation.

EF uses two mechanisms for profile creation. First one creates a keyword list out of documents provided by users for this purpose. These documents are present on user’s local or a shared file system and could include dynamic data such as public email folders to get a tap on their day to day conversations. Another mechanism creates yellow page (YP) style form that is maintained by user themselves in case they wish to shape their expertise profile. A local, client end UI was made to allow people to search for experts by entering keywords.

Based on their evaluation, authors found that while ERS accurately found keywords indicating user’s domain of knowledge, it was inadequate in representing the nature as well as level of their expertise. Various incidents also cropped up of people bloating or expanding their expertise profile by means of YP. Authors present a 4 stage diagram where they envisaged their tool in balancing out workloads and representation of employees.

Review

From the paper it seems authors did a very poor job in building this tool and I describe why below:

• user control over both profile generation mechanism meant that users could bloat their profiles which is exactly what happened.
• choice of testers for tool evaluation was done poorly with all testers knowing each other. This led to a very specific usage pattern where A searched only for people A knew. Sample size of 23 testers also seems to be small for a study of such nature.
• no socialising of profiles was available, an important extension could have been providing ability for people to comment on other’s profile. This could have added some authenticity to YP profiles.
• the tool is too simplistic in nature. For ex. neither level nor nature of expertise was indicated on one’s profile.

Having said above, there were some good things in their study:

• privacy concerns of users were well taken care of.
• plugin based development allowed them to continuously re-deploy their tool easily.
• to their credit, they described their work exactly as they did it without attempting to cover up.

Disclaimer

The work discussed above is an original work presented at CHI 2009 by the authors/affiliations indicated at the starting of this post. This post in itself was created as part of course requirement of CPSC 436.