CSU Live/Web Reference Pilot Project

Interim Report

February 2003

 

 

This is an interim report to the CSU Council of Library Deans/Directors on the CSU Live/Web Reference Pilot Project.  It provides an update on how the pilot project is going by summarizing input from the pilot libraries on the key issues of software performance, customer support, use and user satisfaction, and staff acceptance.   A final report will be issued prior to the COLD meeting in May 2003.  It will include Task Force recommendations on software, the feasibility of a CSU live/web reference consortium, and other relevant issues. 

 

Background: The pilot project was initiated under the auspices of the CSU Live/Web Reference Task Force to test two products, 24/7 Reference and Convey OnDemand.    Nine CSU libraries agreed to participate in the pilot project.  Summer 2002 was spent in training and preparation for the pilot, which began officially in September 2002.  The official pilot project will extend through April 2003; however, most libraries will continue to offer the service to students until at least the end of the academic year.  Chico elected to withdraw at the end of December 2002 because of low use, workload issues, and insufficient hours of service.

 

Summary Comments:   

 

Software:  The 24/7 Reference and Convey OnDemand software functioned well in all basic functions.  Neither product is fully mature and each has advantages and disadvantages. The advantages of 24/7 Reference include good statistical output, 24-hour service and consortium possibilities, California location and meeting room functions for instructional purposes.  The advantages of OnDemand are reliable access to all databases and systems and a persistent icon on databases pages.    Disadvantages of 24/7 Reference include lack of easy access to all databases, difficulty with the Mac platform and with Netscape on the librarian (server) side.   The disadvantages of OnDemand include install problems with the plug-in, inability to use with the Mac platform (both server and client), and possible problems establishing a consortia work group unless we elect to subscribe to QuestionPoint (a product we are not testing).

Training:  Considerable training is necessary to make a live reference service efficient and effective. Two types of training required: training on the software itself and training in answering questions remotely.  The training provided by the vendors for both products was minimally adequate. Each pilot library offered additional training.

Use:   The pilot project does not demonstrate significant demand for remote reference service at this point in time.   It is the least used of all our reference services.  However, use continues to increase.  The 24/7 Reference pilot group had the highest use.  There are at least two possible explanations: 1) the service is available 24 hours a day and 2) some of the 24/7 Reference pilot libraries had established services prior to this pilot study.   We anticipate that demand for live/web reference will grow, if we continue to offer it on a stable platform, 24 hours a day, and students find our responses acceptable.

Staff Acceptance: Participating librarians are generally accepting of the idea of live/web reference, but disappointed in the low level of use.  There is still uncertainty about  student demand for this service and concerns about staffing in the long term. 

          Recommendation: If COLD decides to continue this project,  the Task Force recommends that it sponsor a workshop for CSU librarians.   The purpose of the workshop would be to share the findings of this pilot project with representative reference staff from all CSU libraries and seek input from them on service models and staffing for a system-wide live/web reference service.  Possible times:   May or June 2003, or Fall 2003. 

I. Convey OnDemand (Fullerton, San Diego, Chico): OnDemand is a product of Convey Systems, a company that develops and provides Internet-based services and support.  OCLC's QuestionPoint uses this product.  Convey offers all essential basic functions and two high level functions, "voice over IP" and "video over IP". 

A. Testing the product

• Basic functions are stable -- call/answer with text chat and URL share.

·         Browser share feature -- not stable

·         Voice over IP -- stable but little used

·         Video over IP -- not tested

·         Various problems associated with the required plug-in -- set-up, customization, and inability to download.  Chico discovered that the plug-in is not compatible with their public web browser.  All pilot libraries experienced systems crashes when attempting to use this feature. 

·         Does not work with the Mac platform (patron and librarian), and very problematic with Netscape.

·         Full access to all databases, software applications such as WebCT, and products like Pharos once the patron has downloaded the plug-in.

• Statistics --Basic statistics provided (date/time; call status; library and librarian name; patron name, email address and connection type); does not automatically email transcripts to librarian and patron; transcripts saved only when SAVE clicked.

 

B.  Training and Support

Convey provided a satisfactory initial training using a "train the trainer" model at CSUF and CSUC.  Each library received a three-volume set of manuals.  Training continued within the libraries through group sessions and "pairs practice", i.e. librarians practicing in pairs.  All agree that the Convey system is relatively easy to learn and does not require an inordinately large investment of time. 

Technical support for the initial set up of pages was not structured enough and the documentation was not adequate/clearly presented.   As a result San Diego developed their own materials.  In addition Convey Technical Support is available only during business hours.  Overall setup and installation was problematic and time-consuming.

 

C. Service Models

Service on the virtual reference desk was substituted for service on the Reference Desk, so staff essentially did not increase their existing desk hours.   Chico provided 16 hours of service each week; Fullerton 15 hours/week; and San Diego 20 hour/week of service.  A separate site license is required for each library; therefore, the three test libraries cannot work together as a consortium to provide longer hours to students. 

 

D.  User Satisfaction and Usage

          The Convey libraries have little hard data on satisfaction though anecdotal evidence suggests that student reaction is positive in general.  For example, librarians at Chico created their own survey, which was pushed to users at the end of the each session, seven of which were fully completed. 

Use live reference in the Convey libraries was low overall.  Possible reasons include limited number of hours and the fact that the service is new.   San Diego reports 67 questions through December 2002; Fullerton reports 29 from October 9, 2002 - January 22, 2003; and Chico logged 25 uses from September through December 2002. 

 

II.   24/7 Reference (Pomona, Hayward, Long Beach, Northridge, Monterey Bay, San Jose State): 24/7 Reference is an LSTA grant-funded, California-based web reference service using customized Egain software.  It offers all essential basic functions including browser share. Voice/video over IP are available, but not currently in use by 24/7 Reference. It offers around-the-clock service through a consortium of public and academic libraries. 

 

A. Testing the Product:

• Basic functions stable and satisfactory
• Escort (the co-browsing function) fails some of the time
• Range of useful statistics available, including a user survey
• Satisfactory access from on-campus and off-campus
• Browser/operating systems -- satisfactory for most
• Connectivity to 24/7 Reference - satisfactory
• Compatibility/connectivity with campus databases -- mostly satisfactory, but some databases present problems. Pomona has its own server, other libraries connecting through EZProxy and through Innovative web access management.
• Not functional with the Mac platform on the server side
• Problematic with Netscape on the server side 
• Pharos compatibility, satisfactory for those that tested  (Northridge only)
• WebCT and Blackboard compatibility not adequately tested 
• ADA compliance -- likely poor, since product is frame-based

 

 B. Training and Support

Southern California participants were trained by experienced 24/7 Reference trainers. Northern California participants received inadequate training from the QandA Cafe trainer who had just transitioned from the LSSI to 24/7 software.  We were not successful in arranging experienced training for the northern pilot libraries.  Project lead librarians provided additional in-house training and practice sessions for their participants.     

 

       C.  Service Models   

Each pilot library is contributing between 3 and 5 hours per week to the 24/7 Reference Academic Category -- a category that was initiated by 24/7 Reference in response to our project.  It currently includes the following libraries in addition to the CSU 24/7 Reference pilot libraries: Claremont Colleges, Boston Library Consortium, Florida International University, California Institute of the Arts, Concordia University Irvine, Santa Clara University, and several others.  All of these libraries monitor the Academic category according to a pre-set schedule. Questions can be referred within the system between libraries and specialist librarians.   For example, Pomona has received several referrals of questions asked by our students and picked up by librarians at other institutions.

 

D. User Satisfaction and Usage

24/7 Reference pilot libraries have access to a built in user survey available from the reports and policies page for the service.   Most users who filled out the survey report that they like it. Examples of comments from the Pomona service: "Great service. I'll be back."  "I added you to my book marks."  "This service is wonderful."  "This system is very impressive.  It is great for people like me who do research at odd hours and do not like to disconnect from the Internet to get some help." 

Use of the service varied considerable among the libraries. 

San Jose State:        440 completed sessions (9-16-02 through 2-12-03)

Long Beach:             340 (9-16-02 through 12-20-02)

Pomona:                 138 competed sessions (9-1-02 through 1-31-03)

Northridge:              102 completed sessions (9-10-02 through 1-31-02)

Hayward:                42 total sessions (10-1-02 through 12-17-02)

Monterey Bay:         29 total sessions (9-16-02 through 12-31-02)

 

III. Staff Acceptance/Issues

• High degree of interest in the idea of live/web reference in most pilot libraries
• Willingness to experiment
• Concern about the low current use
• Skeptical about the demand for service
• Cautious about the effectiveness of remote assistance
• Voluntary staffing did not work well for some pilot libraries
• Overall concerns about the ability to staff service on an ongoing basis
• Recognition that a CSU cooperative service is the only way libraries can provide an adequate number of service hours to students

 

IV. Recommendation on a CSU-wide workshop 

 

            If COLD decides to continue this project, the Task Force recommends that it sponsor a workshop for CSU librarians.   The purpose of the workshop would be to share the findings of this pilot project with representative reference staff from all CSU libraries and seek input from them on service models and staffing for a system-wide live/web reference service.  A one-day meeting would give more people a sense of being involved in this project and would provide a means of shared discussion and understanding.  It might be possible to arrange the meeting for May or June of 2003 or for Fall 2003.  

 

 

Submitted by Kathleen Dunn

Chair, CSU Live/Web Reference Task Force

2-17-2003