KY Virtual Library Advisory Committee Minutes
1:00 p.m., August 19, 2005
Conference Call
Present: Shelley Burgett, Anne Chase, Fannie Cox, Diane Culbertson, Charlene Davis [for Jim Nelson], Rose Davis, Carol Diedrichs, Allen Lind, Diane Nichols [for Hannelore Rader], Enid Wohlstein, Laurene Zaporozhetz
Absent: Mike Binder, KDE Representative, Stuart Johnston, Chela Kaplan, Jim Nelson, Norma Northern, Hannelore Rader, Lisa Rice
1. KYVL Collections Work Group Database Assessment Report (C. Davis)
As chair of the Collections Work Group, Charlene presented the report. Charlene commented that there were no surprises from what has been seen in past surveys. The surveys completed earlier this year are just one part of the process to prepare for the RFP. [See next page for report.]
2. KYVL Advocacy
Advocacy for KYVL and funding requests was discussed. Carol suggested providing past data and contract costs. Laurene suggested pie charts for funding and search data, respectively. She will also send links to Ohio and LOUIS information.
3. Next meeting is scheduled for October 21st.
KYVL Collections Work Group
Database Assessment
REPORT
2005
Submitted to VLAC
August 19, 2005
KYVL Collections Work Group Membership
Charlene Davis, Kentucky Dept. for Libraries and Archives, Chair
Charles Brown, Sullivan University
James Burgett, University of Kentucky
Bryan Carson, Western Kentucky University
Ida Cornett, Lexington Public Library
Mary Beth Garriott, Centre College
Martha Geier, Louisville Free Public Library
David Holt, Hardin Memorial Hospital
Victoria Koger, Eastern Kentucky University
Carol Nutter, Morehead State University
Suzy Szasz Palmer, University of Louisville
Margaret Roberts, KSMA President, Scott County High School
Jen Schatz, Murray State University
Lynda Short, Paul Laurence Dunbar High School
Sheila Stuckey, Kentucky State University
Cecelia Tavares, Jefferson County Public Schools
Sheree Williams, Jefferson Community College
Enid Wohlstein, KYVL
KYVL Collections Work Group Assessment Committee Members
Charlene Davis, Kentucky Dept. for Libraries and Archives, Chair
Martha Geier, Louisville Free Public Library
Jen Schatz, Murray State University
Lois Schultz, Northern Kentucky University
Lynda Short, Paul Laurence Dunbar High School
Cecelia Tavares, Jefferson County Public Schools
Sheree Williams, Jefferson Community College
Enid Wohlstein, KYVL
Tim
Chase, KYVL, Resource Person
BACKGROUND
The assessment of the KYVL Databases has been an ongoing project of the Collections Work Group. However, this Assessment is an especially important one since the results of this one will drive the procurement process which will result in the first new contracts for database access since 2001. An Assessment Committee of the KYVL Collections Work Group was formed in October 2004. The general charge of the Assessment Committee was to complete the evaluation of the current databases and present its findings to the Collections Work Group as a whole. The Report would ultimately be presented to VLAC.
In previous assessments, both quantitative and qualitative data were utilized in the process. It was determined that this same pattern would be improved upon and deployed. Subsequently, information which would be utilized in this assessment would include database use statistics, a customer satisfaction survey, and focus groups.
The customer satisfaction survey was the first task because we knew based on previous surveys that it would take considerable time and effort to accomplish. Discussion about previous surveys and their outcomes revealed that the instrument which listed the databases for user reaction was an obviously flawed evaluation strategy as was evidenced by the responses. Respondents were seemingly confused about the KYVL Gateway, predetermined search categories or age appropriate portals, and native interfaces. The Work Group recommended to the Assessment Committee that multiple survey instruments targeted to specific user groups might yield better results. Subsequently, nine separate survey instruments focusing on how specific user groups might be likely to approach the databases were created and deployed. These surveys were created in and deployed through Zoomerang, an online web based survey software.
Database use statistics were brought into the mix as our qualitative assessment. It should be noted that the compilation of these statistics is extremely difficult to obtain although the contracts require that these statistics be provided to KYVL in a prescribed manner.
Because the e-mail focus group concept had proven so successful during the last assessment process, we determined that we would again ask for volunteers to participate in the focus groups through these customer satisfaction surveys as appropriate by user group. While Work Group and Assessment Committee members were not able to commit to conducing formal focus groups, several were interested in conducting less formal discussion groups and adding those results in to the qualitative side of the assessment equation.
Summaries of all of these activities follow.
KYVL Online Resources Customer Satisfaction Surveys Results
The nine (9) KYVL Online Resources Customer Satisfaction Surveys (Attachment A) were deployed on February 15, 2005 via the KYVL website. Various listservs and conventional paper letter with surveys were also deployed to announce the assessment and to garner greater participation in the process. Responses were accepted through both of these means. The Surveys were concluded on April 1, 2005. At the time of closure, there were 803 respondents to the surveys. It is of interest to note that 19% of these were returned in paper form.
Who Responded?
|
SURVEY USER GROUP |
NUMBER OF RESPONDANTS |
RESPONSE RATIO |
|
Academic Libraries |
262 |
33% |
|
Public Libraries |
152 |
19% |
|
Special Libraries |
23 |
3% |
|
Elementary School Librarians, Faculty and Staff |
37 |
5% |
|
Elementary School Students |
34 |
4% |
|
Middle School Librarians, Faculty and Staff |
21 |
3% |
|
Middle School Students |
27 |
3% |
|
High School Librarians, Faculty and Staff |
81 |
10% |
|
High School Students* |
166 |
20% |
|
TOTAL |
803 |
100% |
The Academic, Public and Special Libraries Surveys included responses from librarians, library staff and library customers on a single instrument.
Overall how did they rate the KYVL online resources?
|
SURVEY USER GROUP |
Poor |
Fair |
OK |
Good |
Excellent |
|
Academic Libraries |
2 |
8 |
42 |
116 |
82 |
|
Public Libraries |
1 |
5 |
18 |
72 |
53 |
|
Special Libraries |
0 |
1 |
1 |
11 |
10 |
|
Elementary School Librarians, Faculty and Staff |
0 |
0 |
1 |
11 |
24 |
|
Elementary School Students |
0 |
0 |
2 |
7 |
24 |
|
Middle School Librarians, Faculty and Staff |
0 |
1 |
1 |
6 |
12 |
|
Middle School Students |
0 |
2 |
2 |
9 |
12 |
|
High School Librarians, Faculty and Staff |
2 |
2 |
6 |
30 |
36 |
|
High School Students* |
10 |
25 |
40 |
51 |
37 |
|
TOTAL |
15 |
44 |
113 |
313 |
290 |
|
RESPONSE RATIO |
2% |
5% |
14% |
40% |
37% |
What is it that the respondents wanted which they were not getting in the current online resources?
|
Number of Occurrences |
Subject Areas |
Databases Specific to Subject Areas |
|
32 |
Arts & Humanities including Literature, Poetry, Religion |
|
|
17 |
History/Social Studies |
|
|
15 |
Medicine/Psychology/Health/ Fitness |
|
|
15 |
Education/Library Science/ Careers/Computer Technology/ Culinary Recreation |
|
|
10 |
Newspapers |
|
|
9 |
Reference Shelf |
|
|
7 |
Business/Economics |
|
|
4 |
Science Physics/Biology |
No specific databases were named |
|
4 |
Genealogy |
|
|
3 |
Law |
|
|
3 |
General |
|
|
1 |
Language Learning |
|
What else did we learn from the surveys?
The reason for the targeted surveys was that in past assessments it appeared that respondents were confused about the KYVL Gateway, predetermined search categories or age appropriate portals, and native interfaces. In past responses, we were seeing disproportionate numbers of Never Used responses. The respondents would then indicate that they felt that the databases were excellent. Did the targeted approach help?
In the instruments that asked about predetermined search categories and age appropriate portals, it seemed that respondents were less confused and more familiar with the resources.
|
SURVEY USER GROUP |
Response Ratio Indicating Good or Excellent Rating |
Response Ratio Indicating Not Applicable |
|
Public Libraries |
84% |
17% |
|
Elementary School Librarians, Faculty and Staff |
97% |
10% |
|
Elementary School Students |
94% |
19% |
|
Middle School Librarians, Faculty and Staff |
90% |
20% |
|
Middle School Students |
84% |
27% |
|
High School Librarians, Faculty and Staff |
86% |
22% |
|
High School Students* |
54% |
10% |
Only two of the instruments utilized the previous survey formula; e.g., each database was listed individually and the respondents were asked to rate each one.
|
SURVEY USER GROUP |
Response Ratio Indicating Good or Excellent Rating |
Response Ratio Indicating Not Applicable |
|
Academic Libraries |
79% |
39% |
|
Special Libraries |
91% |
45% |
High Not Applicable ratios in the academic responses were found in the children’s targeted databases. However, these were not huge spikes that then drove the percentage up as is evidenced in the table below. It should also be noted that 62% of the respondents identified themselves as library customers.
|
Academic N/A Responses by Ratio |
Occurrences |
|
0-10% N/A |
1 (8% N/A – Academic Search Premier) |
|
11-20% N/A |
1 (20% N/A –Newspaper Source) |
|
21-30% N/A |
9 |
|
31 – 40% N/A |
17 |
|
41 – 50% N/A |
11 |
|
Over 51% |
10 |
The Not Applicable results in the Special Library responses were closer to what one might expect from such a user group. It should be noted that 78% of the respondents identified themselves as librarians.
|
Special N/A Responses by Ratio |
Occurrences |
|
0-10% N/A |
4 (0% N/A – Academic Search Premier) (10% N/A –Business Source) (10% N/A—CINAHL) (10% N/A – ERIC EBSCO) |
|
11-20% N/A |
5 |
|
21-30% N/A |
4 |
|
31 – 40% N/A |
10 |
|
41 – 50% N/A |
6 |
|
Over 51% |
20 |
Interesting data resulted from the responses to surveys targeted to public libraries, high school librarians, faculty and staff and high school students. These users were asked to rate access by both predetermined search categories; e.g., through the KYVL Gateway and by connecting directly to the online resource; e.g., native interface.
KYVL Gateway Responses
|
SURVEY USER GROUPS |
Ratio Responses--Good |
Ratio Responses--Excellent |
TOTAL |
|
Public Libraries |
43% |
23% |
66% |
|
High School Librarians, Faculty, and Staff |
42% |
36% |
78% |
|
High School Students* |
31% |
22% |
53% |
Native Interface Responses
|
SURVEY USER GROUPS |
Ratio Responses--Good |
Ratio Responses--Excellent |
TOTAL |
|
Public Libraries |
35% |
37% |
72% |
|
High School Librarians, Faculty, and Staff |
39% |
45% |
84% |
|
High School Students* |
28% |
17% |
45% |
* It was noted that the number of high school students’ responses were quite high because the survey had been a mandatory class assignment. Based on some of the written comments, these results may be suspect.
KYVL Database Usage Statistics
Obtaining usage statistics in a viable form has been extremely difficult for the KYVL staff. The change in the number of user communities has not been consistently applied by all vendors. Subsequently, for purposes of this Assessment, we will be looking at overall usage statistics for 2004-2005.
The Top Ten Databases are listed below. 63% of the total searches were conducted in these 10 of the 42 reported databases. The 43rd database: IIMP is not part of this process. Although after a great deal of dogged determination, KYVL did receive IIMP usage data in August, they appeared flawed. Subsequently, it will not be utilized in this assessment process.
While not listed, it should be noted that the 11th most used database was EBSCO Newspaper Source. 8% of the overall total searches are conducted in either Alt-Press Watch or Newspaper Source.
TOP TEN
FOCUS/DISCUSSION GROUPS
While survey results and actual usage statistics are helpful in an assessment process, there is always the need to have a more interactive exchange with those who use the databases. In past Assessments, we have employed both very formal focus groups and virtual e-mail focus groups to accomplish that task. With this experience in mind and the lack of human resources available to address this issue, the Assessment Committee determined that they would utilize the e-mail focus group again. The idea of the informal discussion group was also adopted by the Assessment Committee members.
Lexington Public Library Reference Staff conducted four discussions groups. Twenty five people participated in these informal sessions. The following are part of the summary of those sessions.
The most frequently cited complaints were problems with the KYVL interface. Complaints were that it was not precise or particularly intuitive. Speed was also cited as a problem with KYVL-many participants noted that it is often slow. Participants wanted to rely on full-text delivery: sometimes they thought that they could get full text articles, then they found that the ones they wanted were not available in full text. One participant wanted a better authentication process than the password system.
Among topics they felt could be better covered were environment/ecology, auto repair, practical advice for homeowners, do-it-yourselfers, library science, more pre-1980 periodical articles, Kentucky history, Spanish language materials, genealogy, easier access to KRS Statutes, religion, politics, law, periodical reference tools in both legal and business areas, current events, government statistics, newspapers, full-text consumer health including more bi-lingual, i.e., Spanish text articles available, local history of Kentucky counties, and local biographies of prominent county residents.
Among the recommended databases were Modern Language Association, O.E.D., those relevant to trade schools e.g., Mitchell and Chilton manuals for automobile repair, Library Literature, New York Times online, J-Store, NADA.com, Lexis-Nexis, LEGALTRAC, Sanborn Fire Insurance Maps, Kentucky Encyclopedia, Heritage Quest; AncestryPlus, “Cochrane database of health care reviews,” Reference USA, “History Reference Center (EBSCO) or History Resource Center (Gale) (U.S. or World),” ABC-CLIO history databases.
Northern Kentucky University also held a discussion group. The six participants spoke in favorable terms of the EBSCO products and of the option of searching through the native interfaces. They felt that coverage of history, humanities and engineering were weak and that newspaper coverage needed to be broader and deeper.
The group ranked preferred future databases by subject
They also had suggestions for next set of databases
KYVL Online Resources Customer Satisfaction Survey Follow Up
As a part of the Customer Satisfaction Surveys, respondents were asked if they would be willing to participate in an e-mail focus group. 71 of the respondents (9%) indicated that they would be willing to participate and provided contact information.
The KYVL Online Resources Customer Satisfaction Survey Follow Up Questions (Attachment B) were distributed to the participants who had indicated that they were willing continue. We received email responses from 25 individuals. While no specific details were captured about these participants, it appeared from their responses that the majority were librarians and educators from the K-12 and academic communities. Most of these users utilized the resources at least daily. Overall, they made positive comments about the EBSCO and WorldCat databases. A barrier that was identified was the KYVL interface with its “one flavor fits all.” They wanted more full text but some verbalized the realization that publishers could prove to be the sticking point. They wanted more sophisticated access with two respondents requesting a unified list of available journals with the source database and full text availability listed.
Here are excerpts from the responses to the question about types of information that they would be most likely and least likely to search the KYVL databases for:
Here are excerpts from the responses to the question asking if they usually found the information that they were looking for.
CONCLUSIONS
The qualitative assessment methods worked well and have yielded both interesting and helpful data. While deploying nine survey instruments seemed excessive, this change appears to have assisted respondents and subsequently gained us more usable data. The informal discussion groups were easier to manage than the formal focus groups and were as productive to this particular process. The e-mail focus group participants while small in numbers took the process very seriously and responded in depth.
The quantitative data was not as helpful during this assessment process as it has been in the past. Because of the change in user groups, KYVL staff had great difficulty obtaining meaningful usage data from the vendors. While we were not able to obtain detailed information, we did receive numbers of searches at the database level. It is of interest to note that the overall number of searches continues to escalate at the rate of 25% per year. Although numbers are not the sole indicator of success, it is a factor.
The qualitative data indicates that the users of this family of databases are happy with them; e.g., 77% of the respondents indicated that they felt that the databases were either Good or Excellent. The areas which respondents indicated were lacking continue to be the same as in the past: arts and humanities, history, medicine/psychology, education, careers, newspapers. While in the past and during the current assessment, Kentucky newspapers have and are frequently listed, it should be mentioned that Alt-Press Watch which was added in 2003 as a direct result of users expressed need for additional newspaper sources is currently the 3rd highest used database.
What have we learned? We need to retain access to most of the current family of databases and attempt to provide access to others which meet our users’ expressed needs. Simple, yes, easy, no.
ATTACHMENT A
KYVL ONLINE RESOURCES
CUSTOMER SATISFACTION SURVEY
FOR ELEMENTARY SCHOOL STUDENTS
Help us provide you with the online information resources that you need. Please complete the following survey and submit it to us.
Please rate these resources based on how successful you are in finding the information that you need.
|
Online Resource Name |
Poor |
Fair |
OK |
Good |
Excellent |
Do Not Use |
|
Searchasaurus Encyclopedia of Animals |
|
|
|
|
|
|
|
Searchasaurus General Encyclopedia |
|
|
|
|
|
|
|
Searchasaurus Middle Search Plus |
|
|
|
|
|
|
|
Searchasaurus Primary Search |
|
|
|
|
|
|
|
Searchasaurus Dictionary |
|
|
|
|
|
|
|
Searchasaurus Pictures |
|
|
|
|
|
|
|
Grolier New Book of Knowledge |
|
|
|
|
|
|
|
Grolier Multimedia Encyclopedia |
|
|
|
|
|
|
Overall, how would you rate the KYVL online resources that you use?
Which of the following subject areas would you like to be better covered in the KYVL online resources?
What subject areas other then those listed above would you like to see as a part of the KYVL online resources?_______________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
Thank you for your help!
Return completed form to
Myra Morton
Ky. Dept. for Libraries & Archives
P.O. Box 537
Frankfort, KY 40602
FAX: 502-564-5773
KYVL ONLINE RESOURCES
CUSTOMER SATISFACTION SURVEY
FOR MIDDLE SCHOOL STUDENTS
Help us provide you with the online information resources that you need. Please complete the following survey and submit it to us.
Please rate these resources based on how successful you are in finding the information that you need.
|
Online Resource Name |
Poor |
Fair |
OK |
Good |
Excellent |
Do Not Use |
|
Magazines and Articles—All topics—EBSCO |
|
|
|
|
|
|
|
Magazines and Articles—Computing ProQuest |
|
|
|
|
|
|
|
Magazines and Articles—Career and Technical Education--ProQuest |
|
|
|
|
|
|
|
Encyclopedia Americana--Grolier |
|
|
|
|
|
|
|
Multimedia Encyclopedia--Grolier |
|
|
|
|
|
|
|
Biographies--Wilson |
|
|
|
|
|
|
|
NoveList |
|
|
|
|
|
|
|
Searchasaurus Encyclopedia of Animals |
|
|
|
|
|
|
|
Searchasaurus General Encyclopedia |
|
|
|
|
|
|
|
Searchasaurus Middle Search Plus |
|
|
|
|
|
|
|
Searchasaurus Primary Search |
|
|
|
|
|
|
|
Searchasaurus Dictionary |
|
|
|
|
|
|
|
Searchasaurus Pictures |
|
|
|
|
|
|
Overall, how would you rate the KYVL online resources that you use?
Which of the following subject areas would you like to be better covered in the KYVL online resources?
What subject areas other then those listed above would you like to see as a part of the KYVL online resources?_______________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
Thank you for your help!
Return completed form to
Myra Morton
Ky. Dept. for Libraries & Archives
P.O. Box 537
Frankfort, KY 40602
FAX: 502-564-5773
KYVL ONLINE RESOURCES
CUSTOMER SATISFACTION SURVEY
FOR HIGH SCHOOL STUDENTS
Help us provide you with the online information resources that you need. Please complete the following survey and submit it to us.
Please rate these resources based on how successful you are in finding the information that you need.
There are two ways that you can search these resources after you click on the Find Books, Articles and More link. One is by a group of databases (online resources) by subject area. The other is to directly connect to the resource provider. If you use either/or/part of these, please respond where appropriate.
|
Search a Group of…Databases: Online Resource Name |
Poor |
Fair |
OK |
Good |
Excellent |
Not Applicable |
|
General & Reference |
|
|