Role of the library in research evaluation

Jenny Delasalle, University of Warwick

This afternoon breakout session focused on what can a librarian do to use their expertise and knowledge to help support their institutions in research evaluation.

Jenny set the scene by outlining the Research Excellence Framework (REF) and explaining how REF 2014 ratings will range from 4*(outstanding) to U, and then moved on to discuss how institutions look for the impact of their research and the different metrics they employ.

Measuring the impact
There are many different ways an institution can measure the impact of research, including:

• Bibliometrics
• Outputs that can be counted and citations and calculations based on this
• Involvement as peer review
• Journal editorships
• Research grant applications and research income
• Prestigious awards
• PhD supervision load of staff

Citations still remain core to output measurement, but Jenny noted there are many different motivations behind a citation, including paying homage to experts to those likely to be peer reviewers; to lend weight to own claim; giving credit to peers whose work you have built on; providing background reading criticising or correcting previous work signposting under-noticed work or simply self-citation.

What else can we measure?
Jenny then covered a huge variety of output-based measures, across bibliometrics, webometrics and altmetrics, that can be used in addition to the traditional paper counts, Impact Factors and citations:

• H-index, calculated using number of publications and number of citations per output
• M-index, = h/m where n is number of years since first published paper
• C-index,  measuring the quality of citation
• G-index, which gives more weight to highly cited articles
• H-1 index, showing how far away a researcher is from gaining 1 more point on H index
• E-index, looking at surplus citations in the h set
• Contemporary H-index: recent activity
• Google’s i10-index, showing the number of papers with at least 10 citations
• Number of visitors
• Number of blog entries, likes, tweets etc.

A great deal of this information is available through library citation sources and data repositories managed by librarians, so librarians are ideally placed to advise on different metrics to help researchers and be the expert advisor for all bibliometrics.

Why measure outputs?
These outputs can be very valuable to researchers as well. Keeping a record of what a research has published is useful for his or her CV, for webpages that describe the work, as well as providing information to institutional data-gathering exercises.

Keeping an eye on who is citing the researcher’s work will help him or her identify future collaborators, maintain awareness of other research in their field and be aware of which articles are influencing their research profile the most.
 
The data can also underpin ideas for the researchers to tell a story to sell themselves:

• List articles published, with the citations for each, comparing with the average citations per paper over 2 years old.
• Establish if this is high for their discipline.
• Compare their article’s citation number with the journal average for that year
• List any outstanding individuals that have cited their work.

Other valuable metrics for the institution could be the number articles with no citation, or the number of joint articles (particularly good to identify levels of collegiality and interdisciplinarity)

Altmetrics and webometrics
Jenny gave two examples of publicly visible data for articles.  The PLoS website shows views, downloads, citations, bookmarks, likes, and tweets, giving authors more context and detail about their articles.  The Warwick Research Archive Portal (WRAP) similarly has publicly visible data in repository for every article.

Advice on gaining visitors
Jenny also shared some recommendations on getting more people to view researchers work.  Having more visitors to your paper will "boost your Google juice" so authors should put links to their papers everywhere you can, including Academia.edu, as well as getting someone to cite your paper (even in draft) as Google Scholar will pick it up.

The discussion was opened to the floor, with contributions being made around the following themes:

• Not all libraries are involved in researcher evaluation and, of those that are, some have started doing so on their own initiative where others are given a remit to do so.
• As information professionals, librarians should be involved, having excellent understanding of scholarly publishing and institutions. The library is the linchpin in an institution, and academics look to librarians for expertise and understanding.
• The importance of checking that Web of Science and Scopus links are correctly linking to researchers’ papers was noted, as well as working with the researchers themselves - they will recognize if their top paper is missing from their list.  
• Some pan-industry solutions for this include Project ORCID and the Names project, but none have come to fruition yet.
• Queries from researchers are extremely varied, with researchers wanting to help on deciding where to publish, how to get lists of Impact Factors, how to work out their H-indexes and where to get ideas for collaboration options.
• Jenny said that when she is approached by researchers for advice on collaboration options, she only offers a source that researchers can use to find options themselves, rather than providing actual selections
• It was felt that there was sometimes a lack of clarity over Shanghai Rankings and the point was made that universities should make clear to their faculties their expectations regarding rankings in terms of output.

Business Models

Ken Chad’s (@kenchad) breakout session on Business Models was a really useful plain English walk through everything any business or organisation should think about when shifting focus or approaching a new challenge. Devoid of jargon and full of references to business and marketing books and articles backing up his points, it was skilfully pitched to appeal to the multiple audiences attending the conference. I came away with some great tips I’ll tap into from time to time. Librarians could do the same when making strategic plans to face the future. Slides are here and give a very good taste of the session without overdoing the word count.


A helpful definition Ken used was “‘a business model describes the rationale of how an organization creates, delivers and captures value.” Crucially, for a UKSG audience, he explained that a business model “applies as much to a public sector organisation and not-for-profit, social ventures as to a a commercial company. To survive every organization that creates and delivers value must generate enough revenue to cover its expenses, hence it has a business model”.


He began by positing that organisations involved in scholarly communication face the challenge of relentless, disruptive, technology-driven change and tough economic times. Scary but best to face facts head-on! The start of any journey into developing a business model is to be clear on your organisation’s mission and the strategy. Sounds simple but getting these right is the cornerstone of the business model and probably often not given enough time or credence. He went on to explain that strategy is not goal setting but is ‘a cohesive response to an important challenge and that good strategy includes a set of coherent actions.’ 


Ken spent a bit of time on “the capabilities approach” and most useful to me was asking the question “what are the three to six capabilities that describe what we do uniquely better than anyone else?” Once determined, ask “can everyone in the organization articulate our differentiating capabilities?” and “is our leadership reinforcing these capabilities?”  To help determine capabilities he suggests we focus on value. “What’s valuable/special about what we do? Why should people use our products/services instead of alternatives?”


Towards the end of the session there was a helpful section on the building blocks of business strategy which incorporated the following six areas: Customer Segments; Value Propositions; Channels; Customer Relationships; Revenue Streams; Key Resources; Key Activities; Key Partnerships; Cost Structure. Boiled down, the key elements are about understanding your value propositions and how they seek to solve customer problems and satisfy needs and how successfully delivering on these value propositions creates the organisation’s revenue streams. 


I’d recommend taking 5 mins to skim the slides when you are next considering business models or need to focus on any of the building blocks. Slides are text-light and I think some key questions therein could produce useful food for thought and a way in to what can seem a daunting task to those who might consider themselves non-business types.


Finally, he has read a lot of books so you don’t have to. As I tweeted whilst at the session, I would not get time to do that much reading until I retire. The irony!

Identity before Authentication

Breakout Session - The Importance of Global Identity to Education led by Mark Williams

This session examined the many problems of a users identity, how that identity is established and how the user gains access to the protected resources they were looking for. The key theme was:

You shouldn't need a piece of string to navigate the internet

And yet in many cases the process of getting to a resource can be complicated and time consuming and many users get frustrated and give up.

At this point in the session Mark demonstrated the process of getting to an article through Shibboleth from a URL someone had sent to him in an email.

First click on the URL, select the PDF, get taken to a pay per view screen. Realise that you already have access and so select the tiny institutional login at the bottom of the screen. Skip over the username/password option and finally get to a list of countries, scroll to the bottom of this list, choose UK, enter details, website automatically takes back to the journal homepage and you have to find the article all over again.

He suggested ways that publishers can improve these processes. For example, how does the real-estate on your login screen divide? Is it a large individual login box, as in the example, with a small institutional option off to one side? Test how these choices affect abandonment throughout the process. Make the user journeys as simple as possible for as many users as possible. Remove the need for the string, journal sites should not be a labyrinth.

Publishers need to think about when login should be as well as where. Get sharing options to provide a one click to article experience. And provide clear logged out wording as well as dealing with login failures in clear language: we could not login you in because ...

Mark then went on to discuss the trend for portals and half-jokingly termed it:
One ring to rule them all

But he stressed that publishers cannot expect all users to be channelled into the same pathways, and that there is a need to offer sign-in at any point during the article or resource discovery process. As a way to start standardising this he referenced the recent espresso (Establishing Suggested Practices Regarding Single Sign-On) document published by NISO which provides crib sheets of best practice for publishers to consult during development programs.

Espresso is not just about authentication. It includes discussion around design of landing pages, discovery pages, protected pages, login pages (institution), and recommendations on rewriting open URLs, use of SP and error handling and branding.

Mark then opened the floor to discussions around things we would like from JISC - as a mixed room of publishers, librarians and vendors this proved to be an interesting and lively discussion. Key recommendations from the attendees spread beyond the remit of JISC but were very thought provoking, these included:

• A glossary of key terms and acronyms
• Error diagnostics - is the system working or is it me?
• A widget that is the best practice implementation and is downloadable and customisable for publishers and librarians
• Geo-location on drop-down menu on login form on JISC
• Subdivisions within institutions