Performance Based Research Funding (PBRF)

Evidence Portfolios and the PBRF
Research Outputs
Library Resources
Journal Rankings and Impact Factors
... Journal Citation Reports (JCR) database
... What is an Impact Factor
... To find an Impact Factor
... Other journal/article ranking measures included in JCR

Article "Times Cited" Information
H-Index
Other Evaluation Tools and Measures

Contact: Sue Foggin

Evidence Portfolios and the PBRF

The PBRF staff quality assessment process requires each eligible staff member to prepare an Evidence Portfolio (EP) detailing his/her research activities. The three main components of the Evidence Portfolio are:

• research outputs (e.g. publications and creative works; 70% of score)
• peer esteem (e.g. prizes, awards, citations of your work; 15% of score)
• contributions to the research environment.(e.g. leading a research team, mentoring postgraduates and visiting researchers; 15% of score)
• For definitions and fuller details of these and other PBRF components, read the material available here

For the 2012 PBRF assessment, research material must have been published or publicly disseminated in the period
1 January 2006 to 31 December 2011 to be eligible for inclusion.

For research published before 2006, reviews or citations (made within the assessment period) are eligible as peer esteem (PE) examples, and use of it (e.g. use of a methods paper or of software) is eligible as evidence of contribution to the research environment (CRE).

For more information on PBRF and Evidence Portfolios, see a range of information from the University here.

Research Outputs

The module allows each researcher to maintain details of his/her publications and other outputs in the Research Outputs section; and details of activities such as teaching, prizes, fellowships, can be added to the Professional Activities section, to provide documentation of Peer Esteem and Contributions to the Research Environment. A PBRF tab is also provided within Research Outputs for eligible researchers, to allow them to manage the PBRF process.

To log in to Research Outputs and for links to a range of help (e.g. the User Guide, brief online videos, FAQs), go here (login with NetID/Password).

For assistance with using Research Outputs, or to inquire about training, please contact Jennifer Graham (Research Outputs-Client Support); ext 84057 or email j.graham@auckland.ac.nz.

Library Resources

A range of library resources can help you to improve your Evidence Portfolio - potential uses include:

• identifying high-quality journals in which to publish
• providing measures of the quality of the journals in which you have published (e.g. high journal impact factor or other rating)
• generating information to demonstrate a particular article is of high quality e.g. high citation count and/or is being cited by top researchers or in highly ranked journals
• generating an H-index (indicates impact of your work, a potential Peer Esteem item if above average)
• generating information on impact of publications – a potential Peer Esteem item if you can demonstrate pre-2006 publications maintaining a consistent or high citation count. Note: can include pre-2006 publications; but you may only count citations within the assessment period (2006-2011)

Library databases can also help you find details of your publications which are not already in Research Outputs (or which have incorrect/incomplete details).

Information on some resources is given below; Philson Information Services staff are happy to help with specific queries.

Journal Rankings and Impact Factors

For PBRF it can be helpful to indicate the relative "quality" of the journals in which you have published. The Impact Factor, available via the Journal Citation Reports database, is one of the most common and accepted measures used to determine journal quality, however other measures are gaining popularity.

Journal Citation Reports (JCR) database:

Contains Impact Factors (IFs) for individual journal titles and is published in two editions - science & technology, and social sciences.

• 7,347 journals across 174 Science subject categories
• 2,242 journals across 56 Social Science subject categories

What is an Impact Factor (IF):

• Commonly used measure of a journal's "worth", helps identify the relative importance of a journal within its subject area - a high IF is seen as better than a low IF
• For a given year, a journal's IF indicates the average number of times an article published in that journal during the two previous years has been cited (i.e. 2010 impact factor data will be calculated from citations in 2008-2009)
• A Journal Impact Factor of 3.0 means that, on average, the articles published in that journal during the previous two years have been cited three times
• 5-year Impact Factors are also included
• Commentators have pointed out many limitations of the IF, e.g. see: Cartwright VA, McGhee CN. Ophthalmology and vision science research. Part 1: Understanding and using journal impact factors and citation indices. Journal of Cataract & Refractive Surgery. 2005.31(10):1999-2007.
• Looking at the Impact Factors of journals in a subject area may be helpful in deciding to which journals you should submit articles for publication
• It is important to note that what might be a high IF in one subject area (the JCR lists around 170 subject areas) could be considered low in another subject area, consequently IF should be used in context where appropriate

To find an Impact Factor:

• Go to the Journal Citation Reports (JCR) Connect page >> click Connect; select a JCR edition and year (you will usually want to look at the most recent).

Then:

• Select from the three options offered - the default is View a group of journals by subject category. This is often the best option - when you are looking at the IF of a specific journal, it is more meaningful to see it ranked with other journals on the same subject
• Click Submit
• At the Subject Category Selection screen, choose a category from the list; leave View Journal Data sorted by Journal Title
• Click Submit
• Journals in that category are listed alphabetically. The IF for each is shown as a number in the Impact Factor column (e.g. 2.341)
• Journals are listed 20 per page; at the top left is an indication of how many are included in that category - e.g. Journals 1 - 20 (of 84). You may have to go to the second, third or later page to find your title in the alphabetical sequence
• Once you know the IF for your journal, re-sort the list by Impact Factor - in the Sorted By dropdown menu, select Impact Factor and click Sort Again
• Scroll down the list to find your journal's impact factor, and where your journal is ranked - e.g. its IF may rank it 3 out of 21, or 26 out of 84
• You can re-sort by any column - e.g. re-sort by the 5-Year Impact Factor to see if your journal's IF has increased or decreased with a longer year range
• A journal may be in more than one subject category. Its IF will not change between categories, but its numerical ranking will be different within different categories

You may also compare a journal with all other journals in that JCR edition (rather than just with other journals in a subject category) - when you connect to JCR, choose View all journals from the three options, then follow steps above;

Or:

You may view a single journal, by choosing the third option, Search for a specific journal. This will give you only its Impact Factor, without the context of its ranking among other journals in the same subject category.

When citing the IF for a journal, it is often useful to include the IF year (some IFs change significantly from year to year – see the IF trend); and the subject ranking e.g.: 2010 IF = 3.54, subject rank = 5/76

For additional details of Journal Impact Factors, their limitations and factors to consider, and use of the Journal Citation Reports database, see the section starting on p. 12 of this Philson Library pamphlet.

The section starting on p. 15 of the same pamphlet gives some information on alternative measures.

Other journal/article ranking measures included in Journal Citation Reports:

Journal Immediacy Index

• The average number of times an article is cited in the year it is published - indicates how quickly articles in a journal are cited
• Can provide a useful perspective for comparing journals specializing in cutting-edge research

Eigenfactor Score

• Produced by a different organisation to JCR, but included in JCR for convenience
• Based on the number of times articles from a journal published in the past five years have been cited in the JCR year
• Also takes account of which journals have contributed these citations - highly cited journals influence the score more than lesser cited journals
• References from one article in a journal to another article in the same journal are removed, so Eigenfactor Scores are not influenced by journal self-citation
• For more detail, go to www.eigenfactor.org

Article Influence Score

• Produced by a different organisation to JCR, but included in JCR for convenience
• Determines the average influence of a journal's articles over the first five years after publication
• Calculated by dividing a journal's Eigenfactor Score by the number of articles in the journal, normalized as a fraction of all articles in all publications
• Roughly analogous to JCR's 5-Year Journal Impact Factor in that it is a ratio of a journal's citation influence to the size of the journal's article contribution over a period of five years
• The mean Article Influence Score is 1.00. A score greater than 1.00 indicates that each article in the journal has above-average influence, less than 1.00 indicates that each article in the journal has below-average influence
• For more detail, go to www.eigenfactor.org

Article "Times Cited" Information

The number of times a specific article has been cited can give some indication of that particular article's "worth" or impact (a Peer Esteem item in the PBRF assessment). Three databases which provide this detail are Scopus, GoogleScholar, Web of Science. Citation numbers may differ between these three databases, so we recommend you check all three. Note that recently-published articles are likely to have fewer citations than older ones.

There is a comparison of the three databases here. Brief instructions on using each of them follow. For more assistance, contact one of Philson's Information Services staff.

1. Scopus

Go to the Scopus Connect page >> click Connect

Scopus can be used to find a single known article, to generate a tablulated Citation Overview for a specific author, and to calculate a researcher's H-Index.

To find a single known article:

• In the Search box, type the complete article title; from the dropdown menu, choose in: Article Title; click Search
• From the resulting list (of one or more articles) find the one you want, check the number in its Citations column (at far right) - shows Scopus statistics on how many times the article has been cited by others since publication
• You can click on this number to view details of the citing articles

You can also search using various combinations e.g. surname and some words from the title; or the journal name and some keywords; etc.

To generate a tabulated citation overview for a specific author:

• Cllick the Author Search tab, in the Search box type the author details, click Search (Note: Click Search Tips for help with finding variations of an author's name, and distinguishing between authors with similar names; consider restricting by Author Affiliation to eliminate false hits; etc)
• Once you have a set of references as complete and accurate as possible for that author, click View Citation Overview to view the tabulated results
• This will also give an H-index (see The H-Index for more detail)

For assistance with using Scopus, contact one of Philson's Information Services staff. The Philson Library also provides a pamphlet giving much fuller details of the H-Index, its limitations, and how to use the Scopus and Web of Science databases to search for an author and calculate an H-Index.

2. Web of Science (WOS)

Go to the Web of Science Connect page >> click Connect

Web of Science (WOS) combines five citation databases: The Science, Social Sciences and Arts & Humanities Citations Indexes, plus two Conference Proceedings Citation Indexes.

WOS can be used to find a single known reference, to generate a Citation Report for a specific researcher, and to calculate a researcher's H-Index.

To find a single known article, use the Cited Reference search function:

• Click on the Cited Reference Search link (top of screen)
• Follow the instructions to enter author's name, the work's source, and/or publication year (see Notes section below for search details)
• Before clicking Search, check the Current Limits section and make selections on (a) which WOS databases to search - default is all; and (b) any date limits you want - default is all years
• Click Search

Notes on searching WOS:

• Author's names may appear as surname plus one initial (e.g. Dutton R) or surname plus first and second initial (e.g. Dutton RP); to get both, type dutton r or dutton rp. Use an asterisk (dutton r*) to find all R Duttons with any (or no) second initial
• For surnames with spaces, an apostrophe, hyphen or other punctuation, enter the surname both with spaces and as a single word e.g.: grant mackie ja or grantmackie ja to find J.A. Grant-Mackie
• Use alternatives when there is a good possibility that the surname may be incorrectly indexed
• All of these options may retrieve articles by different authors with similar names; put details in the Cited Work and/or Cited Year(s) boxes to increase search accuracy; or use the Author Finder feature
• In the Cited Work box, you must use the WOS abbreviated title - click the journal abbreviation list link to find this, then copy and paste it

After clicking Search, a list of one (or more) results appears.

Notes on results in WOS:

• The results list gives brief details of each article as it appeared in the work that cited it, including any errors – e.g. in the journal title, volume/issue number, pagination etc. So - the same article may be listed more than once with small variations (e.g. issue number instead of volume number); compare entries carefully to identify if two or three entries are in fact for the same article, include each of them to get the total number of citations
• Most entries in the list have a View Record link. If there is no link, there is no reference to that item in WOS - either because the reference was so incorrect in the citing article that WOS cannot recognise it; or it is a reference not covered by WOS (e.g. a reference to a book or a thesis)
• Click on View Record to see details (full journal name, other authors, abstract etc)
• Author's name in capitals indicates s/he was first author; three dots (…) in front of author's name indicates s/he was second, third etc author

When you find the specific reference you want:

• Check the check box to its left, then click the Finish Search button (check more than one box if you have identified more than one entry for the specific reference you want)
• The total number of works identified by WOS as having cited this reference is given, with details of each

To generate a Citation Report for a specific author:

• Search for a specific author by putting details in the search box; select Author from the dropdown menu; click Search (NB: see Notes on Searching for details on searching author names)
• The resulting list may include citations of one or more authors depending on how unique the name is and what search techniques you have used
• Combine the author search with an address search if too many 'false hits' occur; or view the Distinct Author Sets for that author
• Once you have a set of citations as complete and accurate as possible for that author, click Create Citation Report to view the tabulated results
• This will also give an H-index (see The H-Index for more detail)

For assistance, contact one of Philson's Information Services staff; the Philson Library also provides a pamphlet giving fuller details of the H-Index, its limitations, and how to use the Scopus and Web of Science databases to search for an author and calculate an H-Index.

3. GoogleScholar

Go to the GoogleScholar Connect page >> click Connect.

To find a single known article:

• In the Search box, type the complete article title; click Search
• Find the article you want in the resulting list. The Cited by... link gives GoogleScholar statisitics on how many times the article has been cited by others since publication
• You can click this link to view details of the citing articles
• For assistance, contact one of Philson's Information Services staff

The H-Index

"A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np – h) papers have ≤h citations each" Hirsch JE, 2005 [Np=total number of publications].

Designed to give an estimate of the importance, significance, and broad impact of a scientist’s cumulative research contributions, the H-index is based on both the set of a scientist's most cited articles, and the number of citations those articles have received in other people's publications. This is expected to improve on simpler measures such as total number of citations or total number of publications.

The H-index goes up as citations accumulate, so depends on the length of a researcher's career; it works properly only for comparing researchers in the same field. The index can also be applied to the productivity and impact of a group of researchers, such as a department or university.

The Research Outputs (RO) module provides an H-index for many researchers; however, it is unreliable, as it is based only on citations trawled from the Web of Science database by the RO search engine. If the citation in RO is obtained in any other way, it is not included in the RO calculation of the H-index for that researcher.

A researcher's H-index can be obtained more reliably using calculators provided on databases such as Scopus and Web of Science (see sections above). When using these, it is important to identify a comprehensive list of an individual researcher's publications in order to generate the most reliable H-index. This involves both searching for all variations a single name may have, and identifying/removing any articles which belong to another researcher with a similar name. Where a researcher's name has changed (e.g. upon marriage), articles published under both names should be searched and included.

The Philson Library provides a pamphlet giving fuller details of the H-Index, its limitations, and how to use the Scopus and Web of Science databases to search for an author and calculate an H-Index.

Other Evaluation Tools and Measures

Citation Impact Center from Thomson Reuters (producers of Journal Citation Reports). Blog, comments and articles on citation analysis

eigenfactor.org uses the same citation data as Journal Citation Reports, applies it in a range of ways

Harzing.com comments and papers on citation analysis, H-index, G-Index, plus Publish or Perish software

SCImago Journal and Country Rank uses citation data from Scopus; includes impact factors, or SCImago journal rank (SJR) indicators for journal titles not in Journal Citation Reports, e.g. NZ Medical Journal

Article-level metrics from PLoS - "...research articles should primarily be judged on their individual merits, rather than on the basis of the journal in which they happen to be published."

Journal Evaluation Tool for nursing and midwifery journals - from the University of Wollongong

Scopus Journal Analyzer - compare up to 10 journals indexed by Scopus, using a variety of parameters including the SCImago journal rank (SJR). To access, go to the Scopus Connect page >> click Connect >> click the Analytics link; to view a tutorial on this function, click Help > Tutorials > Browsing and Analyzing Journals.

New measures of scholarly impact - an article from "Inside Higher ED".