Measuring research impact is a great way to understand research that is making a difference. It helps identify dedicated researchers who are doing work that matters. It opens up doors of promotion and tenure to deserving candidates. It is, in theory, a great way to predict the influence of a published article and is a valuable metric in merit-based funding distribution.
In this blog we will look at the ways research impact has been measured through the years, its evolution and modern-day relevant alternatives. One thing to keep in mind is that, even though there are more means to measure research impact now, besides the popular Impact Factor, the overall picture is still not complete. Research impact is a complex measure and even the current set of comprehensive metrics cannot give an accurate view.
A Brief History of Impact Measurement
The most important and popular research impact measure, the journal Impact Factor, has been around since 1955 when it was first proposed by Eugene Garfield, one of the founders of bibliometrics and scientometrics. Impact Factor uses the number of citations a journal receives for its articles, generally after two and five-year intervals, as the defining factor for measuring impact.
Initially created with the goal to help scholars find source journals, Impact Factor is now used by publishers and editors as a mark of prominence. Journals with high Impact Factors are considered prestigious and those with low or no Impact Factors less so. Getting published in a journal with a high Impact Factor is considered a mark of research quality and directly affects researchers’ promotions, tenure and funding opportunities.
However, Impact Factor has several shortcomings as we’ll see in the next section. Several metrics were developed as alternatives to Impact Factor in the last couple of decades. These include the Immediacy Index, g-index, h-index, Eigenfactor, etc. All of these alternative metrics, however, are also dependent on the citation count of the article or journal either directly or indirectly. And, this dependency, unfortunately, creates a very skewed picture of research impact as we’ll see below.
The Shortcomings of Impact Factor
Evaluation bodies like funding or tenure committees continue to rely on bibliometric research impact measures even though they know that these measures are inaccurate. Impact Factor, particularly, continues to be given huge importance despite its many shortcomings. Some of which are:
1.Impact Factor relies only on citations: This is true for bibliometrics as a whole, where most metrics are heavily dependent on citation analysis. While citations were a fair metric for gauging research impact earlier, using them as the ‘only’ metric for today’s scholarly publications is highly unjust.
For one, citations are not standalone — they are influenced by a lot of external factors. The pace and volume of research activity in a certain discipline, the age of the journal, how broad or niche the journal’s focus is — all of these, and more, influence the number of citations received.
Secondly, citations are measured over an extended period of time. In today’s fast-paced online scenario where researchers are expected to show the impact of their work quickly, waiting for citations to amass is not feasible.
And, finally, some publishers game the Impact Factor metric by asking authors to cite articles published in the journal earlier (self-citations) or by publishing more review articles that tend to get more citations. To learn all about citations, read The Fundamentals of Journal Citation.
2. Impact Factor is a journal-level metric: Research outputs are published in a variety of formats and via more varied mediums than ever. Impact Factor, however, is a journal-level metric and does not measure article-level impact or impact of research output published outside of scholarly journals. Also, only journals that are indexed in Web of Science (which has indexation criteria that cannot be met by all journals) are given an Impact Factor. Therefore, a huge pool of published scholarly content today cannot be measured by Impact Factor.
3.The positive feedback loophole: Because of the undue importance Impact Factor commands, researchers flock to journals with high Impact Factors. This leads to their impact factor staying high year after year. On the other hand, journals that are relatively new or lower on the Impact Factor scale have difficulty attracting quality submissions and continue to stay low and the disparity widens.
All these reasons clearly show that journal Impact Factor is not an accurate measure of research impact — it is, in fact, flawed in more ways than one.
Open Access And The Rise of Altmetrics
Most journals with high impact factors are owned by large corporate publishers. With the growing momentum of the Open Access movement, researchers looking for alternatives to traditional journals have to publish in journals that support OA, but may have low or no Impact Factor. This is a tough decision. By foregoing high impact journals, the onus is now on the scholars to prove to funders and their institutions that their research is having an impact.
Some funding organizations, such as the Wellcome Trust, that mandate OA publishing are also seeking alternative ways to measure article-level impact that don’t rely on citations alone.
This is where altmetrics come in. Altmetrics are alternative indicators of impact that attempt to measure the broader influence of scholarly content beyond the scholarly community. Because bibliometric only measures impact via citations, it cannot give an accurate picture of the real-world impact of an article, including how individuals and businesses are consuming it, how it is affecting public policy documentation, what is being said about it online, and what kind of discussions it is driving. Altmetrics attempt to address this lack of clarity by supplementing citation analysis with various non-traditional article-level metrics that are of high relevance now.
There are several companies and tools that help track altmetrics for scholarly publications. The most notable amongst these are PlumX, Impactstory and Altmetric.com. Altmetric.com is probably most recognizable for its data donut that shows up on many publisher websites.
Benefits of Altmetrics
Many publishers are implementing altmetrics alongside citation counts to give authors a more complete and relevant picture of how their content was received. For example, this is the kind of data Nature makes available to its authors via Altmetric.com.
PLOS also gives scholars a comprehensive report of their articles’ impact which includes both traditional and alternative metrics. It’s Article Level Metrics (ALM) Report tool lets you see article-level impact insights for any PLOS article.
Benefits for Scholars
A key benefit of altmetrics, and the reason for its rapid rise in popularity is its almost real-time nature. For scholars who had to wait for years to get the citation-based impact of their research, to now be able to see how their work is being viewed, shared, discussed and implemented in real-time can be extremely gratifying.
Imagine being able to see in real-time when an eminent scholar in your field mentions your research in a blog post or a group of researchers have a heated discussion over your article on Twitter. That’s the kind of feedback altmetrics makes possible.
Scholars can now see the impact of their work for themselves and present to decision-makers when making funding or job applications. With altmetrics enabling tracking of research and its dissemination beyond academia, it is now easier for scholars to prove the influence of their research and findings on governments, businesses, and public discourse.
Benefits for Publishers
Altmetrics are helping to level the scholarly publishing field to some extent. For small and medium OA publishers, going up against larger, corporate publishers that own most of the high Impact Factor journals is no mean feat. Besides, many small publishers are not even in the running, because to be eligible to receive an Impact Factor rating, you need to be indexed in Web of Science which has its own criteria for inclusion that many small journals cannot meet. Altmetrics, on the other hand, are quick and easy to access and only need a DOI to get started.
Besides using altmetrics to demonstrate impact, publishers are putting it to work in other ways. For instance, they use it to identify content and types of content in their journal that consistently do well and then use this information to drive editorial plans. Altmetric also serves as a great way for publishers to measure the result of their marketing strategies. Their marketing teams are now realizing the importance of online marketing towards broader dissemination of research outputs.
Publishers are also using altmetric data to compare their competitor’s performance against their own and devise growth strategies to match up or excel.
Getting Started With Altmetrics
For Publishers
Here’s what publishers looking to integrate altmetrics alongside their journal content online can start off with:
1. Explore PlumX, Altmetrics.com and Impactstory. Here’s a quick overview
2. Use the free Altmetrics bookmarklet to analyze your competitors’ and your own journal’s metrics.
3. Take a look at the altmerics used by larger publishers — Nature, Elsevier, Taylor and Francis etc.
4. Analyze your journal’s target audience and your author base to understand what metrics you need to track to make the insights gained useful for both you and your authors.
5. Reach out to the altmetric services mentioned above and explore the possibilities for your journal.
For Researchers
Researchers looking to harness the possibilities of altmetrics for their articles could start off with these simple steps:
- If you’ve already published research, you can see how it's performing with the Altmetric bookmarklet.
- Publish in OA journals that give readers direct and complete access to your work. This increases impact and reach.
- Market your own content — share it on social media, talk about it during conferences, contribute guest posts to further your findings. All of these will make their way back to your altmetrics numbers.
- Create a free ImpactStory profile to continuously monitor the impact of your research.
Altmetrics are not full-proof, but together with bibliometrics they can give a more holistic view of research impact. As the scholarly publishing landscape evolves and traditional print publications give way to online ways of research dissemination, we can expect research impact metrics to keep pace.
SciSpace is a platform built specifically for scholarly communications publishing. It helps you automatically typeset, format and convert to JATS XML, PDF, HTML, PMC XML, Crossref XML and just about any other publishing format within seconds and at a fraction of the cost of manual typesetting. Besides autoformatting, SciSpace has a slew of powerful features including plagiarism checker, integrated HTML publishing, article-level analytics, academic SEO and more. Most of these features are perfectly primed to help you boost your journal’s altmetrics.
SciSpace (Formerly Typeset) is a software developed to streamline publication workflows. Using SciSpace, publishers are able to automatically convert author submissions to any publication format, including JATS XML. SciSpace publisher offers solutions to generate XML-first articles that comply with major scientific databases like PubMed, Crossref, and Scopus.
SciSpace publisher works with over 400+ journals worldwide and offers solutions for generating XML-first articles. See how you can simplify your workflow and save over 80% of the time and cost of producing journal articles.
Explore all of SciSpace’s capabilities and book a free demo to see what we can do for your journal.
Before you go
Talk to us if you are targeting to improve your impact factor
Appendix:
If you found the above article interesting, the following blogs might also interest you.
