The HIP-index: A Better Measure of Research Impact November 16, 2013Posted by Andre Vellino in Statistical Semantics, Citation Analysis, Bibliometrics.
Tags: Citation analysis
Eighteen months ago, Xiaodan Zhu, Peter Turney, Daniel Lemire and I embarked on an experiment to see if we could identify the features in an article that would enable us to identify the critical (vs. incidental) references. We thought that being able to identify references that are crucial would help us devise a better researcher productivity index – one that was better than the h-index.
I am happy to report that we were successful! In September I gave an overview presentation to the U. Ottawa School of Information Studies that describes the problem we were trying to solve, our methods and results. Since then our paper has been accepted for publication in JASIST, most likely in a 2014 issue.
To automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper, we examined the effectiveness of a variety of candidate features – positional features, semantic features, context features and citation-frequency features – that might be predictors of the academic influence of a citation. We asked the authors of 100 papers to identify the key references in their own work and created a dataset in which citations were labeled according to their academic influence (note that this dataset is made available under the Open Data Commons Public Domain Dedication and License). We then used supervised machine learning to perform feature selection and found a model that predicts academic influence effectively using only four features.
The performance of these features inspired us to design an influence-primed h-index (the hip-index). Unlike the conventional h-index, the hip-index weights citations simply by how many times a reference is mentioned. We show that the hip-index has better precision than the conventional h-index at predicting ACL Fellows on a collection of 20,000 articles from the ACL Digital Archive of Research Papers.
P.S. (Nov. 18) Daniel Lemire in his related blog post gives the following credit, which I entirely share: Most of the credit for this work goes to my co-authors. Much of the heavy lifting was done by Xiaodan Zhu.