jump to navigation

The HIP-index: A Better Measure of Research Impact November 16, 2013

Posted by Andre Vellino in Bibliometrics, Citation Analysis, Statistical Semantics.


Eighteen months ago, Xiaodan Zhu, Peter Turney, Daniel Lemire and I embarked on an experiment to see if we could identify the features in an article that would enable us to identify the critical (vs. incidental) references.  We thought that being able to identify references that are crucial would help us devise a better researcher productivity index – one that was better than the h-index.

I am happy to report that we were successful!  In September I gave an overview presentation to the U. Ottawa School of Information Studies that describes the problem we were trying to solve, our methods and results. Since then our paper has been accepted for publication in JASIST, most likely in a 2014 issue.

To automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper, we examined the effectiveness of a variety of candidate features – positional features, semantic features, context features and citation-frequency features – that might be predictors of the academic influence of a citation. We asked the authors of 100 papers to identify the key references in their own work and created a dataset in which citations were labeled according to their academic influence (note that this dataset is made available under the Open Data Commons Public Domain Dedication and License). We then used supervised machine learning to perform feature selection and found a model that predicts academic influence effectively using only four features.

The performance of these features inspired us to design an influence-primed h-index (the hip-index). Unlike the conventional h-index, the hip-index weights citations simply by how many times a reference is mentioned. We show that the hip-index has better precision than the conventional h-index at predicting ACL Fellows on a collection of 20,000 articles from the ACL Digital Archive of Research Papers.

P.S. (Nov. 18) Daniel Lemire in his related blog post gives the following credit, which I entirely share: Most of the credit for this work goes to my co-authors. Much of the heavy lifting was done by Xiaodan Zhu.

Learning from Watson February 19, 2011

Posted by Andre Vellino in Artificial Intelligence, Information retrieval, Search, Semantics, Statistical Semantics.

WatsonNow that Watson has convincingly demonstrated that machines can perform some natural language tasks more effectively than humans can (see a rerun of part of Day 1 of the Jeopardy contest), what is the proper conclusion to be drawn from it?

Should we join hands with “confederates” like Brian Christian and rally against the invasion of smart machines? (See his recent piece in the Atlantic and listen to his recent radio interview on CBC)?

Or do we conclude that machines are now (or soon will be) sentient and deserve to be spoken to with respect for their moral standing (see Peter Singer’s article “Rights for Robots“)? Or should we, like NSERC Gold Medal Award winner Geoffrey Hinton,  be scared about the social consequences (in the long term) of intelligent robots designed replace soldiers (listen to his interview on the future of AI machines on CBC’s Quirk and Quarks).

Before coming to any definite conclusion about how “like” us machines can be, I think we should consider how these machines do what they do.  The survey paper in AI Magazine about the design of “DeepQA” by the Watson team gives some indications of the general approach:

DeepQA is a massively parallel, probabilistic evidence-based architecture. For the Jeopardy Challenge, we use more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses….

The overarching principles in DeepQA are massive parallelism, many experts, pervasive confi-dence estimation, and integration of shallow and deep knowledge.

Is this the right model for creating artificial cognition? Probably not. As Maarten van Emden and I argue in a recent paper on the chinese room argument and the “Human Window”, the question of whether a computer is simulating cognition cannot be decided by how effectively a computer solves a chess puzzle (for instance) but rather by the mechanism that it uses to achieve the end.

In this instance DeepQA uses and combines a number of different techniques from NLP, machine learning, distributed processing and decision theory – which is not likely to be an accurate representation of what humans actually do but it is undeniably successful at that task (see this talk on YouTube about how IBM addressed the Jeopardy problem).

Geoff Hinton (in the radio interview mentioned above) speculates that Watson is a feat of special-purpose engineering but that the general-purpose solution – a large neural network that simulates the learning abilities of the brain – is what the project of AI is really about.

What we suggest in our Human Window paper is that one criterion we can use to determine whether machines are performing adequate simulations of what humans do is whether or not humans are able to follow the steps that machine is undertaking. On that criterion, I think it’s safe to say that Watson – although very impressive – isn’t quite there yet.

P.S. If you have the patience, I recommend watching a BBC debate from 1973 between Sir James Lighthill, John McCarthy and Donald Michie about whether AI is possible. The context of this video is the “Lighthill Affair” in 1972, recently chronicled on van Emden’s blog (note that the audio on this thumbnail video is rather out of synch!).

It’s amazing how spectacularly wrong an amateur in artificial intelligence (Prof. Lighthill was an applied mathematician specializing in fluid dynamics) can be about the possibiliy of machines simulating intelligent behaviour. It is real tragedy that Sir Lighthill’s ideological biases had such disastrous consequences for AI research funding in the UK. The attitude of Sir Lighthill reminds me of Samuel Wilberforce‘s objections  to Darwin’s theory of evolution. I find it astonishing that this BBC debate was so civilized in its demeanour.

SIGIR 2007 August 3, 2007

Posted by Andre Vellino in Data Mining, Information retrieval, Statistical Semantics.
add a comment

I’ve just returned from SIGIR 2007. As you’d expect, the participants and presenters at this conference this year were dominated by search-engine research labs: Google, Yahoo, Microsoft, Ask, Baidu.

There’s no doubt that the lure of financial reward is drawing the best minds in Information Retrieval away from academic institutions and into the big search-engine companies. One has to worry about this trend. Even though there is a lot of collegiality between individual researchers from different corporations, there are also market pressures that make novel IR techniques proprietary. This can’t be good for intellectual freedom or the advancement of knowledge.


The Meaning of Semantics June 17, 2007

Posted by Andre Vellino in Semantics, Statistical Semantics.

I worry that the use of “semantics” has become so ubiquitous as to be close to meaningless. For instance, a recent blog post on ReadWriteWeb claims that Hakia offers a practical, “semantics-based” solution to the information retrieval problem on the web.

Sometimes I think this use of “semantics” is intended merely to be synonymous with “better than syntax.” It must be the marketing department trying to commercialize (dumb down?) a complex notion. OntoSem (one of the tools on Hakia-Lab) has a respectable pedigree – even if you disagree with its premise. It is the brain child of Victor Raskin, founder of the Natural Language Processing Laboratory at Purdue and author of Ontological Semantics, who has a web site to promote the book and its ideas.


Human Assisted Search February 25, 2007

Posted by Andre Vellino in Search, Statistical Semantics.

I tried the human-powered “search with guide” feature on the ChaCha search engine the other day. I can’t see human-guided search becoming a business success story in the mass market – for most purposes searching is becoming suffiently easy that we don’t need help any more.

But the idea of having a human guide to help with sophisticated searches (which has been floating around in on-line libraries for a while) may work well in a scientific digital library where the help of a trained librarian or subject specialist could really be welcome. I’m optimistic that this kind of service will be offered in on-line science libraries both because my experience at ChaCha was quite good and because I believe there are situations where there aren’t likely to be automated alternatives.