The BC Library Association has condemned it in writing. BC Archivist Myron Groover was polite but firm on “As It Happens”.

Members of Parliament for the Official Opposition Andrew Cash and Pierre Nantel gave the Heritage Minister a piece of their mind about it in the Canadian House of Commons:

Jim Turk, Executive Director of the Canadian Association of University Teachers (CAUT) gave a clear explanation of what’s at stake in an interview on Radio Canada International.

My question is – we’ve expressed our collective outrage at this Orwellian nightmare – and now what? Do we decide that Federal archival and library institutions are doomed and take on their role on the remaining islands of democracy or “…take arms against a sea of troubles, and by opposing end them”?

At the panel discussion in which I participated, I suggested that to predict the future one need only look at the past. To predict the iPad one needed only look at the Apple Newton (which died in 1998). What was the analog, I wondered, for an information retrieval tool, now dead and buried, that might still evolve into something we all want in the field of information management?

I proposed that the future of information retrieval might be something like an evolved Office Assistant, (affectionately coined “Clippy”) – the infamous, now deceased Microsoft Paperclip that assisted you in understanding and navigating Microsoft products.

My vision for a next generation Clippy was clearly not well articulated since it prompted the following tweet from Stephen Abram:

I think that Siri, (about which I posted a few years ago) belongs to the old Clippy style of annoying and in-the-way-of-what-I-want-to-do applications. I am surprised it has survived so long and was promoted by Apple so strongly. I predict it will join Clippy, Google Wave and Google Glasses on the growing heap of unwanted technologies that were not ready for prime-time.

Watson (who is now going to medical school, and about which I also posted a couple of years ago) is, however, just the sort of Natural Language Understanding component technology that I have in mind for for an interactive, personal information assistant. When a computer that now costs three million dollars with15 terrabytes of RAM can fit in your pocket and cost $500, a Watson-like system that understands natural language queries will be an important component of Clippy++.

What neither Watson nor Siri have – and this is what I foresee in my crystal ball is the most significant attribute about “Clippy++” – is personalization and autonomy. What will make true personalization possible with “Clippy++” is our collective willingness to accept the intrusion of a mechanical supervisor that learns from our behaviour about what we want, need and expect.

This culture-shift his happening right now – we gladly and willingly disclose our information consumption habits to supervisory software and data-analytics engines in exchange for entertainment and social networking. It won’t be long before we’re willing to do that for serious, personalized information management purposes as well.

The key, though, is going to be the interaction – the dialog that we have with Clippy++ – and it will have to have explanations for its actions and recommendations. That’s going to be the hallmark of its evolution to Machina Sapiens.

This is a slightly different prediction than the one made by the Economist in 2005: Death to Folders. Their argument was that folders as a method of organizing files was obsolete and that search, tagging and “smart folders” were going to change everything. My assertion is the very notion of a file – these things that are copied, edited, executed by computers - will eventually disappear (to the end-user, anyway.)

The path to the “end of files” is more than just a question of masking the underlying data-representation to the user. It is true that Apps (as designed for mobile devices) have begun to do that as a convenient way of hiding the details of a file from the user – be it an application file or a document file.  The reason that Apps (generally) contain within them the (references to) data-items (i.e. files) that they need, particularly if the information is stored in the cloud, is to provide a Digital Rights Management scheme. Which no doubt why this App model is slowly creeping its way from mobile devices to mainstream laptops and desktops (viz. Mac OS Mountain Lion and Windows 8).

But this is just the beginning.  There’s going to be a paradigm shift (a perfectly fine phrase, when it’s used correctly!) in our mental representations of computing objects and it is going to be more profound than merely masking the existence of the underlying representation. I think the new paradigm that will replace “file” is going to be: “the set of information items and interfaces that are needed to perform some action the current use-context”.

Consider as an example of this trend towards the new paradigm, Wolfram’s Computable Document Format. In this model, documents are created by dynamically assembling components from different places and performing computations on them.  In this model there are distributed, raw information components – data mostly – that are assembled in the application and don’t correspond to a “file” at all. Or consider information mashups like Google Maps with restaurant reviews and recommendations are generated as a function of search-history, location, and user-identity.  These “content-bundles”, for want of a better phrase, are definitely not files or documents but, from the end-user’s point of view, they are also indistinguishable from them.

Even, MS Word DocX “files” are instances of this new model.  The Open Document XML file format is a standardized data-structure: XML components bound together in a zip file. Imagine de-regimenting this convention a little and what constitutes a “document” could change quite significantly.

Conventional, static files will continue to exist for some time and version control systems will continue to provide change management services to what we now know as “files”. But I predict that my grand children won’t know what a file is – and won’t need to.  The procedural instructions required for assembling information-packages out of components, including the digital rights constraints that govern them, will eventually dominate the world of consumable digital content to the point where the idea of a file will be obsolete.

The inference? Be prepared for Yahoo to read your mind!

I have been a proponent of personalization since 2000, when I worked on developing “Personal Identity Management” services at Nortel. The idea at the time was (for a telecom company) to enable IP devices (routers / gateways) to track / manage / control your on-line identity and provide identity services (single sign-on, personalization of news services, etc.) to the user.

This was conceived at about the time that Microsoft Hailstorm was being launched. The only fundamental difference was – which service provider – “network access” vs. “operating system” vs. “third party service” – would be the trusted source for managing your identity.

From a public relations point of view Hailstorm and its successors Microsoft Passport, and Wallet, were a disaster. Invasion of privacy, identity theft, all the usual public anxiety buttons were pressed and Microsoft dropped a lot of these products – or at least gave them a makeover.

Yet, a few internet generations later, these ideas persist.  Google didn’t make a big PR campaign of it, but everything at Google is about personalization and localization as illustrated most graphically by the (dystopic?) Google Glasses video.

But – fortunately, I might add – I am noticing a (small) swing of the pendulum away from machine-learning, Netflix-style personalization towards a “how do you want it?” style of personalization.

For instance, Google News used to be fully and automatically biased towards your location. Since the summer of 2011, Google has given the end-user a great deal more control.

Marissa Mayer may want to read your mind, but I know that most people don’t want to have their minds read by machines. I think the trend towards great user-control will eventually spread to more personalization and recommender services. I hope so anyway.

My very first thought was “the future of conventional universities is in doubt“. This course alone had 42,000 registrants, 24,000 of which watched at least one video. Only 1,400 of the registrants got a “certificate of achievement” (i.e. completed the course and handed in all the assignments) but in the meantime there were 800,000 video-downloads of the courseware.

Distance-learning or on-line courses have been around for a long time – in the same way that “finger”, “who” and “chat” in Unix had been around a long time before Facebook, Linked-In and Instant Messaging.  The difference now is that major Universities are jumping on the bandwagon and offering them for free.  Why? Perhaps because of decreasing enrolment: free on-line courses are a way to recruit students from everywhere and to show them the best of what universities have to offer.

But also (in the US anyway), education is a business (see the Frontline documentary on the business of higher education: College Inc.)  That universities are feeling the financial pinch and being pressed by their boards to be more agressive in the marketplace was perhaps most visibly illustrated at the University of Virginia (the case against on-line education is elegantly articulated by Mark Edmundson – a professor of English at the University of Virginia – in a New York Times OpEd article).

Making courses on-line available for free will be a moneymaker when they start counting towards a degree, which clearly inevitable in the long run. However, I didn’t expect this development to come so soon after the beginning of the experiment. The Seattle Times reported just yesterday that the University of Washington is going to be offering some of their Coursera courses for credit.

Canada, in the meantime, has its own Canadian Virtual University which lists over 2,000 courses and 300 degrees and diplomas available on-line. The difference with Coursera is that the CVU is not free.

Anyone see any parallels with the publishing industry here?

Since then, there has been a gradual but significant shift in the function of Government research institutions in Canada. For instance, according to a presentation given to “Re$earch Money” by the president of the NRC, its Vision is:

To be the most effective research and technology organization in the world, stimulating sustainable domestic prosperity.

And its Mission is

Working with clients and partners, we provide strategic research, scientific and technical services to develop and deploy solutions to meet Canada’s current and future industrial and societal needs.

The first question that comes up with the Vision is: what is a “research and technology organization”? That phrase – “RTO” for those in the know – means something quite specific. It is a label for the set of things that includes such institutions as the Fraunhofer Institute and Battelle but also Finland’s VTT (“Business from Technology”) and Nato’s RTO.

Organizations like that do interesting things: they are catalysts for exchanging information, they set strategies, give advice, design new products, patent processes and bring mature ideas to commercial reality.  All of this is useful and important but it isn’t “basic research”, at least not in the sense of “advancing knowledge”.

So what is happening to basic research in government?  It is being outsourced to universities. The executive director of the Canadian Association of University Teachers (CAUT), James Turk, put it this way in an op ed column in the Ottawa Citizen a few months ago:

[Minster Goodyear] claims that [the NRC] no longer needs to [undertake basic research] because universities today play that role.

But, Turk also points out,

Many university-based researchers rely upon the NRC for their scientific work. By gutting the basic research program of the NRC, the government will be weakening university research.

Thus, from the government’s point of view, basic research should be an externality because it incurs long-term costs and no short-term benefits. By outsourcing research to universities long-term costs are downloaded to the provinces.

This was Nortel’s strategy too in it’s later years (~ 1995), and it was RIM’s as well (see also Canada’s Vanishing Tech Sector).

Consider this snapshot of the old UI:

The top third of the screen is reserved for image thumbnails that correspond to offerings in the highlighted service.  The remote’s navigation buttons change only the horizontal and vertical menu choices and the menus correspond to the categories of services available. [The top-level thumbnails are also accessible to get to the item directly.]

Admittedly there are some problems with this way of organizing the user’s entertainment options.  One is that the top level categories are not all the same kind of thing.  ”Internet” is a mode of delivery (which, of course, is also the mode of delivery for the rest of AppleTV content), whereas the others are descriptive of the kind of objects that are below the main menu item. What “Internet” means, clearly, is “other, non-apple applications”.  In addition, more recent AppleTV top-level menus also has the “Computer” category, meaning “Content streamed for your local computer running iTunes”, adding a second source-centered category.

However, at least the old interface makes some attempt at grouping content. Furthermore, the interface for the top-level navigation resembles in structure the navigation system implemented for each of the applications.  The interface has the consistency hallmark of Apple interfaces generally: learn the interface for one application and you know (more or less) how all the others behave.

Contrast this with the new interface.  In some respects, it is similar to the old one – thumbnails of content-images appear at the top of the screen, as expected and the content sources are more or less the same.

However, the artificial segregation by source or kind is eliminated altogether: all the applications on the same footing, iPad-App style.

The first serious problem starts manifesting when you scroll just one line down: the 1/2-page sized thumbnails disappear altogether.  Yet the selected applications (I bet) are still generating those thumbnails – you just can’t see them any more.

Right away, this gives screen real estate dominance to the first row of applications – Apple iTunes applications, naturally. Furthermore, you can’t go straight to the items in the thumbnails because you can’t see them any more.

The second major flaw comes from the mixed-mode cognitive models.  The first-level application-selection mode is (vaguely) iPad-like (without the ability to group apps, rearrange them or create screen-pages). However, once you’ve selected an application you’re back to the (more familiar and sensible) menu-navigation system.

What’s worse, though, is that the menu system for each application is now no longer consistent.  ”Movies” (short for “iTunes Movie Store”) has a Mac-style top-level menu-bar rather than a right-side menu navigation bar like all the other applications. Gone is the consistent Apple look-and-feel.

If at least the user had the ability to group applications as they see fit and to delete the unwanted ones (why not, the iPod/iPad allows that?).

Theres just no doubt about it.  Steve was right.

However, as Daniel Lemire points out in his latest blog post, the raw data of mere citations is pretty crude.  In any given article, it’s often hard to tell which of the (typically) dozens of references are “en passant” (to fend off the critics who might think you haven’t read the literature) or incidental to the substance of the article. What’s interesting for the authors of the articles being cited is the question “how citical is this citation to the author who cited me”?

One way to find out (and hence, perhaps, to build a better citation measure) is to train a Machine Learning algorithm to extract “key citations” – by analogy with extracting “key phrases” from a text (see Peter Turney’s 2000 article Machine Learning Algorithms for Keyphrase Extraction). As a starting point, we’d like to compile data from researchers which asks the question: “What are the key references of your papers?”

It will take 10 minute: please fill  this Google-documents questionaire. In it we ask you, as the author of an article, to tell us which 1, 2, 3 or 4 references are essential to that article. By an essential reference, we mean a reference that was highly influential or inspirational for the core ideas in your paper; that is, a reference that inspired or strongly influenced your new algorithm, your experimental design, or your choice of a research problem.

When this survey is completed, we will be releasing the resulting data set under the ODC Public Domain Dedication and Licence so that you can use this data in other ways, if you wish.

It is undeniable that there are real and serious problems with academic publishing (as pointed out very cogently by Fields Medalist Tim Gowers here,  John Dupuis (Head of York’s Science’s Library), here and Barbara Fister in the Library Journal here). And the Open Access movement is one I support. The concentration of control over journals by one for-profit publisher is clearly one of the core problems and the questionable practices (e.g. “bundling”) that they can consequently employ is another.

But who (or rather what) exactly is to “blame” (if that’s the right thing to do) for this situation? Elsevier is behaving rationally – from a market-forces point of view anyway. Maximizing profits is what any private enterprise does, particularly one that is publicly traded on stock exchanges. Elsevier (the publisher) is owned by Reed Elsevier which also owns Lexis Nexis (which offers law information and services) and Reed Elsevier Business (which provides data services, information and marketing solutions to businesses). Is this a portfolio mix that should be permitted by law? After all there are anti-trust laws that prohibit monopoly ownership in other domains.

One fundamental problem is that a public good (knowledge) has been comoditized, marketed and sold by a private, for profit enterprise. The officials within Elsevier who are in charge of the company don’t have a lot of room to manoeuvre if they are to comply with the stock-market forces that urge them to forever greater profitability.

Here’s a suggestion to the signatories of the Elsevier boycott: go to your pension-fund manager (university or government) and find out if any of the mutual funds, exchange-traded funds or stock portfolios they own have stock in Elsevier-Reed. I’m willing to bet they do. Preasure them to boycott those investments – I’m willing to bet that will have more influence.

Of course, the academic boycott has been heard, as evidenced by Elsevier’s open-letter reply of February 6th. That is one way to precipitate some kind of change towards greater openness of intellectual output. But lets not delude ourselves into thinking that this is going to address the root problem: the inadequate funding of publicly-owned channels of knowledge dissemination.

Instead, could we harness this desire for change towards lobying governments for more funding for university and independant open-access publishers (and tone down the rhetoric against Elsevier a little)?

P.S. I think it’s pretty important, for this post especially, to make it clear that these are my personal opinions (as are all my blog posts here) and in no way reflect the views of my employer.

From the Executive Summary.

Tuition Fees: they shouldn’t be frozen, but they also shouldn’t increase faster than inflation.  Freezing them would likely result in “further deterioration of the student experience — larger classes and less opportunity to debate and develop critical thinking skills”

Teaching vs. Research: “Increasingly, universities are letting professors sacrifice teaching commitments to conduct more research. There must be a better balance; excellent research should not trump excellent teaching.”

Overall Recommendations:

The current system is unsustainable from both a financial and a quality perspective.

The Commission recommends the following:

1. Contain government funding and institutional expenses;
2. Use differentiation to improve PSE quality and achieve financial sustainability;
3. Encourage and reward quality;
4. Revise research funding structures;
5. Maintain the current overall cap on tuition-fee increases, but simplify the framework;
6. Re-evaluate student financial assistance; and
7. Generate cost efficiencies by, for example, integrating administrative and back-office functions.

Now to expand on a few of these points (items 2 and 3 in particular).

2. Universities and colleges should not overlap in their functions (degree-granting) and programs (i.e. be more differentiated)

The division of roles between the college and university systems should include the following features:

• After two years of study, college students who meet specific academic achievement criteria should be able to transfer into the university system;
• Colleges should not be granted any new degree programs, but existing programs should be grandfathered;
• The government should approve no new PSE programs until existing programs are rationalized and mandate agreements completed;
• No new professional and specialized programs should be approved without a compelling business case; and
• The Colleges of Applied Arts and Technology should work with the College of Trades to optimize the delivery of apprentice training in non-degree programs.

3. Encourage and Reward Quality means, among other things, focusing on more rewards for teaching:

Resources and rewards should be refocused towards teaching: Post-secondary education institutions should devote more resources to experience-based learning such as internships, allow for more independent study, develop problem-based learning and increase study abroad. Universities should be encouraged to include in their collective agreements flexible provisions with faculty regarding teaching and research workloads. Top-performing teachers and researchers should be recognized with the appropriate workloads and rewards. Eleven Ontario universities already have such flexibility; others should follow. Institutions should redesign incentive systems to reward excellent teachers, as they do now for researchers.

Refocus provincial funding to reward teaching excellence: Provincial funding allocations should be linked to quality objectives, and the funding model should reward degrees awarded rather than just enrolment [sic] levels. Government and PSE institutions should work to ensure that the capacity to integrate ideas and create innovative solutions to problems is at the heart of the higher education experience. This will be critical to the economic and social success of Ontario, in an economy where graduates will be working over their career in ways that cannot even be imagined now.

In early September 2010 (I’m embarassed to count many months ago that was!) I received an Early Access (PDF) copy of “Mahout in Action” (MIA) from Manning Publications and asked to write a review. There have been 4 major updates to the book (now no longer “early access”!) since then and although it is too late to fulfill their purpose in giving me an early access to review (no doubt a supportive quote for the dust jacket or web site), I thought I’d nevertheless post my belated notes.

Mahout is an Apache project that develops scalable machine learning libraries for recommendation, clustering and classification. Like many other such software-documentation ”in Action” books for Apache projects (Lucene / Hadoop / Hibernate / Ajax, etc.), the primary purpose of MIA is to complement the existing software documentation with both an explanatory guide for how to use these libraries and some practical examples of how they would be deployed.

First I want to ask: “how does one go about reviewing such a book”? Is it possible to dissassociate one’s opinion about the book itself from one’s opinion of the software? If the software is missing an important algorithm, does this impugn the book in any way?

The answers to these questions are, I think, “yes” and “no” respectively. Hence, the following comments assess the book on its own merits and in relation to the software that it documents, not in relation to the machine learning literature at large. Indeed, the fact that this book is not a textbook on or an authoritative source for machine learning is made quite explicit at the beginning of the book and the authors make no claim at being experts in the field of Machine Learning.

It’s important to understand that Mahout came about in part as a refactoring excercise in the Apache Lucene project, since several modules in Lucene use information retrieval techniques such as vector based models for document semantics (see the survey paper by Peter Turney and Patrick Pantel “From Frequency to Meaning: Vector Space Models of Semantics“). The amalgamation of those modules with the open source collaborative filtering system (formerly called Taste) by co-author Sean Owen yielded the foundation for Mahout.

Thus, if  there are gaps in Mahout software it is an accident of history more than a design flaw.  Like most software – especially open-source software – Mahout is still “under construction”, as evidenced by its current version number (“0.5″). Even though many element are quite mature there are also several missing elements and whatever lacunae there are should be considered as an opportunity to contribute and improve this library rather than to criticize it.

One obvious source for comparison is Weka – also an open-source machine learning library in Java. The book associated with this library – Data Mining: Practical Machine Learning Tools and Techniques (Second Edition) by Ian H. Witten, Eibe Frank – was published in 2005 and has a much more pedagogical purpose than Mahout in Action. In contrast with MIA, “Data Mining” is much more of an academic book, published by academic researchers, whose purpose is to teach readers about Machine Learning.  In that way, these two books are complimentary, particularly as there are no algorithms devoted to recommendations in Weka and many more varieties of classification and clustering algorithms in Weka than in Mahout.

The Mahout algorithms that are discussed in MIA include the following.

• Collaborative Filtering
• User and Item based recommenders
• K-Means, Fuzzy K-Means clustering
• Mean Shift clustering
• Dirichlet process clustering
• Latent Dirichlet Allocation
• Singular value decomposition
• Parallel Frequent Pattern mining
• Complementary Naive Bayes classifier
• Random forest decision tree based classifier

The integration of Mahout with Apache’s implementation of MapReduce – Hadoop - is no doubt the unique characteristic of this software. If you want to use a distributed computing platform to implement these kinds of algorithms, Mahout and MAI is the place to start.

On its own terms, then, how does the book fare? It is fair to say – for the quotable extract – that Mahout in Action is an indispensible guide to Mahout! I wish I had had this book 5 years ago when I was getting to grips with open source collaborative filtering recommenders!

P.S. This book fits clearly in the business model for open source Apache software – write great and useful software for free, but make the users pay for the documentation!  Which is only fair, I think, since $20 or so is not much at all for such a wealth of well-written software! The same can be said for Weka, whose 303 pages of software documentation still requires the book to be useful.

Next year, I plan follow up with another Sciverse gadget for my citation-based recommender that uses the full power of Elsevier’s API into its collection content.

I want to commend all and sundry at Sciverse Applications for this initiative.  Opening up bibligraphic data and providing developers with a developer platform (a customized version of Google’s OpenSocial platform) is exactly the right kind of thing to do both to benefit third parties (they get access to anotherwise closed and proprietary data) and to enhance their own search and discover environment.

There are, already, several advanced and interesting applications on Sciverse. My favourites are: Altmetric (winner of the Science Challenge prize – see YouTube demo video below) NextBio’s Prolific Authors and Elsevier’s Table Download.

And there will be more to come. An open marketplace like this where the principles of variation and natural selection can operate will, I predict, make for a richer diversity of useful search and discovery tools than any single organization can develop on its own.

Now I am very interested in text, text retrieval (and music IR too) and I found the panel discussion most rewarding.  But it wasn’t aboutwhat I had been expecting it to be about (from the title) and I was perplexed by this use of the term “data” in this context. After all, the subtitle of the JCDL 2011 conference is “Bringing Together Scholars, Scholarship and Research Data”.  So the context for “data” was (for me) “research data” in the sense of the term that is pretty much the same the first 3 sentences of the Wikipedia entry for Data:

The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data (plural of “datum”) are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which information and then knowledge are derived.

So I was somewhat taken aback by the argument that ensued. Everyone, it seems (except me), was quite happy to speak of “Big data” and “large amounts of text” as synonymous.  As though the streams of bytes that are common to readings from an NMR spectrometer, digital music and electronic journal articles were in all significant respects indistinguishable.

Of course, large volumes of byte-sequences share some kinds of problems like storage, preservation and search. But “text data” is a different kind of beast, isn’t it? For one thing, text typically has meaning – cognitive content that is different from, say, music or images or spreadsheets of temperature variations in Glasgow over the past 500 years. It has more structure too, as evidenced by how efficiently it compresses and how (relatively) easy it is to search.

I’m happy to speak of data about text that is inferred by the act of mining text.  Word frequencies, ngrams, term clusters, sentiment categories etc. fit the definition of “data” above. Even the textual “meta-data” about text is data of a certain kind. But the text itself just doesn’t seem to be that kind of thing (qualitative or quantitative attributes of a variable).

Should we join hands with “confederates” like Brian Christian and rally against the invasion of smart machines? (See his recent piece in the Atlantic and listen to his recent radio interview on CBC)?

Or do we conclude that machines are now (or soon will be) sentient and deserve to be spoken to with respect for their moral standing (see Peter Singer’s article “Rights for Robots“)? Or should we, like NSERC Gold Medal Award winner Geoffrey Hinton,  be scared about the social consequences (in the long term) of intelligent robots designed replace soldiers (listen to his interview on the future of AI machines on CBC’s Quirk and Quarks).

Before coming to any definite conclusion about how “like” us machines can be, I think we should consider how these machines do what they do.  The survey paper in AI Magazine about the design of “DeepQA” by the Watson team gives some indications of the general approach:

DeepQA is a massively parallel, probabilistic evidence-based architecture. For the Jeopardy Challenge, we use more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses….

The overarching principles in DeepQA are massive parallelism, many experts, pervasive confi-dence estimation, and integration of shallow and deep knowledge.

Is this the right model for creating artificial cognition? Probably not. As Maarten van Emden and I argue in a recent paper on the chinese room argument and the “Human Window”, the question of whether a computer is simulating cognition cannot be decided by how effectively a computer solves a chess puzzle (for instance) but rather by the mechanism that it uses to achieve the end.

In this instance DeepQA uses and combines a number of different techniques from NLP, machine learning, distributed processing and decision theory – which is not likely to be an accurate representation of what humans actually do but it is undeniably successful at that task (see this talk on YouTube about how IBM addressed the Jeopardy problem).

Geoff Hinton (in the radio interview mentioned above) speculates that Watson is a feat of special-purpose engineering but that the general-purpose solution – a large neural network that simulates the learning abilities of the brain – is what the project of AI is really about.

What we suggest in our Human Window paper is that one criterion we can use to determine whether machines are performing adequate simulations of what humans do is whether or not humans are able to follow the steps that machine is undertaking. On that criterion, I think it’s safe to say that Watson – although very impressive – isn’t quite there yet.

P.S. If you have the patience, I recommend watching a BBC debate from 1973 between Sir James Lighthill, John McCarthy and Donald Michie about whether AI is possible. The context of this video is the “Lighthill Affair” in 1972, recently chronicled on van Emden’s blog (note that the audio on this thumbnail video is rather out of synch!).

It’s amazing how spectacularly wrong an amateur in artificial intelligence (Prof. Lighthill was an applied mathematician specializing in fluid dynamics) can be about the possibiliy of machines simulating intelligent behaviour. It is real tragedy that Sir Lighthill’s ideological biases had such disastrous consequences for AI research funding in the UK. The attitude of Sir Lighthill reminds me of Samuel Wilberforce‘s objections  to Darwin’s theory of evolution. I find it astonishing that this BBC debate was so civilized in its demeanour.

Mendeley, the on-line reference management software and social networking site for science researchers has generously offered up a reference dataset with which developers and researchers can conduct experiments on recommender systems. This release of data is their reply to the DataTel Challenge put forth at the 2010 ACM Recommender System Conference in Barcelona.

The paper published by computer scientists at Mendeley, which accompanies the dataset (bibliographic reference and full PDF), describes the dataset as containing boolean ratings (read / unread or starred / unstarred) for about 50,000 (anonymized) users and references to about 4.8M articles (also anonymized), 3.6M of which are unique.

I was gratified to note that this is almost exactly the user-item ratio (1:100) that I indicated in my poster at ASIS&T2010 was typically the cause of the data sparsity problem for recommenders in digital libraries. If we measure the sparseness of a dataset by the number of edges in the bipartite user-item graph divided by the total number of possible edges, Mendeley gives 2.66E-05.  Compared with the sparsity of Neflix – 1.18E-02 – that’s a difference of 3 orders of magnitude!

But raw sparsity is not all that matters. The number of users per movie is much more evenly distributed in Netflix than the number of readers per article in Mendeley, i.e.  the user-item graph in Netflix is more connected (in the sense that the probability of creating a disconnected graph by deleting a random edge is much lower).

In the Mendeley data, out of the 3,652286 unique articles, 3,055546 (83.6%) were referenced by only 1 user and 378,114 were referenced by only 2 users. Less than 6% of the articles referenced were referenced by 3 or more users. [The most frequently referenced article was referenced 19,450 times!]

Compared with the Netflix dataset (which contains over ~100M ratings from ~480K users on ~17k titles) over 89% of the movies in the Netflix data had been rated by 20 or more users. (See this blog post for more aggregate statistics on Netflix data.)

I think that user or item similarity measures aren’t going to work well with the kind of distribution we find in Mendeley data. Some additional information such as article citation data or some content attribute such as the categories to which the articles belong is going to be needed to get any kind of reasonable accuracy from a recommender system.

Or, it could be that some method like the heat-dissipation technique introduced by physicists in the paper “Solving the apparent diversity-accuracydilemma of recommender systems” published in the Proceedings of the National Academy of Sciences (PNAS) could work on such a sparse and loosely connected dataset. The authors claim that this approach works especially well for sparse bipartite graphs (with no ratings information). We’ll have to try and see.