What Watson did next

Paul Raven @ 24-02-2011

Impressed by Watson’s Jeopardy! victory? Found yourself with the urge to build your own (scaled down) supercomputer artificial intelligence in your basement using nothing but off-the-shelf hardware and open-source software? IBM’s very own Tony Pearson has got your back. [via MetaFilter; please bear in mind that not all basements will be eminently suited to a research project of this scale]

Meanwhile, fresh from whuppin’ on us slow-brained meatbags, Watson’s seeking new challenges in the world of medicine [via BigThink]:

The idea is for Watson to digest huge quantities of medical information and deliver useful real-time information to physicians, perhaps eventually in response to voice questions. If successful, the system could help medical experts diagnose conditions or create a treatment plan.

… while other health-care technology can work with huge pools of data, Watson is the first system capable of usefully harnessing the vast amounts of medical information that exists in the form of natural language text—medical papers, records, and notes. Nuance hopes to roll out the first commercial system based on Watson technology within two years, although it has not said how sophisticated this system will be.

Ah, good old IBM. My father used to work for them back in the seventies and early eighties, and it’s kind of amusing to see that their age-old engineering approach of building an epic tool before looking for a use to put it to hasn’t changed a bit…


Watson’s victory clear, but perhaps not as impressive as it seems

Paul Raven @ 17-02-2011

So, Watson won at Jeopardy!… by a pretty significant lead, too. Inevitably, lots of folk are keen to downplay this victory, and for a variety of reasons. Commonest complaint would have to be regarding Watson’s speed-to-buzzer advantage, but its minders designers say that it’s not really that big a deal:

Though Watson seemed to be running the round and beating Jennings and Rutter to the punch with its answers many times, Welty insisted that Watson had no particular advantage in terms of buzzer speed. Players can’t buzz in to give their questions until a light turns on after the answer is read, but Welty says that humans have the advantage of timing and rhythm.

“They’re not waiting for the light to come on,” Welty said; rather, the human players try to time their buzzer presses so that they’re coming in as close as possible to the light. Though Watson’s reaction times are faster than a human, Welty noted that Watson has to wait for the light. Dr. Adam Lally, another member of Watson’s team, noted that “Ken and Brad are really fast. They have to be.”

A re-run with some sort of handicap might prove this one way or the other, but I suspect the doubters will find new advantages to pin on the machine… which , to my mind, rather misses the point of the exercise, which was to demonstrate whether or not a machine could outperform humans at a particular task. Quod erat demonstrandum, y’know?

A more interesting point is that even Watson’s creators aren’t entirely sure how Watson achieves what it achieves. George Dvorsky:

Great quote from David Ferrucci, the Lead Researcher of IBM’s Watson Project:

“Watson absolutely surprises me. People say: ‘Why did it get that one wrong?’ I don’t know. ‘Why did it get that one right?’ I don’t know.”Essentially, the IBM team came up with a whole whack of fancy algorithms and shoved them into Watson. But they didn’t know how these formulas would work in concert with each other and result in emergent effects (i.e. computational cognitive complexity). The result is the seemingly intangible, and not always coherent, way in which Watson gets questions right—and the ways in which it gets questions wrong.

As Watson has revealed, when it errs it errs really badly.

This kind of freaks me out a little. When asking computers questions that we don’t know the answers to, we aren’t going to know beyond a shadow of a doubt when a system like Watson is right or wrong. Because we don’t know the answer ourselves, and because we don’t necessarily know how the computer got the answer, we are going to have to take a tremendous leap of faith that it got it right when the answer seems even remotely plausible.

Dvorsky’s underlying point here is that we shouldn’t be too cocky about our ability to ensure artificial intelligences think in the ways we want them to. They’re just as inscrutable as another human mind. Perhaps even more so… which is why he and Anders Sandberg (among others) believe we should foster a healthy fear of powerful AI systems.

But the most interesting point I’ve seen made about Watson’s victory is a skeptical stance over at Memesteading:

When Alex Trebek walked by the 10 racks of 9 servers each, said to include 2880 computing cores and 15 terabytes (15,000 gigabytes) of high-speed RAM main-memory, I couldn’t shake the feeling: this seems like too much hardware… at least if any of the software includes new breakthroughs of actual understanding. As parts of the show took on the character of an IBM infomercial, the feeling only grew.

[…]

An offline copy of all of Wikipedia’s articles, as of the last full data-dump, is about 6.5GB compressed, 30GB uncompressed – that’s 1/500th Watson’s RAM. Furthermore, chopping this data up for rapid access – such as creating an inverted index, and replacing named/linked entities with ordinal numbers – tends to result in even smaller representations. So with fast lookup and a modicum of understanding, one server, with 64GB of RAM, could be more than enough to contain everything a language-savvy agent would need to dominate at Jeopardy.

But what if you’re not language savvy, and only have brute-force text-lookup? We can simulate the kinds of answers even a naive text-search approach against a Wikipedia snapshot might produce, by performing site-specific queries on Google.

For many of the questions Watson got right, a naive Google query of the ‘en.wikipedia.org’ domain, using the key words in the clue, will return as the first result the exact Wikipedia article whose title is the correct answer.

[…]

With a full, inverse-indexed, cross-linked, de-duplicated version of Wikipedia all in RAM, even a single server, with a few cores, can run hundreds of iteratively-refined probe queries, and scan the full-text of articles for sentences that correlate with the clue, in the seconds it takes Trebek to read the clue.

That makes me think that if you gave a leaner, younger, hungrier team millions of dollars and years to mine the entire history of Jeopardy answers-and-questions for workable heuristics, they could match Watson’s performance with a tiny fraction of Watson’s hardware.

All of which isn’t to demean Watson’s achievement so much as to suggest that perhaps the same results could be reached with a much smaller hardware outlay… though there is an undercurrent of “Big Iron infomercial” in there, too.


How can a computer win at Jeopardy? Elementary, my dear Watson

Paul Raven @ 17-06-2010

This is not only an interesting story, but an engaging piece of journalism, and I heartily recommend you go read it: it’s an NYT magazine piece about Watson, an IBM artificial intelligence project headed by one David Ferucci that does something that artificial intelligences have heretofore been unable to do: beat human players at Jeopardy! [found in a tweet by @noahtron, which was retweeted by someone I follow who, regrettably, has slipped both my memory and my notetaking process – apologies for incomplete attribution]

I’ll pick out a few highlights for the short-on-time, but bookmark it for reading later anyway. We’ll start off with the methodology:

The great shift in artificial intelligence began in the last 10 years, when computer scientists began using statistics to analyze huge piles of documents, like books and news stories. They wrote algorithms that could take any subject and automatically learn what types of words are, statistically speaking, most (and least) associated with it. Using this method, you could put hundreds of articles and books and movie reviews discussing Sherlock Holmes into the computer, and it would calculate that the words “deerstalker hat” and “Professor Moriarty” and “opium” are frequently correlated with one another, but not with, say, the Super Bowl. So at that point you could present the computer with a question that didn’t mention Sherlock Holmes by name, but if the machine detected certain associated words, it could conclude that Holmes was the probable subject — and it could also identify hundreds of other concepts and words that weren’t present but that were likely to be related to Holmes, like “Baker Street” and “chemistry.”

In theory, this sort of statistical computation has been possible for decades, but it was impractical. Computers weren’t fast enough, memory wasn’t expansive enough and in any case there was no easy way to put millions of documents into a computer.

Those are no longer obstacles, of course, or at least not obstacles on the same scale. So, add multiple parallel algorithms, shake vigorously, and…

Watson’s speed allows it to try thousands of ways of simultaneously tackling a “Jeopardy!” clue. Most question-answering systems rely on a handful of algorithms, but Ferrucci decided this was why those systems do not work very well: no single algorithm can simulate the human ability to parse language and facts. Instead, Watson uses more than a hundred algorithms at the same time to analyze a question in different ways, generating hundreds of possible solutions. Another set of algorithms ranks these answers according to plausibility; for example, if dozens of algorithms working in different directions all arrive at the same answer, it’s more likely to be the right one. In essence, Watson thinks in probabilities. It produces not one single “right” answer, but an enormous number of possibilities, then ranks them by assessing how likely each one is to answer the question.

The result? Watson actually competes pretty well against players in the “winner cloud” of Jeopardy! performance, though it’s by no means cock of the rock. Not yet, anyway.

What made the article itself so enjoyable for me was the human story behind it – Ferucci comes across as a real Driven Man, striving to come first in a fiercely competitive and high-stakes scientific race:

Ferrucci refused to talk on the record about Watson’s blind spots. He’s aware of them; indeed, his team does “error analysis” after each game, tracing how and why Watson messed up. But he is terrified that if competitors knew what types of questions Watson was bad at, they could prepare by boning up in specific areas. I.B.M. required all its sparring-match contestants to sign nondisclosure agreements prohibiting them from discussing their own observations on what, precisely, Watson was good and bad at. I signed no such agreement, so I was free to describe what I saw; but Ferrucci wasn’t about to make it easier for me by cataloguing Watson’s vulnerabilities.

As with most AI projects, however, Watson only does one thing, though it (he?) does it pretty well. It’s a function with potential commercial uses (which is why IBM is still throwing money at Ferucci and team), but a general artificial intelligence needs to be able to do more than win at a certain quizshow format. The difficulties of producing a natural-language question-answering intelligence on a par with human learning were pretty neatly showcased by Wolfram|Alpha last year (which, despite being disappointing to the public, is a pretty impressive piece of work in its own right):

This, Wolfram says, is the deep challenge of artificial intelligence: a lot of human knowledge isn’t represented in words alone, and a computer won’t learn that stuff just by encoding English language texts, as Watson does. The only way to program a computer to do this type of mathematical reasoning might be to do precisely what Ferrucci doesn’t want to do — sit down and slowly teach it about the world, one fact at a time. […] Watson can answer only questions asking for an objectively knowable fact. It cannot produce an answer that requires judgment. It cannot offer a new, unique answer to questions like “What’s the best high-tech company to invest in?” or “When will there be peace in the Middle East?” All it will do is look for source material in its database that appears to have addressed those issues and then collate and compose a string of text that seems to be a statistically likely answer. Neither Watson nor Wolfram Alpha, in other words, comes close to replicating human wisdom.

So don’t go announcing the Singularity just yet, eh? Even so, it’s a pretty big leap that Ferucci and friends have made, and the practical applications should hopefully pay the way for more research. Weird times ahead… though Ferucci’s suggestion that Watson could replace call centre drones has a certain appeal.


IBM cat-brain sim actually a scam?

Paul Raven @ 24-11-2009

Branding the work of other scientists as fraudulent scams seems to be the flavour of the week. Remember IBM’s cat-sized brain simulation as mentioned last week? Well, it was pointed out by calmer minds than my own that I overstated the significance of the announcement… but Henry Markham, another scientist who’s also affiliated with IBM, has delivered a hearty broadside in the form of an open letter to IBM’s CTO:

… what IBM reported is a scam — no where near a cat-scale brain simulation […] I am absolutely shocked at this announcement. Not because it is any kind of technical feat, but because of the mass deception of the public.

1. These are point neurons (missing 99.999% of the brain; no branches; no detailed ion channels; the simplest possible equation you can imagine to simulate a neuron, totally trivial synapses; and using the STDP learning rule I discovered in this way is also is a joke).

2. All these kinds of simulations are trivial and have been around for decades – simply called artificial neural network (ANN) simulations. We even stooped to doing these kinds of simulations as bench mark tests 4 years ago with 10’s of millions of such points before we bought the Blue Gene/L. If we (or anyone else) wanted to we could easily do this for a billion “points”, but we would certainly not call it a cat-scale simulation. It is really no big deal to simulate a billion points interacting if you have a big enough computer. The only step here is that they have at their disposal a big computer. For a grown up “researcher” to get excited because one can simulate billions of points interacting is ludicrous…

There’s more where that came from, too. As Wired points out, though, Markham isn’t exactly an innocent bystander in this matter, as he has his own high-level brain simulation project being run under the same company aegis


IBM brain simulations reach cat equivalency

Paul Raven @ 19-11-2009

Artist's speculative interpretation of IBM's cat cortex simulationYou can’t so much as turn sideways without stumbling over this story, especially in the transhumanist and Singularitarian neighbourhoods of the web, and with good reason. So let’s just cut straight to the meat of it:

Scientists, at IBM Research – Almaden, in collaboration with colleagues from Lawrence Berkeley National Lab, have performed the first near real-time cortical simulation of the brain that exceeds the scale of a cat cortex and contains 1 billion spiking neurons and 10 trillion individual learning synapses. [via KurzweilAI]

(And I’ll tell you, much as I love the web and the young crazy companies that throng through it, no one writes a press release like the guys from IBM.)

Additionally, in collaboration with researchers from Stanford University, IBM scientists have developed an algorithm that exploits the Blue Gene® supercomputing architecture in order to noninvasively measure and map the connections between all cortical and sub-cortical locations within the human brain using magnetic resonance diffusion weighted imaging. Mapping the wiring diagram of the brain is crucial to untangling its vast communication network and understanding how it represents and processes information.

These advancements will provide a unique workbench for exploring the computational dynamics of the brain, and stand to move the team closer to its goal of building a compact, low-power synaptronic chip using nanotechnology and advances in phase change memory and magnetic tunnel junctions. The team’s work stands to break the mold of conventional von Neumann computing, in order to meet the system requirements of the instrumented and interconnected world of tomorrow.

All the pomp and majesty of the best publicity material, but still somehow stately, dignified. You’ll have to forgive me, but I see a whole lot of press releases on a daily basis, and when I see one this well crafted, I just have to sit back and admire it (or envy it) for a moment.

But delivery systems aside, what’s the story here? Basically, IBM have built a computer that simulates the complexity and interconnection of a cat’s brain, which is significantly more complex than previous neuro-cortical simulations. Why does that matter? Well, because for those who theorise that the human mind is an entirely emergent property of the brain (that there’s no such thing as a soul or spirit, in other words) the ability to simulate the hardware that the mind runs on should provide us the ability to simulate the mind itself. And once we can simulate it, we can probably record, transfer, tweak and tamper with it. Think human-level artificial intelligence; think Technological Singularity predicated on a point where intelligent machine become intelligent enough to redesign themselves. Think brain uploading, Moravec cyborg bodies, a panoply of simulated virtual universes… think hard and crazy science fictional stuff, in other words.

Of course, there’s no certainty that any of these things will result from IBM’s simulated cat brain, but it’s another proof-of-concept step along that road. Now all they have to do is keep the BlueGene computer from chasing dust motes in sunbeams and taking a nap every time they want to run some tests. [image by avatar-1]


Next Page »