Contextual Turing Tests

Paul Raven @ 13-04-2010

The thing with the Turing Test is that it’s such a loose experiment; it becomes (arguably) much easier to pass if the intelligence being tested is tuned for the context in which the test is being performed, and to the human respondent.

Here’s a QED via web-culture cynic and curmudgeon Nick Carr, who points out that Ben Weber has taken a leaf from ELIZA’s playbook and created a chatbot for the Starcraft online games:

Once every thirty seconds, EISBot randomly selects a message from a pool of 75 messages and sends it to the console. The results can be quite convincing:

  • EISBot: Whats up with Savior recently?
  • EISBot: Tasteless is my homeboy
  • EISBot: It says I need more pylons, what a pylon?
  • EISBot: What to ally victory?
  • Player: damn you
  • Player: im not stupid
  • EISBot: Wanna join my clan
  • Player: dunno
  • Player: im not that great
  • Player: lol
  • EISBot: So, this game is pretty much over
  • Player: yeah
  • Player: lol
  • Player: im playing off race
  • Player: lol
  • EISBot: Quit wasting time, come kill me
  • EISBot: What kind of hardware are you using? I’m on a 486 with 16M ram, pure luxory!!!
  • Player: i dunno

Says Carr, from beneath a grubby flatcap, with a wet-eyed grayhound curled up at his feet:

Note that the bot’s one major flaw is that its command of the English language, particularly the use of punctuation marks, is much too sophisticated in comparison with that of the human. The sure way to distinguish the computer’s messages from the human’s is to recognize that the computer has a rather sentimental attachment to the apostrophe and the comma.

I take this as another indication that I am correct in my suspicion that when computers finally pass the Turing test it won’t be because computers have become smarter; it will be because humans have become dumber.

Oh, how right you are, Mister Carr. Why, until maybe forty years ago when those pesky computers came on the scene, young people were almost universally literate, and spoke in long erudite sentences when talking with their peers on matters of mutual interest! How the mighty have fallen…

… although, with that said, three cats and a catnip-dusted keyboard would probably be enough to pass the Turing Test if it were conducted in a YouTube comment thread. YMMV.


Loebner Prize winner doesn’t believe in Turing Test anyway

Paul Raven @ 13-10-2008

Yesterday saw Reading University here in the UK playing host to the annual Loebner Artificial Intelligence Prize event – a contest based around Alan Turing’s famous benchmark for artificial intelligence that can really think, namely whether or not it can successfully imitate human communications.

The bronze medal (for fooling a quarter of the judges) went to Elbot, a chat-bot program created by Fred Roberts, but Roberts himself seems to be not so impressed by Turing’s theory:

“I don’t think it’s anything like thought,” he said of Elbot’s conversational prowess. “If you know a magic trick, you know how it’s done, it’s not magic anymore. Sorry to be so pessimistic.”

With the caveat that I have no expertise in cognition or expert systems, I’m inclined to agree with him. [via The Guardian]