Loebner Contest makes Turing Roll in his Grave

I got up early this morning to watch a live streaming of the Loebner Prize contest, a formal enactment of a Turing test where human judges must hold chat conversations with humans and chat bots simultaneously, and try to determine which one is the human and which one is the bot. Hosted at Bletchley Park this year, the contest was divided into four rounds where a total of 14 entries were to be judged.

Unfortunately, there were several technical issues. The Loebner website was poorly laid out and I had a hard time finding the live streaming URL to begin with. Once I got there, I found the web streaming interface was rather ineffective. Conversations between one judge, one human and one bot were divided into panels of four unlabeled windows where characters appeared as they were being typed, making it difficult at first glance to see what part of which conversation was happening where.

There were also connection and text formatting issues, with some text never appearing, connections being broken, and some chat bots never producing newlines, which gave them away pretty quickly. Apparently, the people behind the Loebner contest chose to design their own chat protocol. Probably not a wise choice. They would likely have been much better served by a private IRC server.

The bigger problem, of course, is that most of the entries were absolutely terrible. It seems that none of the judges were "fooled" for more than a minute. Most bots were completely unable to follow a conversation in any way. Some kept asking random questions over and over. Most couldn't answer any questions from the judges, to the point where some judges decided to simply ignore the bots. It's rather appalling to see that even today, in 2012, several of these bots do worse than plain old Eliza. I'm fully aware that programming a competent chat bot is no easy feat, but it seems like this competition simply hasn't attracted the cream of the crop.

After the thrill of watching Watson spank humanity's ass on Jeoparty, I'm convinced that with modern technology we could do much better than this, if only someone just tried. I wish that the people at Cycorp would try and program a chat bot and show us what they can do with that fancy knowledge base and NLP software of theirs. I guess the truth is that there is too little market value for good chat bots, and the people who have the knowledge to implement a serious contender are busy doing something that earns them money. Turing is rolling in his grave right now.