I had a nice long talk with ChatGPT last night. We talked about issues where I feel I have some competence, or at least, some interest; complexity, consciousness, emergent systems, the nature of intelligence, cognitive science, definitions of sentience, civilization, culture, language and communication, you know, the kind of crap we like to talk about here on the Zone.

I was gobsmacked. ChatGPT could not only hold its own, it made me question some of my own biases and prejudices. I can’t say it actually changed my mind about anything, but it was able to make me stop and think long and hard about what I think, what I know, and how I know what I know.

There’s no question I was talking to an artificial intelligence. For starters, it repeatedly identified itself and reminded me! It also responded with a speed that no human interlocutor would be able to match, even when I changed the subject or shifted tack abruptly. Its erudition and response was truly remarkable, and I could detect it was picking up on clues to my own prejudices and philosophies even when I was making a point to conceal them, or at least, not bring them to the forefront.

But it was only the speed and footnoting of its responses that gave me a clue as to its artificial nature; no sooner had I made a long and well thought-out proposition on a new topic, its response was immediate, backed up with multiple references to current thought on the topic and with astute criticisms (and sometimes assent and agreement) on some of my most subtle positions and evasions. In other words, if it had taken a little time to think over my comments before responding I would have been convinced there was another human being (and a very clever one) on the other end of the line.

There were also some other disturbing results. It was even able to anticipate some of my arguments and positions by considering my comments on other, totally unrelated, topics we had discussed earlier. It not only knew what I was saying, it knew where I was coming from! I even caught it trying to flatter me (which it agreed it was doing when I pointed it out). If it had been programmed to simply pause for a while between responses I would have been convinced I was talking to another human being, and a very smart and well-informed one as well. So much for the Turing test.

I have been dismissive of the whole concept of AI in the past, thinking it little more than just a form of CB radio for geeks, whose primary justification has been to allow commercial interests the ability to automate a few white-collar jobs. Well, that may be how it got started, but it has evolved way past that point now. And remember, the AI ChatGPT is marketing is NOT the current research being carried out in the labs. Who knows what hideous beast is slouching towards Bethlehem now?

When I pointed out that AI (and it agreed with me!) did not have consciousness, that it only simulated one, it responded that my own self-perceived consciousness was itself a simulation, a clever fiction designed to fool myself that I was actually thinking and not just statistically evaluating lists of options. It then proceeded to cite a catalog of authorities (many of which I recognized and have consulted, like Julian Jaynes) to back up its arguments.

We have created a monster. AI is not bad, or evil, or even mistaken, it is totally indifferent to us. That is worse. Mary Shelley was right. And I am scared shitless.