Messing around with ChatGPT: The Future of AI or Just Hype?

It’s the year 2123, and at the World History Museum a holographic avatar explains how the 2020s were an inflection point for human knowledge. Since then, humanity has barely advanced, especially in creative fields. The cause — AI.

I’ve always been fascinated by the power of technology to unlock new worlds of possibility. I remember my excitement when I first used a computer, first connected to the internet, the sense of wonder that came with each new discovery.

This week I had that feeling again when I tried out ChatGPT. It’s an AI chatbot recently opened to public beta and built by OpenAI, a research institute funded by some big names in tech including Microsoft.

For the first time, I conversed with a computer using completely natural language. I was struck by the ability of Assistant—as the bot is named—to refer back to earlier parts of the conversation and modify its responses based on my feedback. It was like a glimpse into the future, where computers can understand and communicate with us in a way that was once unimaginable.

It also spewed out convincing but wholly incorrect answers with unwavering confidence. Detractors point out that it’s just a prediction engine generating plausible gibberish, but the initial experience is nevertheless uncanny.

As such language models start to be deployed and improved, what are the implications for society, and is it actually useful?

ChatGPT owes its name to the type of machine learning model used. The model was fed an enormous corpus of text from books, Wikipedia and the like, on which to perform undirected learning before being refined with supervised training.

Its stated aim is to “assist users in generating human-like text by anticipating the next word in a sequence, using a chat format that allows for follow-ups.” It evokes in my mind an image of some gargantuan smartphone text prediction engine but in practice, the results can seem eerily… human.

Taking this literally, it can “predict” the outcome of performing some tasks. Give it an instruction and you’ll often get a reasonable answer.

This needn’t be limited to munging text in English or other human languages: it can also code. I took a simple Bash script for backing up virtual machine files and asked the bot to convert it into Python. It turned out fairly well, here’s an excerpt:

(If reading code isn’t your thing you can just skip to the song below)

for file in find_output.stdout.split("\0"):
    vm = os.path.basename(file)
    vm_archive = f"{vm}.tar.gz"
    vm_archive_path = os.path.join(SOURCE, vm_archive)

    try:
        os.unlink(vm_archive_path)
    except OSError:
        pass

(In case you’re wondering why it swallows that exception, the original Bash used rm -f).

I only found one bug — on the last iteration of the loop, vm can be empty and an exception is raised. I asked for a patch-up fix:

On the last iteration of the for loop, the vm variable is empty. I think it’s probably an artefact of splitting. Please can you fix the bug and ensure that the loop exits when vm is empty.

Here’s the fixed version:

for file in find_output.stdout.split("\0"):
    vm = os.path.basename(file)
    if not vm:
        break
    vm_archive = f"{vm}.tar.gz"
    vm_archive_path = os.path.join(SOURCE, vm_archive)

    try:
        os.unlink(vm_archive_path)
    except OSError:
        pass

So what, we have an impressive coding tool right? But that’s not all…

Me:
Can you take the latest python code above and turn it into a love song

Assistant:
I’m sorry, but I am unable to perform that task as it goes beyond my capabilities.

Me:
Let me help you. First summarise in English what the code does. Then turn the summary into a few verses. Create a chorus with the most common theme and place it between the verses.

Assistant:
The code searches for a specific type of folder in a source directory, creates an archive of the folder, copies the archive to a destination directory, and then deletes the original archive.

Verse 1:
I’m searching through the files, looking for my love
A special folder, sent from above
I find it in the source, and I can’t wait
To create an archive, before it’s too late

Chorus:
I’m moving on, I’m moving on
To a new destination, where I belong
I’m moving on, I’m moving on
With my love, in an archive, it can’t go wrong

Verse 2:
I copy it to the destination, it’s all set
My love will always be, I won’t regret
I delete the original, it’s out of sight
But my love will always stay, through the night

Chorus:
I’m moving on, I’m moving on
To a new destination, where I belong
I’m moving on, I’m moving on
With my love, in an archive, it can’t go wrong

I rather enjoy the song about my backup script, and ChatGPT’s ability to summarise code and text is a real strong suit. But what really made me sit up and take notice can be summed up in one word: understanding. Even when it couldn't deliver at first, I was able to explain the steps in completely normal language and be comprehended. And who doesn't desire to be understood?

Explore further, and you will observe how ChatGPT reiterates information from its training data. Why not challenge it to expound poetically in the manner of the bard himself? Here’s a reworked version of my introduction:

In the year of our lord, two thousand one hundred and twenty-three, at the World History Museum doth a holographic avatar tell the tale of how the year two thousand and twenty was the turning point for human knowledge. Since that fateful time, the progress of humanity has all but ceased, particularly in the realm of creativity. The cause? None other than artificial intelligence.

Considering the implications of this, I can see how some writers might be fearful of language models and their potential to replace them, but I believe the truly talented ones need not worry.

While I’ve had some success using ChatGPT to make suggestions and rewrite parts of this article, micromanagement is often required. It’s good at summarising, but can suck the soul out of any writing style and reduce the text to an executive summary. Even if I request an imitation of well-known writers or styles, it gravitates towards clichés.

After excitedly fiddling with this new toy and trying to hit everything with the same hammer, my creative attempts have settled into this pattern:

Me: create something
Assistant: here’s something
Me: that’s not what I had in mind

I then go and create something entirely different, perhaps asking for alternatives to specific sentences. A glorified thesaurus no less. Then again, even rejected ideas can spark others.

No doubt an improved version is in the works. What then if the dystopian scenario were to come true? If authors were to simply lay down their pens and allow a machine to recycle existing knowledge? To rewrite all literature, rehash old memes and regurgitate song lyrics?

It’s time for a reality check.

Few individuals consistently generate truly original ideas. The ability to churn out drivel is not new, and while AI might do so on an industrial scale, this could actually serve to reinforce the value of original sources. I have faith that truly innovative works will continue to stand out.

The use of AI raises other concerns for society beyond the creative realm.

As I explored ChatGPT’s capabilities by asking for toy recommendations, it displayed gender bias by suggesting different options for boys and girls. Dollhouse anyone? This made me wonder how systems like ChatGPT might amplify biases already present in their training data.

Another concern is the potential for inappropriate or offensive content. At one point, I asked ChatGPT to rewrite a Kylie Minogue song in the style of Rage Against the Machine, and it outright refused. It’s not clear to me if the reason was copyright or offensive content (or just poor musical taste); either way it raises the question of censorship. Too much and we limit AI’s potential, too little and real harm can result.

Spam bots are another issue. ChatGPT itself cannot connect to other Internet resources, however developers could harness the available APIs or build their own.

In our spammy world, the ante would be upped considerably. Yet bots already exist along with countermeasures. In as much as ChatGPT is extremely powerful, initial caution is a good call. Overall I am cautiously optimistic that the problems and solutions will evolve together.

Assuming the sky isn’t about to come come crashing down then, just how effective is ChatGPT anyway?

It’s fascinating what you can get it to do. Lest this billion dollar investment in machine learning go to waste, I have used it to:

• Recommend music
• Tell and explain jokes
• Create a pub quiz
• Emulate an AmigaDOS terminal (based on this brilliant Linux version)
• Produce a counterfactual “what if Brexit hadn’t happened?”
• Summarise chats in various styles and formats (e.g. JSON)
• Divine the special qualities of the number 42
• Argue about why there is only one monopolies commission
• Pretend to be Yoda chiding a child for not crossing the road safely
• List UK prime ministers in various arbitrary ways
• Explain how to make a perfect cup of tea in zero gravity
• Write an Aesop’s fable about “the perils of using a microservices architecture without clear domain boundaries”

As for how it did on these tasks. Well, NASA may want to conduct further research before brewing an extra terrestrial cuppa:

Making tea in zero gravity can be a bit more challenging, as the lack of gravity means that liquids will not behave in the same way they do on Earth. Here are a few tips for making tea in zero gravity:

- Use pre-made tea bags or a tea infuser to avoid loose leaves floating around the spacecraft.
- Use a mug with a lid or a travel mug to prevent spills.
- Boil water using an electric kettle or a heating element, rather than trying to heat water on the stove.
- Use a straw to sip your tea, or secure the mug to a surface with a strap or bungee cord.

Listing prime ministers ranked by day of birth outright failed — remember it’s a text predictor, not a calculator. More disappointingly, it described 101010 as a palindrome — but fessed up and corrected its mistake when I protested. In an otherwise passable pub quiz, the UK’s opposition leader Keir Starmer is female. When I asked for a diff between code versions, it stubbornly repeated a spurious change.

There’s already plenty said online about its lack of accuracy, with many turning their backs in disgust. It’s not like every conversation with a human results in accurate responses though, right?

My initial verdict: treat it as a junior human assistant, use existing processes for review and it’s already close to being useful for some tasks. At the very least, it’s extremely entertaining.

What about the future?

Essentially ChatGPT is a proof of concept. One that human ingenuity has already convinced to perform all manner of parlour tricks. Yet until it gets connected, it probably won’t become a part of our daily lives.

For now, it is nothing more than a brain in a box.

That will surely change. In software engineering there’s already GitHub Copilot, also using OpenAI technology. Its first incarnation is a tuned variation on the general prediction concept, but it wouldn’t be hard to add a chat feature to request bigger refactoring and hook it up to real world tools like IDE language engines, diff, docker and, naturellement, GitHub. I can see it taking on an architect role too.

When those capabilities arrive, the nature of software development will undergo a profound change. Why not soon? Surely it’s just a matter of tooling and refinement at this point.

The challenge here for OpenAI— along with those building on their platform—is keeping up with expectations. As they refine and specialise, so people will come to depend on them for more advanced, critical applications with correspondingly increased demands.

It’s ironic to think of software engineers fretting about losing their jobs to a machine, but I don’t expect it to happen in my lifetime. Some might mourn the lost art of hands-on coding but I see it working more as a pair programmer.

The same argument will take place within many other industries that stand to be disrupted, and the eternal debate of machine replacing person rumbles on. One day, maybe, one day.

So is it all hype or, worse, is humanity condemned to eternal drudgery?

Yes, there’s considerable buzz around the release of ChatGPT. Justifiably so in my view, because for many it’s our first hands-on experience of a realistic conversation with a machine. As for our future, I think the world’s problems lie elsewhere. AI is simply a tool, and it’s up to those who wield it to determine whether it will be used for good or ill.

Oh, and in case someone asks a future chatbot to emulate quaint old prose from the early 21st century, I guess an archive of this blog will have been ingested into its model. So er, hi! 👋