Getting computers to do what you want has never been easy

I remember as a youngster, eager to create my own version of the dungeon crawling game Rogue. Firing up the BASIC programming interface and meticulously typing out in plain English what I wanted the game to be like – the adventures, the rules, everything.
Only to be disappointed that despite my belief that I had been very clear with my intentions – all I got in return was an obtuse error message.
My father then redirected my enthusiasm towards a pile of old magazines where a variety of games (admittedly much humbler in scope) – had been detailed in the language and rules of the computer. What followed was a years-long journey to adapt my thinking and so that I could talk to computers on their terms.
But now, with generative artificial intelligence, You can just say what you want, and instantly, there’s a response and it can’t but feel a little magical.

The magic of immediacy

Consider the scenario of using a business intelligence tool to understand your year-on-year sales trends and there isn’t an already made report available. Historically you’d have to navigate the tool’s language and features or communicate your request to someone else, hoping they understand exactly what you need. Instead with advances we’re seeing in generative technologies, there is a far greater immediacy due to the ease of self-serve rapid iteration. In a world that (in my experience) favours speed over exacting precision, the ability to bypass the current paradigm of multiple layers of abstraction (humans or systems), without having to endure the associated delays is attractive.

Added to this is that there are few wholly-truly-new ideas under the sun – in my opinion. Nearly every thought or concept we have today overlaps with existing ideas or concepts. Having a system that can summarise and recall the accumulated years of thoughts and ideas becomes an immensely powerful tool for bootstrapping any knowledge work.

Whereas having to work through the current paradigms and abstractions, means a whole lot of groundwork before you even receive an initial answer. In some cases, the lack of an answer (however precise) makes it challenging to gauge whether the effort is worthwhile. Essentially, you’re paying upfront costs without knowing if the eventual answer will be valuable.

All of this is to say – I get why this is exciting development for the overwhelming majority of people who have been unable to “talk” to computers in a world that is increasingly defined by them.

But there’s no free lunch

In all this excitement, we must consider the implications of how the data that makes this all possible was sourced, and the future dynamics. Now that the genie has escaped the bottle, I can’t help but notice the subsequent clawing back of open access, an “Arms Race of Walled Gardens”, on the platforms where a significant amount of the data was sourced. Such as Reddit’s recent API policy changes limit how user-generated data can be accessed, making it more challenging to replicate the current approach and creating barriers to the development of alternative methods without substantial resources. My inner sceptic can’t help but see the beginnings of a self reinforcing pattern where the rich get richer.

Furthermore, the timeline of The Dead Internet Theory, which essentially posits that in the future, much of online activity will be driven by artificial actors conversing among themselves with relatively few genuine human interactions, has moved from “the future” to “right now”. We find ourselves at a juncture where genuine human-generated content is more valuable and crucial than ever before. However, it must coexist alongside a quagmire of generated and potentially biased content.

As a data person, there are some things I’d like y’all to know

In the past, garbage data was not only flawed in its output but often also in its form, but now given the ability to generate convincingly polished inputs built on dubious foundations, we have lost a heuristic for identifying suspicious data. To guard against this, we must take the effort to be deliberate in stating the expectations and validations of the data we consume and apply them as close to the origin of that data as possible, to minimise the blast radius of potential data pollution.

And while it may be very appealing both from a perception and value generation point of view to deploy these technologies at the external customer interface, there are some real considerations to be worked through. The threats are both malicious and benevolent in their origin, be it opportunistic actors actively attempting to violate your desired parameters or the technology itself inventing outputs that are unintentionally inaccurate or misleading, the consequences can be significant. Customers may act upon this information, leading to dissatisfaction or even harm. In both scenarios, your organisation may be held liable for any resulting damages or unmet expectations.

Generative AIs are an exciting new shiny hammer, so let’s not consider every problem a nail. There are well established ways of solving problems that while more constrained in their specification and more rigid in their inputs, are more reliable in their outputs. In terms of the total cost of operations, generative technologies have greater demands in terms of energy consumption and the skills required to run and monitor them in a production setting are scarce.
Specialised tools with a narrow and deep focus will tend to perform better than a generalist tool, again with fewer consumption and upkeep demands. Anand Subramanian shared a writeup where for a specialised task in the biomedical domain a comparatively simple Natural Language Processing model out performed Large Language Models even after they had been fine tuned (Building a Biomedical Entity Linker with LLMs). This is not to say that this would always be the case, however judicious solution development to solve the challenge and the scale thereof in an efficient manner – is the crux of value realisation.

Even when there is good cause for adopting Large Language Models, simply feeding it information you want it to “know” in the form of a preamble prompt is unlikely to hold up as you scale. It’s not a one and done sort of thing, some follow steps could be any of Retrieval Augmented Generation, Fine Tuning, or training your own model. 

However all these approaches are ill advised without first taking some time to model your world. What are the things that exist in your world and how do they relate to each other, and then applying that construct to the information that is fed into your LLM, is a necessary step to achieve material results.

This process of describing and cataloguing a domain is something that has been in practise for many years (e.g. Library Sciences,Resource Description Framework, The Semantic Web), however it hasn’t had it’s due from people not immersed in those world – so if you’re serious about this – it might be time to borrow from or hire those practitioners that have been working through these sorts of problems for some time already. The Analytics Engineering Podcast have a great episode that goes into so much more depth about the history and value applying structure to your information in light of the AI boom – AI’s Impact in the World of Structured Data Analytics (w/ Juan Sequeda, data.world)

Money, money, money

I’d be remiss to not touch on the money. For many of us, we’ve witnessed technologies break out of academic or controlled environments before, making a significant impact on the zeitgeist. Previous experiences remind us that while riding the wave of innovation can lead to success, not everyone is on top of the wave. It is tempting to believe that we won’t be amongst those crushed by the wave. However, it’s crucial to maintain a sense of proportionality in our eagerness to ride this wave. There’s a delicate balance between seizing opportunities and exercising caution.

Parting snark: prompt engineering

Prompt engineering is a cute way of saying “requirements elicitation” if y’all had put this much time in energy into explaining what you mean and why it is needed – I probably wouldn’t have had as much of a career operating as tech to business translator.
Benj Edwards authored The fine art of human prompt engineering, which if read as satire is a cathartic read which echoes my sentiment. In a way I’m glad that there has been an increased awareness about ambiguity and the need for specificity when seeking an outcome as this has been a large part of my career for many years. I have experienced in my own tinkering with LLMs that being deliberate in how you ask for what you want does make a material difference. If you’re really serious about levelling up how you prompt machines and humans, Gojko Adzic’s book “Specification By Example” is my go to reference.

Going forward I would hope that when I do have to act as a professional five-year-old asking why until I’ve just about frustrated everyone to the point of exhaustion, that will be a new appreciation for “why” and the ambiguity of words. Establishing consensus has never been easy, even in a world of magical auto-complete chatbots.