technology – Jason Kitcat

It has been about 5 months since I last blogged on AI and so much has happened, there is a lot to ponder and learn.

In my specific context, which is relatively low volume but high complexity work (think thousands of transactions or processes not tens to thousands or millions) there are some specific things that we are feeling more and more confident about.

New capabilities
AI is enabling us to do things that weren’t previously possible, or makes the cost of doing them within reach. It is driving more in terms of capability, than it is in terms of efficiency. For example, on business.gov.uk we’re now able to show businesses government funding opportunities, personalised to their sector and region. This is made possible by AI tools aggregating and personalising what was previously disparate, unstructured data across multiple websites and sources. It would have been too difficult and expensive to do this before.
It’s all about the data
Data quality, infrastructure and governance are absolutely critical. Getting these right unlocks more value than anything else. These are difficult to build, hard to persuade people of their importance and slow to achieve. If you don’t get these right your AI work will be drawing on incorrect, out of date information.
What does sovereignty mean for you?
Managing sovereignty has many facets and many approaches. We are just at the beginning of thinking this through in a meaningful way. Does it mean who owns the models, or where the data is processed, or who has their hand on the ‘off’ button for the data centre? All of the above or none. Consensus is some way off, but discussions should be happening now in organisations, before it’s too late to make strategic choices.
Cost control
Understanding cost in this space is incredibly tricky with pricing changing fast, and token usage highly variable between models and use cases. Rigorously evaluating value for money is vital. Don’t try doing this on the back of an envelope, get the analytical professionals in to make sure you look at this robustly.
AI doesn’t mean LLMs for most people
When people say “AI” it’s not just about large language models, it’s about so much more. For most it is now a generic term that means automation, machine learning, modelling big data and general technology transformation. This is an opportunity for digital, data & tech people to influence at the top table, use it!
We still need to build right
Through all of this, we need to keep building in the right way. We still need multidisciplinary working that puts user needs at the core of our work that focusses on outcomes. Building secure by design remains as important as ever. This is still a team sport, and we’re still building critical systems which need to be trustworthy and reliable.

Those are my thoughts at the moment. What are yours?

Some thoughts on AI

“Anyone who tries to tell us they know the future is simply trying to own it.” — Margaret Heffernan in her introduction to Uncharted

Take a deep breath. Nobody knows what is going to happen with AI. You aren’t missing out, and lots is yet to be worked out, if it ever will be. Now is a time for learning and experiments.

It is absolutely incredible what AI models can achieve, and their capability has progressed significantly in the last few years. So I’m not suggesting they should be dismissed. Not at all. But that’s not to say I’m convinced AI will fundamentally change the structure of all public services and private enterprise. Here are my thoughts on why I think we need to be a bit more circumspect.

AI for coding doesn’t translate to all contexts

Much of the hype is being driven by the tech sphere getting excited by how helpful AI is proving in producing code. Techies are going to overvalue that capability, but being handy with coding doesn’t translate to being good at legal judgements or policy trade-offs.

There are a number of reasons why AI seems good at coding:

Code is fairly predictable;
There is lots of programming content online to train the models on (the Internet is full of coders!);
Compilers (the things which interpret code) provide helpful error messages which the models can use to adjust and correct their output with, providing a positive feedback loop.

There are not many (any?) other fields of work where these all apply. Some thought medicine would offer ‘low hanging fruit’ but as an example, despite radiology having the highest level of approved AI applications, radiology roles continue to increase in number.

There are also questions on how sustainable AI coding is for major projects. For hobby software, quick little scripts and prototyping, AI coding has fundamentally lowered barriers and empowered people, no doubt. But maintaining, testing, and iterating complex enterprise software is a different ball game where AI’s limited context window and unreliability become more challenging. If nobody actually wrote the code, who understands what it is doing? Cognitive debt is the new name for this challenge.

Without good data, AI can’t help much.

In my work, we spend lots of time thinking about how to achieve various policy outcomes in support of businesses and the economy. Many of our biggest blockers are around data quality and definitions. How do we identify specific types of businesses to do things with or for them? How could we define an economic sector or area of activity? Invariably, we don’t have data which directly tells us, so how can we infer what we need and what happens when it goes wrong? How does a company get included/excluded from the policy scope without creating a mass of bureaucracy for them and us?

AI and machine learning can help us around the edges of some of these challenges, but ultimately, we need high-quality, structured data first before we can apply any techniques or technologies. Getting there can require changing laws, service redesign, and data infrastructure. These aren’t really technology problems first and foremost.

How much does this cost compared to the alternatives?

While AI is currently becoming much cheaper in terms of cost per use, that’s only part of the picture. It is still expensive compared to many deterministic options such as automation platforms, which many will already have access to through their existing licensing. We also need more difficult-to-recruit skills to properly manage and deploy AI. If we compare it to the move to the cloud, I don’t need engineers looking after racks of servers in the basement anymore. However, I need a team of more expensive cloud engineers to manage the environment we depend on. So while we can do things not previously easily possible, we probably are seeing a greater total spend.

I keep coming back to the ‘missing’ productivity gains all our previous investments in technology are supposed to have given us. I don’t think the billions we spend on public sector technology are wasted, but they’re not a direct driver of cashable savings or productivity in most cases. However, we can do things we previously couldn’t have done: You can renew your passport with a selfie, identify high-growth overseas markets with a few clicks, and undertake mass-scale studies on anonymised NHS data. Amazing, but still we need to keep everything else going on top of the new stuff. AI is more stuff to buy, build, and run. It lets us do things not before possible, which is also amazing, but someone has to pay for it.

How reliable is good enough?

People make mistakes all the time, so we shouldn’t be holding a technology up to an unrealistic standard. But we’ve spent hundreds of years learning how to manage human error in our organisations and ways of working. AI makes mistakes in different ways, ones we are not really used to spotting and accounting for.

This academic post on reliability, using insights from the nuclear and airline industries, is incredibly helpful in explaining that for reliability, we don’t just need accuracy, we need four dimensions of reliability:

Getting things right consistently (Consistency)
Don’t fall apart when conditions aren’t perfect (Robustness)
Being open about lack of confidence rather than confidently guessing (Calibration)
When mistakes do happen, the mistakes are fixable rather than catastrophic (Safety)

What the academics argue is that while AI capability (accuracy) has been improving fast, the improvement in the conditions for reliability has been modest. And the worst issue is AI models aren’t good at knowing when they’re wrong. An AI agent’s degree of confidence in an answer does not correlate to the true accuracy. These are pretty fundamental issues that will need much more work.

AI Spam vs AI filters

In the meantime, I worry about race conditions – as people use AI to spam applications for government services, contact centres and so on; we’re going to put barriers up to getting through to the real decision-makers and doers. Who loses out in that scenario? I fear the genuine service user without access to the latest tools. Also, was the friction AI now overcomes in the process, actually part of the service design?

Looking internally within organisations: If a supplier’s agentic flow is going to seek payments on every contract variation it spots in email chains, our agentic flow will appeal each one and also try to catch the supplier on every breach of their service level agreements with us. We’ll then put agents on the escalations and before you know it, we’ll have an arms race.

We’re seeing this in recruitment where so many AI bots are causing havoc in the world of applying, reviewing, and interviewing. So far, the best countermeasure has been to insist on in-person interviews to prevent AI-assisted cheating on technical tests.

Where does this take us? Well, if I was selling AI usage, a race condition of AI spam vs AI filters sells more usage than ever. For the rest of us, we must think really carefully about what we need to own, control, and understand in this emerging world. I can’t see the value in all of us regurgitating AI-created text, which is AI-summarised and then AI-replied to. Writing is thinking; it’s not simply communications.

So where next?

I worry we’re just too easily impressed by the apparently fluency with which LLMs are conversing with us. We, certainly in the UK, tend to have strong biases towards good talkers. It’s a remarkable achievement to have machines able to do so, and on pretty much any topic you throw at them. Now is the time to keep experimenting and learning, as there is so much we still need to think about and understand. Questions I’m keen to explore centre on reliability, trust, accountability, and deeply considering what it is we expect from people in their work. We are living in interesting times, and we must remain open and curious, certain in the knowledge that the future hasn’t been determined; it is wide open for us to discover.