Your AI projects probably aren’t ambitious enough

McKinsey’s latest State of AI report contains a gap that I find hard to ignore.

80% of respondents say AI has improved their individual productivity. Yet only 37% say AI is making a positive contribution to their organisation’s EBIT, and just 6% qualify as what McKinsey calls AI “high performers”, where AI contributes at least 5% of EBIT and its impact is considered significant.1

So AI is useful almost everywhere, but for most organisations that usefulness still isn’t making its way to the bottom line.

Frustrating, as that doesn’t match what we’re seeing at Icana. AI has created enormous value inside our own company, and we’ve seen what it can do for customers when it’s applied to a sufficiently valuable problem. And while reducing labour can obviously create ROI, I’m particularly interested in what we can achieve without making layoffs the business case: doing things dramatically better, increasing capacity, building things that previously weren’t economical, or creating entirely new products and services.

I don’t think the main constraint is the capability of current AI models. They still have plenty of limitations, some of them serious, but they’re already capable enough to create much more economic value than that 37% suggests.

From our experience, when that value isn’t showing up, the problem is usually in one or more of three places:

1. The project isn’t ambitious enough.
2. The organisation doesn’t yet have the AI skills to solve it.
3. The solution is built around the wrong AI models, tools or architecture.

1. Shoot for the stars: start with something unreasonably valuable

We’ve all been trained to make technology projects realistic. Find a contained problem. Reduce the scope. Show a quick win. Don’t ask for too much money before you’ve demonstrated value.

That used to be sensible advice. Not anymore. In the age of AI you should do the opposite.

Instead of starting with what you think AI can do, start with an outcome that would unquestionably matter to the business and work backwards. The following are not just random examples; our team has been part of each:

• If there’s a bottleneck in a logistics process, don’t start by asking whether AI can automate the troublesome step. Ask what it would take to double throughput.
• If your next ERP renewal is going to cost another fortune, don’t just use ChatGPT to negotiate a better deal. Ask whether you still need the ERP. Could you build your own mini-ERP that only has the functionality that you need?
• If your marketing team is using AI to research keywords and improve website copy, go further. What would it take for AI to run a campaign: research the market, develop the concepts, create the ads including video, put them into market, analyse what worked and adjust the next round?
• If your developers are using AI to fix bugs, ask why you’re waiting for a customer to find them. Could it watch the logs, spot warnings and errors, reproduce the problem, write a test and make the fix?

You probably can’t do all of that reliably today.

Good.

Now you have an interesting project.

When you shoot for the stars, you’ll often not reach them. Maybe you got to Mars, no problem. You’ve learned what AI can’t do reliably (yet), and where you still need human input. Now, work backwards. Find the parts current AI can do well, the parts where it needs help from deterministic software, and the points where you still need a human making the decision.

McKinsey’s data gives me some confidence that this isn’t just our experience. Its small group of AI high performers are 3.3 times more likely than everyone else to be aiming for fundamental business transformation, and nearly three quarters say they’ve redesigned workflows around AI. Among everyone else, only about a quarter have.1

They aren’t just doing more AI projects. They’re being more ambitious. They’re shooting for the stars.

2. Find the people who can actually build it

Once you’ve picked an ambitious problem, the next constraint appears fairly quickly: there simply aren’t that many people yet who know how to squeeze the maximum out of current AI.

Most organisations have capable IT teams, but AI is its own discipline. A great software engineer hasn’t necessarily spent years working with frontier and open-weight models, evaluations, agent harnesses, retrieval, fine-tuning, inference costs or multimodal systems. They may not particularly want to either.

At the other extreme, being good at prompting ChatGPT isn’t enough. Nor is being able to vibe-code an impressive prototype if you don’t understand architecture, scaling, security or what happens when your AI behaves differently on the 1,000th try.

The people I’d look for are the ones who have product and software skills and have been living and breathing AI and, importantly, have tried to make it work on real problems. Even better if they can combine it with machine learning experience from before LLMs. They know that getting a model to do something impressive once and building a system you can depend on are very different achievements.

Those types of people are really hard to find. We have some, and we might be able to help you (more on that later).

In the meantime though, work with what you already have. Look for the people in your organisation with a true passion for AI. The tinkerers, the ones that run their own personal agents at home. Some will be in technology, but certainly not all of them. Every company seems to have a few people who have gone much further down the AI rabbit hole than their job title would suggest.

Put them together and see what they can do as a team.

3. Wrong agent for the job

An “agent” is an LLM with a harness, tools and a level of autonomy to work on a goal provided by a human (or another agent).

For a serious implementation you may need to choose between different frontier models, smaller models, open models running in your own environment, or several of them doing different jobs. Then there’s the harness around them: what context they receive, what tools they can use, how much autonomy they have and when a person needs to step in.

And some parts shouldn’t use an AI model at all.

If a problem has a deterministic answer, write deterministic software. It’s cheaper, faster and easier to test. AI coding tools have also made that software dramatically cheaper to produce, which changes the economics of building bespoke systems. Nearly a third of organisations in McKinsey’s survey say they’ve already decided against buying at least one software product or feature because they could build it internally with AI coding tools.1

The best systems we’re seeing increasingly mix all of these things. AI where you need language, reasoning or messy inputs. Ordinary software where you want certainty. Humans where the consequences justify one.

And then evaluations around the whole thing, to validate the efficiency, costs and security of the solutions.

What we’re doing about it

All of this brings me back to McKinsey’s 37%.

I’m genuinely passionate about moving that number. There is too much capability sitting inside current AI systems for most organisations to still be struggling to show a financial return from them.

And we have a somewhat unusual position at Icana.AI. We’re a product company with a team that spends its time solving applied AI problems in production. That gives us experience across models, modalities, evaluations, agents and the less exciting engineering needed to turn them into something reliable.

So we’re going to make some of that team available to other organisations.

I want to be explicit here: Icana.AI isn’t pivoting into consulting. We have no ambition to build a large consulting practice, rent out developers by the day or spend the next eighteen months producing someone’s AI transformation strategy. We’re a product company and intend to stay one.

We’ve called the initiative Icana Proof.

The idea is fairly simple. Bring us an ambitious AI problem where solving the difficult part would create real value. We’ll agree upfront on what success looks like, test it on your data and build a proof of concept around the hardest part.

If we can’t prove it works, you don’t pay the proof fee.

If we can, we’ll show your team how we did it and hand the work over. They can build from there themselves, with us available if they need more help.

That last part is important to me. The goal isn’t to make an organisation dependent on Icana.AI. It’s to use the experience we’ve accumulated to get them over a difficult technical hurdle faster, and transfer as much of that experience as we can while doing it.

1. 1Test weekWe agree the test, the data and the bar with you.
2. 2Build, four weeksWe build against the test, with a weekly written update.
3. PassYou pay the proof fee and your team takes it from there. MissYou pay nothing more and get a written post-mortem.

Final thoughts

If I were trying to get serious ROI from AI today, I’d find the most AI-pilled people in the organisation and put them together. I’d include strong technology people, but I’d explicitly look beyond IT for people who understand the business problems and have been experimenting with AI themselves.

Pick an ambitious idea. Get the team to shoot for the stars.

When they get stuck because they’re trying to do something they haven’t done before, get help from people who have.

Feel free to contact us, or any other team that has done it before.

But most importantly:

The goal isn’t to put more AI into your organisation.

It’s to get more value out of what AI can already do.

Erik van Eekelen