Why your Copilot rollout disappointed everyone

Every article about failed Copilot rollouts cites the same American numbers. MIT: 95% of generative-AI pilots deliver no measurable business impact. Microsoft: 70% of the Fortune 500 are on Copilot. Gartner this, Forrester that. All true, all overseas, and none of it tells an Australian business what actually goes wrong in an Australian office.

There is a better data point, and almost nobody is using it — because it is inconvenient.

The Australian Treasury ran the experiment for you

Over 14 weeks in 2024, the Australian Treasury put Microsoft 365 Copilot in front of 218 of its own staff and, unusually, published what happened. Not a vendor case study. A government evaluation, with the numbers left in.

Before the trial, 75% of participants expected Copilot to help with at least some of their tasks. After using it for three months, 38% said it had.

The sharper end is worse. Before, 15% expected it to help with most of their work. Afterwards, 2%.

Read that again. This is not a story about people who never tried it. These are staff who used the tool for fourteen weeks and came out the other side thinking less of it than when they started. Treasury’s own explanation is the important part: expectations at the outset were unrealistically high, and — in its words — some staff were discouraged by the performance and gave up.

The tool did not get worse over fourteen weeks. The gap between what people were told to expect and what the tool actually did got wider, until they stopped bothering.

The failure was upstream of the software

Notice what is not in that explanation. Not “the model isn’t good enough.” Not “we picked the wrong product.” The trial went sideways for two reasons that had nothing to do with Microsoft:

Expectations were set by a sales narrative, not by reality. “Copilot will transform how you work” is a great line for a licence renewal and a terrible thing to put in a staff member’s head the week before they try to make it summarise a messy document. When the reality is “useful for some things, useless for others, and you have to learn which,” the gap between the promise and the Tuesday-morning experience does the damage.

Nobody was taught how to use it well. Treasury’s participants said it themselves — extra training would have been valuable. Handing someone a licence is not enablement. It is the beginning of the problem, not the end of it.

There is a small, brutal number buried in the same evaluation: to justify the licence cost, a mid-level public servant needed to redirect only about thirteen minutes a week to higher-value work. Thirteen minutes. And it still could not clearly show it had happened — because no shared way of working had been built to capture the gain.

Why this keeps happening everywhere, not just at Treasury

The Treasury trial is not an outlier. It is the most transparent version of the most common AI story in Australian business right now:

Leadership buys licences under pressure to “do something about AI.”
Access is switched on, with a link to some Microsoft help pages.
Expectations, set by the vendor, are sky-high.
Reality is uneven — brilliant at some tasks, hopeless at others.
People quietly conclude it is overhyped and drift back to the old way.
The renewal comes up and nobody can prove it was worth it.

At no point in that sequence did anyone build the thing that makes AI stick: realistic expectations, a shared standard for what to use it on, and the skill to tell good output from confident nonsense. The licence was the whole strategy, and the licence was never the hard part.

What a rollout that works does differently

The fix is not a better tool. Copilot, Claude and ChatGPT are all capable enough that the model is rarely the bottleneck. The fix is the part Treasury’s trial skipped:

Reset expectations honestly, up front. Name what it is genuinely good at and what it is not, so the first bad output is expected rather than disillusioning.
Train people on their real work, not on demos. The skill that matters is judging output and knowing which tasks to hand over — and that only develops on actual tasks.
Agree a shared standard for what goes into which tool, what gets reviewed, and what good looks like — so wins spread instead of staying trapped with one enthusiast.
Measure the thirteen minutes. Decide up front what “working” looks like and check for it, so the renewal conversation is evidence, not vibes.

None of that is a software project. It is a capability and change project, which is exactly why buying more licences never fixes a disappointing rollout — it just widens the gap Treasury measured.

If your Copilot rollout landed with a shrug, the honest diagnosis is almost never that you bought the wrong tool. It is that the tool was the only thing you bought.