Does Counting Prompts Tell You Anything About Performance?
What Has a Business Learned When Its Employees Double Their AI Usage?
It has learned that its employees are using the measured tools more often. It still needs to discover what happened to the work. A team that doubles its prompts might be completing twice as many good assignments. It might also be asking the same question twice because the first answer keeps failing. Both teams can produce an identical upward line on a dashboard.
The distinction became tangible on 10 November when Financial News reported on A&O Shearman’s use of anonymised data to track AI adoption across practice groups. The firm has announced a substantial investment in lawyer-led AI development. A measure of adoption can tell leaders whether tools are reaching the intended teams. Productivity is a further claim about the relationship between resources consumed and useful work delivered. It requires a different set of observations.
How Much Does a Faster First Draft Really Save?
Take a fictional legal team producing a standard client note. Without AI a lawyer spends 60 minutes researching and drafting, then a reviewer spends 20 minutes checking. The result takes 80 minutes of staff time. With an AI assistant the draft takes 30 minutes and the review takes 35. The drafting stage has halved. Total time has fallen from 80 minutes to 65, a saving of 15 minutes, or 18.75 per cent.
Now suppose one note in five contains a problem that takes another hour to correct after review. Spread across the five notes, the additional work averages 12 minutes per note. The expected total becomes 77 minutes, leaving a saving of three minutes against the original 80. If the original process also generated rework, its measured rework belongs in the comparison too. The rates here are illustrative, invented to expose the calculation.
Salary can change the answer again. A 30-minute saving for one employee accompanied by 15 additional minutes from a more expensive specialist may make good economic sense, or it may make the job costlier. If that specialist is already the person everyone waits for, even a net saving in labour time can slow delivery to clients. A single number called “time saved” leaves open whose time was saved and what happened to the queue.
What Is a Unit of Useful Work?
A law firm might choose completed advice accepted by the responsible lawyer. A support team might choose customer problems resolved without being reopened. A data team could choose recurring reports delivered on time with the figures reconciled. Each definition places the finish line beyond the instant a tool displays an answer. The definition should be settled before the results arrive, while nobody knows which version will flatter their favoured tool.
Consider customer service. An assistant can increase tickets closed per hour by drafting replies faster. If customers return the next day because those replies failed to solve the problem, closures overstate useful output. Track a sample through to resolution and the calculation improves. The same analysis can reveal an unambiguous success if faster replies also reduce repeat contacts.
For software development, counting lines of code rewards volume while ignoring maintenance. A small change that removes the cause of a recurring fault may be more valuable than a thousand new lines. A manager looking at AI-assisted coding therefore needs to know which tasks were completed, whether the changes passed review and what happened after deployment. A tool can be exceptionally productive when its most visible contribution is a suggestion to remove a line.
Why Is a Before-and-After Comparison Difficult?
Workloads change. A team might introduce an AI assistant just as an unusually complicated project ends. Completion times would probably improve even without the new tool. Alternatively, managers might ask their strongest staff to pilot it on the toughest cases. A superficial comparison with colleagues doing routine work could make a beneficial tool look worse than doing nothing.
Selection can run in the other direction. An employee may use AI on tasks they already expect it to handle well and avoid it on tasks needing sensitive judgement. Comparing assisted jobs with all unassisted jobs would then confound the tool’s effect with the employee’s choice of jobs. That choice is itself valuable knowledge. It suggests where deployment might work, even if it cannot establish how much faster the tool made each job.
Expectations also affect estimates. A suggestion that removes the most tedious ten minutes of an assignment can make the whole afternoon feel faster. A suggestion requiring an irritating correction can make a genuine overall saving feel like a loss. Asking people how a tool feels is worthwhile, particularly where stress and satisfaction matter. It should be recorded as a report of experience, alongside the measured result.
What Would a Fair Trial Look Like?
Choose a recurring piece of work, define an acceptable result and collect a baseline from the existing process. Assign comparable tasks to assisted and unassisted conditions in a way that avoids letting all the easy jobs drift into one group. Where feasible, random assignment can help. Where staffing and workflow make that impractical, document the differences and keep the strength of the conclusion proportionate to the design.
A useful trial might ask two reviewers to assess finished outputs without telling them which method produced each. Give them pre-agreed criteria such as factual accuracy, completeness, clarity and the need for correction. Record disagreements. If experienced reviewers cannot agree what good work looks like, an extra decimal place on a speed measure will do little to rescue the experiment.
Include the parts of the working day that are easy to lose. Time spent locating the right source, preparing an input, checking a reference, explaining a change to a colleague or revisiting a client query is still time. Training and setup costs need an appropriate horizon. Charging the entire initial investment to the first week can understate a durable benefit. Pretending training was free can overstate one.
An employer can also test whether gains persist after the novelty has worn off. A pilot supported by a specialist sitting beside every participant may demonstrate technical potential without establishing ordinary operating costs. The most revealing follow-up may be a quiet week three months later, with normal supervision and a mixed workload.
Why Might Beginners and Experts Get Different Results?
A beginner may gain access to a useful starting point or a reminder of a standard procedure. They may also be less able to spot a confident error. An expert can verify certain answers quickly but may lose time prompting a tool to rediscover facts they already know about a particular client or codebase. A whole-team average can hide either pattern.
Research on AI and work has found results that vary with the task, setting and people studied. One much-discussed METR experiment with experienced open-source developers examined work on repositories they knew well. Its findings illustrate why measured completion time can differ from participants’ impressions and why a result from one generation of tools should not be turned into a universal verdict on software engineering.
For a manager, the interesting question is narrower. Which staff benefited on which assignments, at what quality and with what support? If new starters improve markedly on standard letters while experts lose time on unusual cases, there is an actionable answer. Compulsory uniform use would obscure the very difference the trial discovered.
What Happens When Usage Becomes a Performance Target?
Employees pay attention to what their employer rewards. If a team is assessed on how often it opens an assistant, the simplest way to improve the number is to open the assistant more often. A highly capable worker who automates a recurring task once might then look less engaged than someone who repeatedly asks a chatbot to do the same job by hand.
The A&O Shearman reporting makes a distinction worth preserving. A practice-group adoption dashboard and a decision about an individual employee’s performance are different activities. A team measure can indicate that a department needs training or that an interesting workflow deserves examination. Moving from that observation to a personal conclusion requires evidence about responsibilities, opportunity, task mix and the work actually delivered.
There are legitimate reasons to measure adoption early. If subscriptions have been purchased but almost nobody can access them, it would be premature to study changes in output. The mistake comes when the early implementation measure survives as evidence of business value long after the organisation is in a position to measure outcomes directly.
Can Better Work Count Even When Nobody Finishes Earlier?
Imagine a payroll analyst producing a reconciliation in the same two hours as before. With an assistant helping generate checks, the analyst now tests a larger set of exceptions and catches an error that would previously have reached employees’ payslips. The clock shows no improvement. The organisation might quite reasonably decide the work is more valuable.
Such an outcome needs care in measurement too. A team that reports more defects after introducing better checks may look worse on a dashboard counting defects found. Distinguish errors caught before delivery from errors that reach customers. Record the burden of fixing each. Otherwise an organisation can create an incentive for people to find fewer problems in precisely the project intended to help them find more.
Which Result Would Convince a Sceptical Employee?
Show the whole path of an ordinary job. Give its original completion time, review time, correction rate and final quality. Show the same information after the change, identify who did the checking and explain what happened to any time released. If clients now receive equally reliable answers sooner and staff have fewer late revisions, the claim is intelligible without a league table of users.
A good outcome can be smaller and more interesting than a dramatic percentage. Perhaps AI makes no difference to the total hours but lets a junior employee understand a task earlier. Perhaps its main benefit is fewer omissions rather than faster drafting. A business that measured only prompts would miss both effects. An employer can find out whether AI improves productivity, but only by following the work far enough to see who benefits after the answer appears.
Sources
AO Shearman accelerates AI strategy with a USD11bn lawyer-led commitment
We are Changing our Developer Productivity Experiment Design
