Stop Paying Twice for the Same AI Lesson

What this is, and who it is for

AI Capability Management (ACM) is a proposal. Give someone the job of making sure that what one AI project learns, whether a working model, a diagnosed failure or one person’s judgment, reaches the next project. Then add the few rules that make that worth doing.

It is easy to read this as advice for companies that train their own models. It is not. The same logic applies when you introduce a vendor’s assistant to a department, adopt a tool across six teams, or decide who gets to fine-tune what. In each case something is scarce: the one engineer who understands the data, a budget for evaluation, a place in the vendor’s queue. In the models the scarce unit is called a block of expert time and compute. Read it as whatever your teams actually queue for. Where this article says “compute”, it means that.

One idea comes from retail. A category manager looks across related products, develops what several of them can use, and directs the next investment with a view of the whole. ACM borrows that perspective. It can be a person, a small network of owners, or a provider. It does not have to be a department, and for most companies it should not start as one.

Two statements govern everything below. They are design rules for the models, not findings.

First, the label earns nothing. In every model, a capable local team that has the same information, people and rules as the central group does exactly as well. Calling something ACM adds no premium. If a shared arrangement beats the alternatives, the win has to come from something you can name: information that was otherwise missing, a promise people believe, or money that otherwise would not be spent.

Second, the arithmetic is checked, the organization is not. The models were checked in Lean, a program that checks every step of a proof. That tells you the conclusions follow from the assumptions. It does not tell you the assumptions describe your company. Treat each result as a precise question to ask about your own numbers.

The single most important result is easy to state. For one planner who can see everything, an optional, free and equally informed shared route cannot lower value. Losses come from three places: its cost, forcing teams through it, and teams’ incentives. In a game, where each team decides for itself, even a genuinely better option can lower the total. The rest of this article is that sentence, unpacked.

The research behind this section: Lesson 1, When does doing it once beat doing it $n$ times? and the research overview.

The second invoice

Start with the plainest question. When is it cheaper to do a job once and share it than to let each team do it alone?

The research answers with one inequality. Sharing wins when the number of teams, multiplied by what each team saves, exceeds what the shared route costs to set up and run. That cost has four parts: the fixed cost of building and maintaining the thing, the coordination it needs, the waiting teams do in its queue, and the expected losses from errors. In symbols, the test is $n(d - a)$ greater than $f + q + w + l$. In words: if each of $n$ teams saves $d - a$ by not starting from scratch, the total saving has to beat the overhead.

Exhibit 1 lets you play with it, with the same numbers as research lesson 1. Doing the job alone costs each team 40 units. The shared route has 80 units of overhead, 60 to build, 5 to coordinate and 15 of waiting, and leaves each team 10 units of rework, so each team saves $d - a$ = 30. Start with four teams. Move the slider down to one team: sharing loses, because nobody is there to split the overhead. Two teams are still not enough, 100 against 80. Past about 2.7 teams, sharing wins, and at four it costs 120 against 160.

Now push the second slider. Raise the rework each team has to do after adopting the common model. At 20 units, the break-even moves to four teams. At 35 it is beyond twelve. When local requirements diverge, each product redoes most of the work, and the shared queue costs more than it saves. At 40, as much as starting from scratch, no number of teams helps.

Three lessons for an executive follow. All three come straight from the same inequality.

Recurrence matters more than size. The number of teams is on the left-hand side. A job that two teams face once does not justify a standing group. A job that six teams face every quarter might.

Similarity is what keeps the rework small. The term you should interrogate is what each team still has to do after taking the shared thing. If the answer is “most of it”, you have bought a queue.

Waiting and errors are costs, not footnotes. They are in the inequality on the same footing as salaries. A shared group that is always busy is not a success. It is a queue, and a queue has a price.

The research behind this section: Lesson 1, When does doing it once beat doing it $n$ times?

Keep it optional, and give the label nothing

The most actionable result for an executive is also the least dramatic. Across the models, an optional shared service can never do worse than the teams’ own route, because a team that finds it unhelpful can simply ignore it. A compulsory route can lose.

The reason is not mysterious. When a team must go through the shared route, you charge it the route’s cost whether or not the route helps. In one worked case, a high review fee made a mandatory route a loser while an optional one was simply bypassed. In another, a shared fund that would have added value once running costs were counted turned into a loss when the fee passed a threshold, but only if everyone was forced in. Exhibit 6, later, lets you cross that threshold yourself.

The second half of the rule is the one about the label. Because a capable local arrangement with the same information, rules and costs reaches the same result, anything you claim for a central group has to be traced to one of three sources. Either it gives teams information they did not have, or it makes a promise credible that was not, or it pays for something no single team would fund. If you cannot say which, you have a department with a name.

The research behind this section: Lesson 1, Lesson 5, What a record of failures is worth, Lesson 6, Making the truth worth telling and Lesson 10, Who pays for the backup nobody needs today.

Better expertise, worse fit

The shared group usually has the better specialists. That is half the case for sharing. The other half is the bill from the previous section. There is a third factor that executives tend to forget: fit.

In the research’s model, a method makes fewer execution errors the more expertise stands behind it. A shared method has three-quarters of the available expertise and a local method one quarter. The shared method also carries a penalty: the further it sits from the particular job, the more errors it adds. These are properties of the methods, not of the teams that offer them.

Exhibit 2 shows the contest. At zero mismatch, the shared method wins easily, with about 0.13 expected errors against the local 0.38. As mismatch grows, the shared method’s curve rises. At about 0.71 the two meet, and beyond that the local method wins, despite its thinner expertise.

The title is deliberately about methods, not teams. The research is careful about this. It does not say that a central team is worse than a local one. It says that a method built far from a job pays for the distance, and the distance can outweigh the skill.

The practical consequence is a division of labor that appears again and again: share the base, tune locally, and let the team that bears the cost of an error decide whether the result is good enough. The model does not prove that tuning will succeed or pay for itself. It tells you what to ask: how far is this shared method from our job, and who decides?

Where should the expertise sit?

The same model answers a question executives ask often: do we build a central AI team, or upskill every team? The answer is a threshold, and it is not the one people expect.

Practice builds skill with diminishing returns. If one team does much more of the work than the others, concentrating the practice in that team builds more skill in total. If the workloads are equal, spreading the practice wins. Concentrating pays only when one team’s workload is large enough relative to the others, and the threshold depends on how quickly skill builds. So the question is not “central or distributed?” It is “is the work actually concentrated?” If it is, put the expertise where the work is. If it is not, a central group is a good place to keep a few specialists and a poor place to keep all the practice.

The research behind this section: Lesson 2, Better expertise, worse fit and Lesson 3, How much shared work to fund, and where expertise should sit.

When everything is P1

Suppose you have solved fit, and now there is a queue for the scarce thing. Teams ask for it. Something decides who goes first. If the thing that decides is a label that costs nothing to claim, the research has a blunt result.

Exhibit 3 shows the smallest version: two teams, one slot, and a priority label that is free. Whatever your rival does, saying “P1” is strictly better for you. If the rival says normal, P1 gets you the whole slot instead of half. If the rival says P1, saying normal gets you nothing, where P1 gets you half. So both teams say P1, and you end where you started, with the slot split by a coin toss. The label has not gone away. It has gone silent. It no longer tells you which request matters.

The research does not stop at the diagnosis. It asks what a better request process would need. A good process asks for something that is costly to fake, such as the decision the request would change. But the models also say a request process can still lose, and they name four ways:

• Unpaid records. If writing up a failure costs a team effort and brings it nothing, nobody writes, and the company learns from silence.
• Audits that cannot tell good from bad. A review that scores a weak project and a strong one the same way charges everyone and discriminates against no one.
• Rules enforced only on paper. A rule the sponsor can quietly bend does not change what teams expect.
• The wrong criterion. A gate can separate requests perfectly and still destroy value. In one worked case, it deterred good projects and lost about 0.4 of a unit. A polished presentation is not an economic case.

The management lesson is a modest one. Do not rely on a priority field. Ask each request which decision it would change and what would stop it. Put the scarce thing behind a budget a team can spend, so that claiming priority costs something the team cares about.

The research behind this section: Lesson 4, When everything is P1.

What a record of failures is worth

The next promise of a shared function is memory. A team tried something, it failed, and the reason is written down so the next team does not repeat it. That is worth having, and the research says when.

A record helps only if it is correct, applicable and affordable to read. A wrong record can be worse than none. In one worked case a misleading record led the planner to a loss of more than two units of value, which is what you get when a team trusts a diagnosis that does not apply to its job. This is not an argument against records. It is an argument for the unglamorous work of checking whether a past failure resembles the present one.

Two other results sharpen the point. The first is about what to buy. Evidence is worth buying only if it can change a decision that matters. If you would proceed whatever the test shows, the test is waste. And the best plan is not a fixed list of tests but one that adapts: a test chosen after seeing the last result beats any fixed set chosen in advance, because it can stop when it has learned enough.

The second is about what you cannot see. A dashboard shows approved work. It does not show the test nobody ran, because the request was never made or was turned down. If you know who stays silent, the silence itself is informative, and an audit of unseen cases is something you can buy. Put the missing test in the denominator when you judge how well the process is working.

The research behind this section: Lesson 5, What a record of failures is worth.

Making the truth worth telling

A record is only as good as the willingness to write it. This is where most programs quietly fail, and the research is direct about why.

Take a team that has just discovered its project was poorly chosen. If it reports that, the company learns something, and the team may lose its next slot. If it stays quiet, the company learns nothing, and the team keeps its slot. Staying quiet is the sensible private choice. In the model, with no promise attached, nobody reports and the company allocates blind. That scores 4 units.

Now offer a promise. Teams that report a useful failure get a bounded, believable chance at new work. In the worked case, the company ends with about 4.6 units instead of 4. The promise changes behavior because reporting is no longer the end of the road.

The catch is the word believable. A promise the company can break cheaply is a promise nobody counts on. Exhibit 4 plays this out. When breaking the promise costs the company nothing, teams see through it, stay quiet, and the company ends with 3.75, less than the 4 it got with no promise. It paid the setup cost for a promise and bought nothing. Slide past a cost of 1, where keeping its word becomes cheaper than breaking it, and the promise is believed and the value jumps.

Three further results complete the picture.

A promise can protect too much. If the promise is to keep funding the old work, the company can end up worse off than if it had no promise: it has bought loyalty to a failing project. The research suggests a clean separation. Credit a team for a useful failure, and treat the next project as a separate decision. In the worked case, binding the company to a bad project lost one unit.

Noise in the gate costs money. If the review that decides who is rewarded is unreliable, false positives enter and the benefit shrinks. In the noisy case the value dropped to about 3.2, below the no-promise 4. An unreliable review flips the decision. This is one of the few caveats that reverse the advice.

The ratchet. Teams think a round ahead. Anyone who has run sales knows the pattern: beat your quota and next year’s quota rises. A team that shares a diagnosis may be teaching the company to turn down requests like its own. In one worked case, giving the company a better option for later made teams stop sharing now, and the company ended with 3 units instead of about 8. A smarter process does not guarantee more honest inputs.

The management translation: separate credit for a useful failure from funding for the next project. Make the credit one the company can be seen to honor, and make breaking it expensive for the company, not just awkward.

The research behind this section: Lesson 6, Making the truth worth telling, with the pitch-process results in Lesson 4.

Teaching people, then stepping back

Everything so far concerns what the company knows. This section concerns what its people can do, which is harder to move. The research separates three things that managers tend to run together: a person learns something, passes a test, and is allowed to act alone. A person can complete a task with help and learn nothing. A pass is not a permanent state; it decays. And an authorization gate has a price. Removing it earns more output and admits more people who are not ready.

The model’s answer is that assisted success is not independent judgment. Judge readiness on fresh cases, without help. Reduce help task by task rather than person by person. Restore it when the task changes.

The support function may not want to let go

Here the research reverses a tempting thought. It is easy to say that the best support function makes itself less necessary. The model says to expect the opposite unless something forces it.

Take an adviser who values staying involved, for status, control or simple habit. In the model that adviser keeps advising a team that is already ready. The team sees this coming and skips the last step of learning, because the help will be there anyway. The company ends with less. A promise to hand over when ready did not fix it. A rule that forced the handover did, and the adviser gave something up to follow it.

Exhibit 5 shows where the trap opens. Below a tipping point, the adviser prefers letting go, and nothing is lost. Past it, the company loses half a unit, and the rule that would have prevented it costs the adviser more the more the adviser values staying.

The proposal that follows, and it is a proposal, not a result, is to write the handover into the mandate. State which tasks a team takes over, on what evidence, and when. Do not rely on the adviser’s goodwill.

What changes when AI does part of the job

One result matters especially for companies that are adopting AI. If the tool is reliable, the model stops training people for the task. In the worked case, stopping was worth 12 against 11.25 for continuing the apprenticeship. Make the tool fail half the time and the answer flips: keeping the apprenticeship is worth 10.5 against 10 for stopping. The tool’s outages are what keep the apprenticeship valuable.

This is not an argument against the tool. It says that the value of training people is tied to how often the tool fails, so you need to know that rate before deciding.

The research behind this section: Lesson 7, Teaching people, then stepping back.

Hiring into a queue

Sooner or later the answer to a queue is a head count request. The research does not tell you how many people to hire. No staffing ratio is established. What it does is show that each answer wins somewhere.

In the staffing model, one generalist wins when the alternative is a handoff between specialists. Two specialists win when the task needs depth that no generalist has. A mix wins when both kinds of work appear. And the same total hours that fit when deadlines are spread fail when everything is due on Friday. Breadth does not supply depth. Before asking for head count, name the missing depth and the work that lands at the same time.

Train, hire or contract

For a gap that is not going away, the research compares three routes. In a small worked case, training beats hiring or contracting by a hair, 31 units to 30, when training is certain to work. Make it a coin flip and training falls to 24 while the alternatives stay at 30. The lesson is the thinness of the margin, not the number.

Whether to train depends on three things: the odds that training works, what the learner and the mentor would otherwise have delivered during the training, and whether the skill is still needed when it arrives. The last matters more than it seems. A forecast of demand that turns out wrong defeats the best plan made from it. In a worked case, a plan built on a wrong forecast did worse than a plan that waited for better information. A hire made for a burst of demand is a different decision from a hire made for a recurring need, and both differ from a lull. Name which one you face before you open the requisition.

The research behind this section: Lesson 8, Generalists, specialists, queues and lulls and Lesson 9, Train, hire or contract.

The insurance nobody buys

The last problem is the one that surfaces at the worst time. One person holds the knowledge that keeps a system running. Everyone knows it. Nobody pays to reduce the risk, because the cost falls on whoever pays and the benefit falls on everyone.

The model’s answer is stark. Two teams each depend on the same expert, who may leave. Each could fund a written record, or train an apprentice, or both. In the worked case, a record alone recovers a quarter of the expert’s output, an apprentice three-eighths, and both together three-quarters, though the apprenticeship only works about half the time. Funded jointly, the backup adds one unit of value. Yet each team, deciding alone, funds nothing. Doing nothing is its strict best reply, however much the pair would gain. This is the free-rider problem in its cleanest form.

Two ways out appear: a shared budget, or an enforceable agreement to split the cost. Neither requires a department. What matters is that someone is able to fund the shared benefit that individual budgets leave out. And the fix can cost too much. Exhibit 6 shows the fund’s net value as its running fee rises. Past a fee of 1, which is the whole gain, the fund loses if everyone is forced to join. Left optional, the company simply stays out and loses nothing. Moving the record between teams has a similar ceiling: when the transfer cost passes a threshold, the gain reverses.

The management translation: name the person whose departure would stall an AI system. Fund a record and an understudy from a shared budget, and keep it optional. Ask what the backup costs when needed, and how often it is needed, before you decide whether to keep it.

The research behind this section: Lesson 10, Who pays for the backup nobody needs today.

A reading aid: what to share and what to keep local

The results so far fit on one page. Exhibit 7 puts them on two axes: how often the same need comes back, and how different the local needs are. It is a reading aid built from Exhibits 1 and 2 and the three scenarios behind them. It is not a combined result.

The cells restate the results of Exhibit 1 and the fit result of Exhibit 2. The names follow the management book, chapter 3, where the choice is argued.

The research behind this section: Lesson 1 and Lesson 2. The judgment behind the labels: Learn Once, chapter 3.

Startup to enterprise: the same question at different sizes

A startup can use one internal owner and outside help. It fine-tunes only if tests of prompts and tools leave a specific defect. An SME can protect part-time responsibility for a few people. An enterprise with many teams and recurring demand can fund a dedicated group. In every case, the first move is a role with a few rules.

One qualification matters for most readers. Read literally, the research’s compute model applies only to a company that develops models. Read as a queue for whatever is scarce, whether specialist time, evaluation budget or a vendor’s capacity, the logic carries over. That reading is ours, not a separate result.

The research behind this section: the research overview, which lists the six arrangements, and Lesson 3. For the choice by company size, see Learn Once, chapter 2.

On Monday

None of these steps needs a department or a budget line. Each is a question you can put to a team this week.

When not to bother

The research is as clear about the cases where this does not pay.

Do not bother when needs diverge. If each team must redo most of the work after taking the shared thing, you are building a queue. Exhibit 1 shows the break-even moving beyond reach.

Do not bother when the work rarely repeats. The inequality needs several teams facing the same problem more than once. If it comes up once, buy the answer or write it down.

Do not bother when a supplier’s product already does the job and your teams use it. A new layer on top is overhead.

Do not bother when an existing owner, a platform group or a strong local team already does this work. The label earns nothing. If the arrangement you have shares information, makes believable promises and funds the shared benefit, the models give no reason to add anything.

And if you do start, start small. A role and a few rules. Watch for the signs that it is not working: teams that route around it, requests that all arrive marked urgent, records nobody reads, an adviser who is always there. Be ready to close it.

What this article does not claim

Everything above rests on models, small mathematical worlds with invented numbers, checked in Lean. They show what follows if the assumptions hold. They do not show that the assumptions hold in your company, and they do not measure an organizational effect. The public analogies are real episodes, dated and sourced, and each rhymes with a result without supporting it. Where a source is a secondary report of a primary announcement, the sidebar says so.

For the proofs, read the research. For the convictions of someone who has spent twenty-five years watching organizations do this, read the book, Learn Once. For the literature behind both, see the sources.