Scorecard Weighting: 5 Steps to Weights Stakeholders Won’t Reopen14 min read

Picture a long weighting meeting where every criterion leaves the room rated high. 3 weeks later, a favored idea loses, and its sponsor asks to reopen the weights.

Here’s what usually goes wrong in that room:

• The question on the table is “how important is this?”, so every criterion comes back important.
• The most senior voice goes first, and the numbers settle around it.
• Someone splits a favorite criterion into 3 sub-criteria and quietly inflates its share.
• People can already see the ideas, so they weight toward the ones they like.

The 5 steps below give you weights that hold for the whole cycle. Only triggers you agreed on in advance can reopen them.

Weights are a policy decision, so set them before ideas arrive, anchor them on each criterion’s real range and argue only about gaps that change a ranking.

The goal is a defensible decision that 1 named owner signs and publishes. Consensus is welcome when it happens, but the process doesn’t wait for it.

Why “How Important Is It?” Is the Wrong Question

Ask a room how important customer value is, and you’ll hear very from everyone. Ask about feasibility or strategic fit, and the answer comes back the same.

The question is abstract, so obviously, it gets an abstract answer. Compare it with the questions that produce weights you can defend:

The UK government’s multi-criteria analysis manual, published in 2000, makes this concrete with safety and maintenance costs. Safety sounds more important, yet can earn a similar or lower weight.

That happens when every option offers roughly the same level of safety but varies widely on maintenance cost.

Von Nitzsch and Weber put it plainly: “The greater the range, the greater the weight has to be.” Their subjects still didn’t adjust properly when the range varied.

A review of elicitation methods calls weighting cognitively demanding and open to bias. The results can depend heavily on the method you choose.

So the range comes first, and the importance judgment comes second in every step below. The rest of this process is built on that order.

Before the Workshop: Fix the Criteria and the Room

In our experience, many weighting problems are set up before anyone picks up a pen. This preparation comes before Step 1 and covers 3 decisions.

Check the Criteria for Splits

Weber, Eisenführ and von Winterfeldt found that the more detailed parts of a value tree got weighted significantly higher.

They traced the effect to salience, since attributes spelled out in more detail feel bigger and come to mind faster.

So splitting market into 3 sub-criteria tends to raise its total weight, deliberately or not. Nobody in the room has to argue for that extra weight out loud.

Our rule is to weight at 1 level only, with every criterion at the same level of detail. We cap it at 4 to 6 criteria per idea funnel.

Decide Who’s in the Room and Who Decides

Name 1 decision owner before the session, and tell everyone who it is. Participants give input on the weights, and the owner signs them off.

Kim and Mauborgne’s work on fair process explains why this holds up. People can accept outcomes they argued against when the process itself was fair.

The model has 3 principles, namely engagement, explanation, and expectation clarity. In our experience, the last 2 are the ones weighting sessions tend to skip.

Invite a mix of functions, such as product, finance and technology, plus whoever owns the customer relationship.

Keep the group to 6 to 8 people, so every voice fits into 90 minutes.

Lock Weights Before Anyone Sees an Idea

Uhlmann and Cohen studied hiring decisions, and the finding is uncomfortable. People redefined merit to fit the credentials of the candidate they favored.

Committing to hiring criteria before the applicant’s gender was disclosed eliminated the discrimination.

That was a hiring study, so applying it to idea review is our inference. It matches what we see when weights get reopened mid-cycle.

So run the weighting session before the intake or campaign opens. Never reset the weights mid-cycle unless a trigger from Step 5 fires.

Step 1: Define the Realistic Range for Each Criterion

Before the session, write down the realistic range for each criterion. Each end describes the worst and best idea you’d plausibly see this cycle.

Work through each criterion in this order:

1. Set its 2 anchors. For time to first customer result, the slow end might be 18 months and the fast end 3 months – every swing is measured across that gap.
2. If nobody can describe the worst and best ends, the criterion isn’t ready to be weighted. Fix its definition first, or drop it for this cycle.
3. Check whether every likely idea sits in roughly the same place. If so, it earns little weight, however important it sounds in the strategy deck.

Strategic fit is a common example when the campaign brief already filters for fit. If every submission fits, the swing is small and the weight follows.

Write the ranges on 1 page and share it before the session, so nobody weights a private extreme. Arguments about ranges belong here, where they’re cheap to settle.

The 1st time through, this step feels slow and oddly concrete for a strategy session, but every later number depends on that shared reference.

Step 2: Collect Swing Weights Privately

The swing question replaces the importance question, and it’s harder to ask well. Picture an idea sitting at the worst end of all the ranges from Step 1.

Then ask which single criterion people would move from worst to best first, and run the private round in this order:

1. Hand out the range sheet and the swing question on paper or in a form, with no discussion yet.
2. Each person ranks the swings, from the one they’d move first to the one they’d move last.
3. They give the top swing 100 points.
4. They rate every other swing against it, so a swing worth half as much gets 50.
5. They add 1 sentence explaining their top pick and submit before anyone speaks.

The UK manual describes this family of methods as swing weighting. Our 100-point anchor borrows from Max100, a scale Bottomley and Doyle tested on importance ratings, not swings.

They compared it with direct rating and with Min10, which anchors on the least important criterion. Max100 came out somewhat more reliable on retest than direct rating.

Both beat Min10 by a wide margin. Their retest simulation found the same alternative chosen on 91% of occasions with Max100, 87% with direct rating and 75% with Min10.

Collecting the numbers privately matters too. In Kahneman and Sibony’s decision hygiene, each person judges the material independently of all the others.

Stasser and Titus found discussion tends to be dominated by information that’s already shared and supports existing preferences. We’ve watched that make the first senior opinion everyone’s anchor.

When People Can Only Rank: Rank Order Centroid

Some stakeholders will rank the swings happily and refuse to put numbers on them. For them, or for a 1st cycle, rank order centroid weights are a sound fallback.

Barron and Barrett found ROC weights more accurate than the other rank-based formulae. The table below uses whole percentages, so rounding makes the 5-criteria column add up to 101%.

The trade-off is resolution, since ROC assumes the ranks are all you know. A close 2nd and a distant 2nd get exactly the same weight.

Use it as a first-cycle default, then move to swing ratings once people trust the ranges.

It also works as a check: when someone’s swing ratings sit far from the ROC weights for their own ranking, it’s worth a question.

Step 3: Show the Spread, Then Re-Estimate Once

Work through the 1st-round numbers in this order:

1. Put everyone’s numbers on 1 screen, anonymously, criterion by criterion.
2. Sort by spread, so the widest gaps sit at the top of the list.
3. Discuss only the widest gaps, and ask the outliers to explain how they read the range.

In our sessions, many gaps trace back to different pictures of the worst or best idea, and a short conversation often fixes them.

Then everyone re-estimates privately, and you average the 2nd-round ratings and divide by their total. Don’t go hunting for unanimity with a 3rd or 4th round.

The pattern borrows from the Delphi method, where participants answer anonymously, see the group response, discuss and revise.

It also borrows from nominal group technique, which moves from silent generation through a round-robin without debate and ends with a private vote.

Both keep input private at key stages while still letting people learn from each other. A single revision round keeps that benefit without eating the whole session.

Step 4: Test Which Disagreements Actually Matter

Some disagreement will survive the 2nd round, and that’s fine. The only question worth asking is whether it changes which idea wins.

The UK manual says sensitivity analysis shows whether disagreements between people change the final results. It adds that the analysis can help resolve disagreements between interest groups.

Cambridge’s Institute for Manufacturing warns that weighting and rating are “extremely sensitive to small changes” and easy to “cook the books.”

So before the room spends time settling an argument, test it against real ideas. Each function that still disagrees writes its preferred weights as 1 alternative set.

A Worked Example With 3 Past Ideas

Here’s a worked example using 3 ideas from last cycle, rescored against this cycle’s ranges so 1 is the worst end and 5 the best.

After Step 3, swing ratings were customer value 100, strategic fit 70, feasibility 50 and time to impact 30. Dividing by their 250 total gives the Proposed row below.

Finance argued for feasibility level with customer value, and every score moved. The ranking stayed B, C, A, so that argument doesn’t change the decision.

Note Finance’s view in the rationale and move on without spending more of the room’s time.

Product’s set is different, because it pushes C into 1st place ahead of B. That disagreement would have changed which idea got funded, so it earns the room’s time.

Run every surviving argument through the same test, find the 1 or 2 that behave like Product’s, and spend the time there.

Step 5: Sign Off, Publish the Rationale, Set a Review Trigger

The averaged set (Proposed in the table) is the default, and the owner rules only where a contested set flips the ranking.

Their call stands for the cycle, even if someone still disagrees.

Write the rationale in 2 or 3 sentences, directly next to the weights. Say which argument won, which one lost, and the reason for each.

Publish both to reviewers and idea submitters, with a version number and a date.

Publishing is how you deliver the explanation and expectation clarity that fair process asks for. Submitters know the rules of the game before they write a single line.

Decide What Reopens the Weights

Locking weights means living with a set somebody thinks is wrong for a full cycle. Agree in advance on the only events that can reopen them:

• A change in strategy or in the portfolio’s mandate
• A new gap in the portfolio that the current criteria can’t see
• The end of the cycle, as a scheduled review

Everything else waits, including the idea that lost by a few points last week. A triggered change gets a new version number and applies to unscored ideas only.

So when the sponsor from the opening asks, however loudly, point to the published rationale and trigger list. Their argument goes on the agenda for the next scheduled review.

A platform that stores the weights with each scorecard version, and shows the calculation, keeps the decision traceable for anyone who asks later.

AI can sort the private round’s 1-sentence explanations by the range reading behind them, or re-run the sensitivity check when a trigger fires.

Tools can also amplify a disciplined process and keep weight changes auditable. The discipline itself comes from the owner and the session.

A 90-Minute Weighting Workshop Agenda

Here’s how the steps fit into 1 session of senior time. Ranges are drafted beforehand, so the 90 minutes go to weighting and testing.

90 minutes of senior time is a real cost. At the same time, swing questions take more effort than a quick 1-to-5 rating.

Weigh that against a full cycle of funding decisions made on weights nobody can defend. Run it once per cycle, and the cost spreads across every decision that follows.

Set the Weights Before You Meet the Ideas

Weighting goes wrong when it’s run as an open discussion with the ideas already on the table. Run the 5 steps, and that sponsor’s request gets answered, not granted.

Before your next cycle opens, check these:

• Every criterion sits at the same level of detail, and there are 4 to 6 of them.
• Each criterion has a written worst and best end for this cycle.
• 1 named owner will sign off the weights, and everyone in the room knows who.
• You have 3 ideas from last cycle, rescored against this cycle’s ranges (or 3 realistic test profiles in a 1st cycle), ready for the sensitivity test.
• The reopening triggers are written down before the first idea arrives.

Cambridge’s warning about fragile, easily cooked weights still stands, since weighting stays a judgment call. This process gives you weights you can explain to someone whose idea lost.

Published weights also give submitters a clear target, so their creative effort goes where the scorecard will actually reward it.

Download our ebook on idea evaluation and governance. It covers structured scoring methodologies, governance frameworks and the handoff from approved ideas into portfolio management.

Or book a demo to see how Accept Mission handles scorecards, automated routing, escalation workflows and audit trails across every stage of your idea funnel.