Priorities for Prioritizing Prioritization
And no - I don't mean let's just do everything
It's not enough to be busy. So are the ants. The question is: What are we busy about? - Henry David Thoreau
Productivity is meaningless unless you know what your goal is - Eliyahu Goldratt
What does a prioritization rubric look like
A lot of people agree that prioritization drives success, and is about knowing what to say no to. That’s good general advice - but easier said than done. I’m fond of a gold standard evaluation you can use to see if you’re on the right track. The test goes something like this …. If you ask a team - “what are your priorities?” ideally you’ll hear something like;
“our priority is to work on _____, and then ____,
and if there is additional cycles we focus on _____.”
The best answers have each of the blanks filled in with extreme specificity. Meaning that it’s easy to find things that don’t fit into any of these categories. If the prioritization buckets will accommodate most choices of work then you’re not there yet.
Meeting these conditions, eliminating the vast majority of multi-tasking and forcing the saying of “no - we cannot do that now” determines if you’ve sharply prioritized or not.
Once you’ve obtained the golden template answer - ask several random members of the team/org the same question (ie; NOT just the product or engineering manager). If you get the same from everyone then you’re good to go. Otherwise it’s time for more discussions and sharing of context in terms of goals.
What drives priorities that stick are shared context - so if people don’t think they’re all working on the same goals, then any single set of priorities won’t make sense. Priorities that don’t “make sense” aren’t followed no matter how much you re-explain them.
A simple state of priorit with between one and three “beats” is what I’m pitching as a “prioritization rubric.” When you have that then it becomes easier to detect deviation from this plan organically. Make clearer when it’s wise for the team to break the rules briefly, or update the rubric given new circumstances (success, changing reality, etc).
An example: Pricing Team Priorities
I’ll share a real world example from my time at Convoy1. The Pricing team built a predictive model for truck costs (among other things). Given the characteristics of a shipping job (aka ‘a load’) it would estimate what it would cost us to hire a truck to move that load. This was a key input to how much the company should bid for work from Shippers.
This was an incredibly skilled team with deep knowledge of ML, price modeling, and the overall business ecosystem in which they lived. The quality of outputs was good. Their ability to effectively predict prices of future work was a significant driver of Convoy’s competitive advantage. It wasn’t a large group and they worked hard to deliver the engineering and ML systems required to do a hard job. That’s all a long way of saying that this is intended as an example of how nailing a prioritization rubric drives high leverage alignment AND creates local autonomy on a high functioning team. It works just as well if the team has core performance issues too - but that’s not what this example is about.
Each quarter the team wrote complex and detailed plans on what would be worked on. The product manager (reasonably) wanted to optimally utilize the team’s capacity when it came to work planning. Therefore, there would be a LOT of things going on at once. In a later post I’ll discuss how this reasonable but incorrect assumption of the Agile/Scrum planning process is almost optimal for producing a lot less than you’d hope for in terms of throughput. Especially without a simpler prioritization framework. For now, just recognize that if you asked the team members what the prioritization rubric was you’d get an answer ranging from “I don’t know” to something that was definitely not consistent across team members2.
The process that led to the start of this story is in itself interesting, I’m thinking to save the longer version for another time. But it’s worth sharing some of the undesirable characteristics3 of the current environment. In no particular order
There had been several pricing error events where something had gone wrong with the model system and predicted prices were considerably worse than expected tolerances (for an extended period of time). In several cases this had been driven by an undetected change in the underlying data delivered by an upstream source.
Operational teams outside of our org often shared examples/concerns where the model predictions “seemed wrong.” This was a probabilistic system and the pricing team expected variation but believed overall it was working well. There was a long simmering sense of discontentment from the operational teams who felt the pricing team might not understand, or be working on the highest value things. A symptom of this was a lot of time for the team scientists responding to questions about specific price quotes.
There was an observable disconnect in higher level discussion between the pricing teams and their ops counterparts. External observers noted that the pricing team was basically saying “the model is good overall and here’s the proof” and the ops contingent was saying “The model is bad in these specific segments, can’t you just Moneyball those?” DS teams don’t love to hear that phrase it turns out4.
The ops teams had called out issues that were part of these pricing events, but because the team was so used to these they didn’t result in the problem being noticed for a while. “A while” means 3 weeks in one case and 2+ months in another. Due to the vagaries of pricing predictions being used in a bidding model we may have made money on these mistakes - but it still wasn’t a great look.
As we postmortemed the larger scale event pricing issues (one after another) I started to get the sense that the team didn’t feel like hardening the system was important (preventing big losses potentially). Building better predictions seemed to be the functional priority. It’s not that they were in favor of losses but felt the issues that had occurred primarily should have been prevented by the upstream teams that fed in core data. ie; they were somewhere between “out of our control” and “someone else’s fault/problem.”
Some folks (definitely me) felt that the small pricing team seemed to be working on a lot of things at once.
There’s a lot to unpack here. Given the focus today I’m going to jump to what came about after a lot of listening and discussion. There were a lot of fingers being pointed about what happened - but what to do and what not to do needed higher clarity. We ultimately decided that the team’s priorities were;
(1) Prevent models from making extremely bad predictions, and then if time allowed (2) Improve the three most business impacting segments of model underperformance, and if any time remains (3) invest in ways to be better and faster at (1) and (2).
In more detail the tasking became to;
Ensure the predictive model was working as expected - in layperson’s terms, reduce the risk it’s making totally dumb ass broken predictions. This was important because such major defects had been prone to happening and then not getting noticed for over a month, costing a ton of money. Making sure stuff was working as the #1 goal made it clear we actually wanted time spent here - and not just hoping we were lucky.
Provide a segmented model performance scorecard and work only on improving the top 3 underperforming segments. This solved for two problems
Building Trust: Segmenting the model by use case forced people’s understanding and assumptions to align. The team would also quote that “the model worked well” while people in the financial trenches thought the model sucked. The reality is that both were true.
The model “overall” did work well5. Averaging over all use cases the model didn’t get us into hot water with it’s prediction. However, if you only dealt with refrigerated trucks, or drop shipping jobs then maybe the model was notably worse than the average. Because these were just a part of jobs perhaps overall things were good. But if your job was refrigerator (reefer) transit then the model could be absolutely crap6.
Building a scorecard that showed model performance by actual business use case would bring arguments from the general to the specific. When you own a prediction/forecast/decision engine it is somewhat inevitable that consumers of your output will be bugging you about X or Y. Having a scorecard allows you to say “overall the model does well in the N areas that represent 80% of our business, but in the two areas you brought up we agree it’s very poor and our short term goal is to double performance in that area this quarter - even though we will still have a long way to go.” That is a lot better, or at least more specific than the ongoing sniping - which often leaves a team “answering a lot of questions” instead of improving things.
It’s worth noting that I’m not saying the pricing team was unaware of local business area performance of the models. I’m just saying it wasn’t the lingua franca between the areas of contention.Making the model “better” in an area it was already “good enough” wasn’t as important for the business as making the weaker segments reasonably good. Showing the segments explicitly in a scorecard lets the team that owns improving predictions set better goals. And goals that are more aligned with other’s pain/assumptions about what is important.
If time exists beyond these - Invest in activities/infrastructure that let us iterate faster on #1 or #2.
Once agreement on this is established and agreed to (subject to lots of open debate of course) it can become a remarkably useful tool7.
Using the rubric in plan reviews
After your prioritization rubric is established you might find yourself a few weeks later reading through a plan from the team on time investments for the next few months. Skimming through this doc, if you see something not on the list you can ask - “why are we doing this?” - or much better yet “which of our core priorities does this area of work connect to?” After hearing the answer you can iterate again with the priorities. This reinforces that you really mean “only these priorities” and allow you and the team to recheck they’re the main controllable inputs to the thing the business and customers care about.
Something small on a team “to-do” list may be outside the scope of the priorities. This to be an area where it pays to be persnickety8. After focusing the Pricing Team a monthly update mentioned a new senior engineer named Bob (not his real name) was assigned something outside of our priorities. Putting aside that having one person named specifically in a task is a bad sign - the task Bob was on definitely was not in our three focus buckets. The team’s reasonable response - Bob was new and they wanted to give him a “warm up project.” The issue though is that the warm up project wasn’t in one of the categories. When asked, people admitted they’d be spending time working with Bob if he got stuck in the training exercise. First I noted what seemed like an inconsistency with a snarky comment that I thought it would be better Bob sit around and eat bon-bons rather than a training project outside the priorities9,
Next, I asked if there was really no way that a super experienced ML engineer like Bob couldn’t contribute to the three priorities. While I shouldn’t have made the bon-bon crack, the following discussion created ways for Bob to learning while contributing toward bettering model performance segments. It was even realized that his fresh eyes might be an asset.
Once we agreed the priorities were the key - then keeping people “busy” on non-priorities wasn’t viewed as valuable/worth doing.
One cannot remind folks too much that “resource utilization”
is unimportant relative to goal aligned throughput.
Conclusion
If you don’t say no to anything then you’re prioritizing everything, which means you’re prioritizing nothing. That’s an old expression because it’s true. Avoiding that trap is benefitted from a few rules of thumb. This article shared a basic test that lets you check if you really only have a single focus, and then confirm by diving deep/auditing if your team is acting on that.
Getting to what that focus should be is an entirely different matter. To overly simplify, the focus point should have the following characteristics
It’s super high leverage to your business. You make more customers happy/loyal and make more money if you do it.
It’s focused on a controllable input that’s related to a main bottleneck in the business.
When explained to others it will be so clear it’s referred to as being “obvious.” This may piss you off because of how hard you worked to make it obvious.
You can explain it in a single sentence, and others in the group if randomly sampled will say the same thing (without having to be aided in their recall).
It is useful in making decisions around minimizing multi-tasking and reducing work in progress (WIP).
It will be easy to find that most things you’re asked to do will NOT fit into the the prioritization rubric. At least not in the “this is the main, and secondary thing.”
How to go about figuring this out is pretty challenging much of the time. Though the team always knows the answer, you just have to listen, translate and reflect well enough to highlight the systems thinking of the group.
If you’d like some help doing this in practice I’m currently available to talk through it with you. Please reach out and we can setup an intro call to see if this would be useful. No selling from my end - just curious discussion to get started.
What’s next?
In my next article I’ll share some basic categories that will help you brainstorm, or just narrow down where to find your priorities. Yes, usually the team already knows what they should be. But as “all happy families are alike; each unhappy family is unhappy in its own way” there are common priorities that usually should be used. Knowing these categories can accelerate determining the unique to your problem areas priority to set.
Though, spoiler alert - to paraphrase Tom Cruise in Risky Business the dream priorities is are always the same.
Subscribe so you don’t miss out. Share so your friends don’t either.
Convoy was a high tech digital freight brokerage that was destined to drive change in an industry, but not survive the transition. Yes - it was a major (and expensive) bummer.
At least one member would likely burst out laughing at the question.
I’ve swiped this term “undesirable effects” from Eliyahu Goldratt’s writing. An early step in his Theory of Constraints thinking processes is to list out undesireable effects (UDE’s) of the current system. He wrote wisely that everyone loves to complain so finding things that aren’t good is an easy way to start finding the problems in the system. Not all of these problems are necessarily important but they can be used as a map to problems that constrain progress against the core goal, and root causes.
For better or worse I really enjoy Michael Lewis’s writing, though I haven’t read his most recent book. FTX just felt too soon. I especially like his earliest work such as Liar’s Poker. While I know he claims it was written as a cautionary tale it’s easy to see someone taking it as a roadmap. For example, building a whole life philosophy around the advice he’s given as a trainee to let the other side see the knife first and then they’ll relax “The customer isn’t stupid. The first thing he wants to know is how he’s going to get fucked. Once he knows that, he’s perfectly happy to do the trade.”
I’ll leave for another time getting into MAPE or other assessments. Anyone feel like writing a guest post on that? :-)
This is just an example for illustration. I’m not saying the model was bad at one specific thing vs. another. I don’t recall it down to that detail with confidence - but I am confident that discussing it at the average contributed to the issues between teams.
I’m using a smidge of dramatic license here. I’m not saying that everything played out exactly this way - in that the Pricing team adopted this approach and lived happily ever after. I left Convoy after a finite period of time from this story - so I’m not 100% sure if folks kept the focus on this approach. It is though a nice way to describe an approach to aligning on true priorities I’ve seen work on more than one occasion.
Amazon often refers to this as having unreasonably high standards at times. Or something like that.
I’m not that proud of that. You probably shouldn’t manage with sarcasm. But I do suspect in this case the point may have been more memorable so I’d like to think that’s why I broke the rule here. OK, I’m a little proud.