Learning from the failures of others
An (unexpected and unplanned) album release party post
"A smart person learns from his mistakes, but a truly wise person learns from the mistakes of others.” - various forms attributed to many
“When it comes down to it, the problems are usually all the same.” - paraphrasing the coffee chat wisdom of Tim Prouty
Doing serious work and minding my own business when suddenly I got a silly idea
As an engineer I love learning from failure. Post-mortems, pre-mortems, COE’s - I dig it all. Conditions just conspire against us in a clear “the universe laughs at our plans” sort of way. Trying to outwit it is both a fools errand, and incredibly important. There’s a reason that one of my first posts was about the Galactic Empire’s single point of failure in Death Star (all of them). Being wrong quickly, not being wrong, etc. are all good mantras.
This is a long way of saying that I pause from time to note antipatterns that may be swirling around me. If you’re not familiar with the term it’s often described as “a commonly used, superficially appealing solution to a recurring problem that is actually ineffective, counterproductive, and leads to long-term technical debt." Or put in simpler terms - a way of doing things that makes sense, is thus commonly used, and is usually wrong. They’re easy to fall into without noting, so I occasionally try to think mindfully about them (and other inherent assumptions).
This is the “fun,” nerdy activity I was engaged in yesterday. Analyzing what I thought was jamming up some work. Jotting down bullet pointed notes in an open file. Wondering “I think this would be valuable discuss with others - what’s the best way to land a productive discussion?” I’ve noticed that outside of Amazon, sadly no one wants to read a doc at times, or listen to a presentation. Hence, trying to be more intentional to meet others where they are.
I was stuck, and totally not browsing the internet. Then it hit me. How about a “listicle”!? For some reason I hadn’t seen one of those in a while. Seemed worth a shot - “top antipatterns that drive work to get bad.” Maybe not the snappiest slogan - but I figured we’d “clean it up in post.”
Next steps - My process
What followed is a blur. Next thing you know I quickly passed through the stages of creativity that I’m sure we’ll all recognize - make a list, get distracted, adapt it to align with a shallow nod to pop culture concepts, and write produce a 7 song album in an afternoon.
No - that’s not how you roll? Well - my mother always has said I’m different.
Here’s a look into my mental steps yesterday afternoon. I offer this as explanation - not as a suggestion for your own workflow. Though the first few steps seemed solid. It’s only later on it went off the rails a bit. Or was awesome! Determining which is left as an exercise for the reader.
Origin point: I thought I could see through the matrix to define set of things worth calling out as areas to improve (on a particular workstream I’m engaging with). As previously established I’m an engineer who loves learning from failure. Finding talking about it, during pre-mortems, post-mortems, and sometimes at cocktail parties, to be of great fun and value.
I wrote stuff down in a list. Thought about drafting a narrative around it.
Then I thought about whether anyone in environments outside of Amazon would listen to someone complain about stuff in a well reasoned doc? Nope - not bloody likely.
How about a presentation? Nope.
After some tears about this, and pouring out a cup in honor of the unread docs of OP1 future I decided to get real. Listicles! If it was good enough to build the unassailable Buzzfeed empire1, then why not here?
Thus was born a list of common
antipatternssins that cause work and teams to not meet their full potential. Seven important … maybe let’s call them sins. Even deadly ones.On the first list I made was the serious entry “normalization of deviance.” A turn of phrase with grounding in safety culture. The challenger disaster often gets mentioned in the same breath.
As I finished the list I started wondering “that would be an awesome name for a metal band” and “how many bands exist with that cool name?”
After some important research I had a new, even more important question “do I really want to live in a world without a nerdcore band called ’Normalization of Deviance’?”
Then I got distracted and decided I should re-write the list of antipatterns to be labeled as commonly viewed Judeo-Christian “sins.” That was quickly abandoned (largely because it was too hard) and replaced with the easier problem of giving all the anti-patterns names that could be mistaken (if you squint) for Metallica song titles.
After basking in the glow of success, I went back and decided to fix the problem of #9 in this list. You’re welcome.
A shockingly short time later I released my AI first nerdcore2 pop/punk3 album “Sins against throughput” based on this list. It’s wild living in the future. Also - I’m not actually joking. I hope you’ll checkout the whole album - free of charge ‘natch.
Technically it’s more a range of punk inspired styles - not just pop-punk. Musically perhaps not that inspired, but the lyrics still are reasonably sharp and make me laugh in a knowing way. I’ve included a couple later in the article.
Sins Against Throughput
Oh … you’re still here? Maybe some of you are here for the antipatterns and not moshing your way to geeky flow nirvana.
Tyranny of the Urgent: Consistently choosing the short term better over the long term. Long term is harder, but more valuable.
Vacuum of Intent: Lacking top-down alignment on the few things that need to be true in the future. Local and short-term decisions will fill the gap.
Fragility of the Few: A small number of subject matter experts limit progress. If work piles up in front of these bottlenecks it’s key to have a plan. This is true about all bottlenecks - but often especially ignored with people.
Normalization of deviance: Serious defects should trigger an andon cord full response. Focus should remain until truly resolved.
Communication complexity: A risk when many people need to talk to too many other people for many decisions.
Oral tradition. A risk if people taken when they should write. Knowledge stays in people instead of in shared systems.
Perpetual Triage: Managing via expediting instead of fixing underlying constraints.
I’m sure I didn’t get them all - seven is an arbitrary number, and they’re perhaps biased towards recent experiences. In other words, don’t worry - there are more. At least enough material for a second album.
Never write your second album without new fresh collaborators. Please drop your suggestions in the comments on throughput antipatterns. I’ll be sure to at least credit you in the future liner notes. Also - I’m thinking maybe country as the next genre. :-)
Lyrical Sampling
Some folks aren’t going to click a link no matter how awesome (or stupid) the content on the other side might be. So I figure I’d drop one set of lyrics in.
No North Star (Vacuum of Intent)
[Intro]
Compass spins.
Signal gone.
Everybody local.
Nobody wrong.
[Verse 1]
Every team has a map of their hallway
Every hallway has a locked door
Every door has a metric on it
Nobody knows what the building’s for
We optimized the smallest square
Made it perfect, made it fair
Then wondered why the whole machine
Dragged its heart through gasoline
[Pre-Chorus]
In the vacuum of intent
Every local truth gets bent
When the future is not named
The nearest fire gets the flame
[Chorus]
No north star
No signal, no map
Every local win tightens the trap
No north star
No line through the dark
We built the maze and called it an arc
In the vacuum of intent
We pick a direction
And call it smart
[Verse 2]
Top-down silence, bottom-up noise
Everyone busy with clever toys
A thousand arrows, none aligned
All good answers, wrong design
What has to be true?
Nobody said
So the urgent came and made its bed
[Bridge]
Name the bet
Name the scar
Name the reason
Name the star
[Final Chorus]
No north star
No signal, no map
Every local win tightens the trap
No north star
No line through the dark
We built the maze and called it an arc
In the vacuum of intent
We pick a direction
And call it smart
Stop the Line (Normalization of Deviance)
Stop the Line (Normalization of Deviance)
[Intro]
Cord hanging.
Alarm blinking.
Everybody sees it.
Nobody’s saying.
[Verse 1]
We wrote the risk in lowercase
Then shipped it with a smiling face
The warning flashed, we called it fine
Another corpse beside the line
Missing spec and broken test
Everybody knows the rest
We made a scar into a style
Then called the damage versatile
[Chorus]
Stop the line
If it’s wrong, don’t make it rhyme
Stop the line
No more later, no more fine
Don’t bless the wound
And call it shipping
Pull the cord
The whole floor’s listening
[Verse 2]
First time failed, we called it rare
Second time, we learned to stare
Third time, someone made a chart
Fourth time, it became “just part”
Part of launch, part of cost
Part of what we always lost
Fifth time, nobody pulled the cord
Now the defect owns the floor
[Bridge]
Normalization of deviance
Is how the broken learns to stay
Normalization of deviance
Is how the warning fades away
[Breakdown]
Is it fixed?
No.
Is it owned?
No.
Is it safe?
No.
Then why go?
Stop.
The.
Line.
Pull.
The.
Cord.
[Final Chorus]
Stop the line
If it’s wrong, don’t make it rhyme
Stop the line
No more later, no more fine
Don’t bless the wound
And call it shipping
Pull the cord
The whole floor’s listening
PULL IT NOW!!!
ooof - nevermind!
I likely am mis-using the label “nerdcore.” My knowledge of the topic begins and ends in 2008 with the oddly timed releases of Nerdcore for Life and Nerdcore Rising. I won’t take sides on my preferred film. Except say that I think we all probably should agree that Nerdcore for Life nails the correct title for the genre. This also reminds me that maybe I should have tried a hip-hop spin on the material.
Yes - I know some of the imagery is more “metal” than “punk.” I started thinking “normalization of deviance” sounded super metal. But when I started “producing” the album my personal bias of punk over metal came through. I’m thinking of producing a metal, and perhaps country version of the works. But first wanted to see if anyone besides me cared.