Problem solving in someone else's world: A fun trick to try at home or work

An Amazon war story of getting the Buyer-Seller messaging service off the ground

You cannot tell anyone anything. Think about this and be sure you agree with me. You can only help people see for themselves. - Jim Camp

Negotiating well by not focusing on what’s important to YOU

Companies these days all have leadership principals. Amazon’s are particularly famous - and super useful. Convoy had them as well - with “love problems not solutions” being particularly resonant for me.

In that spirit of loving problems, today’s story describes a problem we solved way back for Amazon’s marketplace. How we were enabled to solve it rested on two moments when we realized that “the problem” itself was different for different groups - even if the solution was the same. I firmly believe that the only way to really get agreements that stick is to be able to frame the problem in someone else’s world or pain (or problem). For a fuller treatment of how to discuss/negotiate/think this way I’d suggest either Start with No or The Mom Test¹.

I’ve hinted at this my my earlier writing about technical debt - reframing it in a way that makes sense as a real problem in the world of different constituencies. Today I’m going to share a war story about a system that we’re super glad to have built and which continues to provide big value at Amazon through the present day. We were only able to do this by continuing to explore with curiosity how the current reality impacted different groups differently. Understanding the problem from other’s perspectives meant we not only were able to move forward, but we actually built a much more complete solution than we would have otherwise.

A special thanks to the Buyer-Seller Communication team: Thomas Park, Aaron Eakin, Jeffrey Lui, John Sitar, Bijoy Nair, Hiral Bhatt and Greg Niejadlik who made the magic happen. And to the many patient execs who heard us ask to build this thing (over and over) along the path to eventual alignment. Including Joseph Sirosh, Brian Valentine, and Sebastian Gunningham. With a special thanks to the insightful and incisive Jeff Wilke. Who in a key way is the hero of this story². Read more to find out how.

The Buyer-Seller Messaging service story

The problem

During my tenure at Amazon I spent the single largest chunk of time there working on 3P performance management issues. I’ve gone into this in some detail in the story of Order Defect Rate (ODR) wherein a unified measure aligned multiple work-streams along with Sellers’ behavior in a constructive way. But that wasn’t a magic pill by itself and we still had lots of underlying issues to deal with. Systems within the Performance Management world included Seller Feedback (how to solicit input from buyers about their experience), operational tools (how to provide efficient views of data and skill based workflows to support human investigators in decision making), and claims processing (how to take complaints about orders direct from buyers or via credit card companies and resolve them quickly and fairly). Along the way folks identified a gap in our knowledge - the direct communication between buyers and sellers on the platform.

I won’t even remotely claim credit for identifying this knowledge gap. I believe it likely came out of the A-Z claims team. That org was vocal about the benefits if we could observe the discussions between buyers and sellers. When the marketplace was designed if a buyer had a question they just emailed the seller, and if the seller needed to communicate with a buyer they similarly just shot over an email. Both parties had the other’s direct email, and as such Amazon was totally unaware of things going on between them. Until at least, the buyer complained about the seller in the form of a claim. Then one or both parties might show up to the A-Z claims team with tons of documentation defending their perspective of what went wrong.

There were two problems with the situation. First, the emails could have been written by anyone (someone who watched too much Law & Order would point out that there was no “valid chain of custody.”) The review teams didn’t really know how much validity to give the content. Second, a claim was what we’d call a “super duper incredibly late lagging indicator” of a problem. Even something as simple as the number of emails a seller got per order, and their rate of timely response would likely give you a huge amount of earlier insight into later outcomes. Once this is said out loud it’s so obvious you’re probably visually nodding. Platforms such as AirBnB and others show the average response time for sellers/hosts as a way to behaviorally drive good customer service and build trust. But back in 2006 (give or take a year) this was a less obvious conclusion. More importantly, reaching this conclusion at Amazon back then wasn’t especially useful. Parties had been emailing each other directly so seeing a response rate or the content of the communications wasn’t possible.

The A-Z org saw this problem and proposed a solution - a web based communication mechanism between the parties so that buyers could communicate on the website and sellers could respond with other online tools. I don’t really recall all the versions that were tried - there was work done on a web to web version (both parties had to communicate through a webform), as well as a one directional one where buyers initiated a contact on the website but sellers just received a direct email. I wasn’t deep into the goings on at this point in the story, but these approaches didn’t take. My understanding was that there was a large amount of seller pushback and without a truly closed loop system (where everything was visible) the juice wasn’t worth the squeeze. ie: the ROI on direct and future opportunity cost of dev time and maintenance wasn’t there - for the more business oriented readers.

My recollection was that the A-Z org was unsatisfied where things were left off and proposed continued investment in the area for at least a few budget cycles (OP1’s for Amazon folks) without success. The seller performance teams (aka - me and our merry band of well intentioned pirates) thought this as a real tragedy because we had become convinced that the data in the emails would be a treasure trove. Letting us detect problems earlier, as well as resolve issues with a verifiable historical truth. But at the same time we realized that there just wasn’t appetite for the funding, and I had begun to suspect there was a structural problem with the solutions that had been proposed before.

The first solution - Nailing the seller’s problem in all this

Many of us were excited about the idea of being able to have access to all communication between buyers and sellers around their orders. This resulted in lots of kibitzing (shooting the shit for people not from NYC) about what we could do about it. Eventually folks converged on that earlier adoption failed due to expecting Enterprise sellers to use web tooling instead of their multi channel CRM. As big as Amazon was (even then) most sellers sold their products on multiple channels/website. As such they had built or purchased tools to support their customer service needs, and switching from that over to an Amazon only one encountered steep resistance. After the web only approach didn’t work I believe the teams had proposed randomly generated email addresses of buyers - which was ideal in the sense of keeping transactions and identity separated, but still caused big issues for sellers who wanted to track identify over transactions (or even in response to a single transaction) with the goal of consistent customer services.

There were ways around all of these objections, including solutions such as pressure as Amazon was a big marketplace even then, incentives (bribes), or building new API’s. But all of these required the sellers to do a bunch of work - and like everyone else drinking from the firehose of e-commerce, doing more work wasn’t likely. Especially for a problem/opportunity that in our most honest moments we’d admit was hard to quantify. It probably didn’t help that if you were a seller you’d probably prefer not to have Amazon all up in your business all of the time.

As these things happen in large companies, some folks left for greener pastures and eventually the larger seller performance org, including A-Z claims ended up reporting to me. I loved “reading all communications” idea and really really wanted it to work for us. We sort of put together an unofficial working group that would chat from time to time about how much it sucked that we didn’t have a path to solving for this email visibility/transparency. Based on our new assumption that a web based approach was a nonstarter, we pivoted towards “just use email in both directions.” Some of our engineers mapped out how such as system would work - reliability was extremely key as we were beginning to think of assigning someone in the process an “amazon email address” that served as a relay between the parties. But we were still a little stuck as to how this might work in practice from the product perspective.

One day I was having a coffee at Starbucks³ and staring off into space thinking about the problem part of a possible solution came to me. If the issue was that sellers needed to keep using their CRM for a solution to be palatable then keeping emails as the primary index between parties was key. If we wanted to know what was going on that email had to go through Amazon controlled email domain. That part we knew - but just giving everyone an Amazon email address and forwarding didn’t seem ideal. Somewhere in the midst of that cappuccino it came to me that using the email address itself to encode some key information would be useful. This idea was greatly expanded on by the team (especially Thomas Park who worked out the core bits) to fully encoded the buyer-seller relationship. Meaning that given an email address we would be able to both see all communication between the parties (because the email went to an Amazon domain) but also we could from the email itself know which buyer and seller were communicating. The email “addresses” wouldn’t be easily readable for a human - but it could potentially work.

As envisioned this had several benefits (not all of which we recognized at first)

• We could see all the communications between the parties because it went through an Amazon domain.

• Each seller had unique email addresses for the same buyer, which reduced an Amazon privacy concern that a seller could resell the email addresses of our customers.

• If the seller was compromised by some sort of malicious takeover (which happened from time to time) we could invalidate the email addresses and protect the customers from phishing schemes and the like.

• We could protect buyers from phishing by automatically removing unsafe content from email before it reached them, and by providing an online version of each message provide a way to validate “authentic” communication.

• We could monitor all communication for potential buyer (or seller) problems and intervene quickly if we noticed anything from non-responsiveness to direct written abuse.

• Most importantly for adoption, it still worked on a consistent email as the primary key across all purchases for a given seller. For existing enterprise sellers they might need a one time guide to which “new” emails mapped to older customers, but it was a much smaller lift than anything we’d suggested in the past.

Craigslist at the time had a similar but less complete approach that worked in only one direction. Once you wrote to a seller whose true email was encrypted behind their relay any response revealed their full contact info. When we went out and spoke to sellers about the idea there was some concern, but we felt it was addressable. We’d learned enough to ensure we framed the problem Sellers faced when they couldn’t rely on their email records to contest an A-Z claim or service chargeback effectively. Allowing us to explain the new communication system as a solution to that problem gave us traction in them hearing us out, and incentive for them to be OK making the required shift in process (even it if was relatively minimal).

With this extra idea about encoded email addresses our engineering team felt they could make this work. I’m sure we all celebrated our brilliance and prepared to wow the marketplace with our new data driven insights. Until we realized the 2nd problem…

The second solution - creating vision, about coupons?

We were very proud of our business/behavioral/technical solution to the problem. The patent lawyers were impressed too - filing a patent on the idea, and then somehow spinning off two more⁴. Though it quickly became clear that patent attorneys who got paid to file patents were an easier audience than SVP’s with tight budgets.

There were several attempts to sneak the 4 or so engineers we felt we needed to launch the program into the budget. This wasn’t the sort of thing we wanted to do as an off the books skunkworks because in truth it was quite risky. If we didn’t have a bulletproof technical solution and we disintermediated buyers and sellers in a bad way (by crashing and losing their emails) the entire marketplace would be at risk. We were experienced in building systems to stop problems, but had not previously built something that was Tier-1 - ie able to stop people from transacting. Having a team with programatic focus on this problem was a condition to building and launching.

At the time our org reported into Brian Valentine who in a memorable fashion asked me at the start of an (OP1) planning meeting something along the lines of “If I open this document is there going to be that ‘Communication Manager’ thing in there? There’s always some project that the engineers are really hell bent on doing and they bring it up year after year.” I laughed and told him that yes, it absolutely was in there. I didn’t take this in a bad way - and I certainly now and then recognize that just because something is super cool doesn’t mean it’s super important to the business’s goals. It’s easy for anyone to get caught up in that feeling if you love tech. But we felt we had a real problem that needed solving, so we kept with it. That discussion (and many others) taught me that I hadn’t really figured out how to build vision that this needed to get done.

On more introspective days I also forced myself to question whether I was wrong and this wasn’t actually the next high leverage question. At this stage in addition to ODR we’d also launched “late ship rate” and “cancel rate” as leading indicators of negative buyer outcomes, and this consistently felt like the next frontier for us. Therefore, per Brian’s joke we kept bringing it up.

Around this time I was lucky enough to get to spend more time with leaders across the seller business including Sebastian Gunningham. In conversations about Seller Performance and other topics I just kept listening with deep curiosity to understand the business and its opportunities in as many ways as possible. I’d picked up other problems about the marketplace that frustrated key execs not yet on my radar. A category of concerns for leaders was indirect, undesirable effects that marketplace operations could have on Amazon in the longer term.

One of the key agreements that sellers made with Amazon when they sold was that Amazon maintained ownership of the customer. This is a sort of inside baseball phrasing, but in simpler language it meant that if you got an order from Rich via Amazon, you couldn’t suddenly start ending him a bunch of ads and incentives to purchase from you directly - cutting Amazon out of the process. Of course sellers being business folks often took steps to work around this when they could. A common tactic was to email buyers a coupon (or toss one in the box) that promised a discount for a future purchase on the seller’s own website.

These coupons were a common topic of frustration for VP’s and SVP’s. Often during a Seller Performance discussion they’d mention getting a coupon with their Amazon order. We picked up on that being a problem that a closed loop email system could totally solve. But we still hadn’t gotten enough alignment for someone to let us risk building it.

Towards the end of this story we were presenting our annual plan (OP1) to the S-team team made up of Jeff Bezos and his direct reports. As is widely known, Amazon doesn’t do Powerpoint (thankfully!) so we’d written up the standard 6-pager along with many required appendices. One of the standard FAQ questions we were asked to fill in was “What are the things you most regret not having the ability to get to?” It probably was worded slightly differently - but that was the general point.

I’d been asked to contribute to the 1-2 sentence answer. Of course at this point the next most valuable thing we weren’t working on (in my view) was taking the big swing at closed loop communication. What I wrote for that “what are we sad about” question sounded something like,

“Given the priorities discussed in our document we are unable to staff a closed loop communication system which would give us insight into all written communication between buyers and sellers. This would allow us to better resolve claims, build new early warning systems for bad buyer systems and stop the sharing of off-platform coupons by sellers.”

To be completely candid, I didn’t think a whole lot about the addition of the “stopping coupons” reasoning. I’d just heard it enough times that I threw it in with the kitchen sink.

The meeting proceeded as those meetings tended to. Lots of tough and deep questions, but a lot of support for our past work and next steps. As I remember it, at the close of the meeting JeffB asked the group for any final thoughts. Jeff Wilke (aka JAW) spoke up and said “it’s a real shame we can’t do this thing to stop the coupon problem.” I’d be surprised if a video of that moment didn’t show a big smile on my face.

JeffB nodded at this point and asked “what keeps us from funding that effort?” At which point several folks jumped in with offers of incremental funding for us to staff the effort. I was going to write “fell all over each other to help.” But while that feels accurate in retrospect it just seems an obnoxious way to phrase it. Memory is fallable - so I’m sure that’s not what actually happened. But it does seem that way to me when I picture things now. ;-)

That amazing pivot gave us just enough opportunity to launch Buyer-Seller communication manager with the incredible scrappy team I thanked in the introduction. It did allow us to declare war on the coupon problem, make claims processing better, and get lots of new information to continue early intervention in buyer experience improvements. We also found new opportunities such as combatting anti-seller fraud, compromised account recovery, and making a dent in the biggest third-party pain point for the Customer Service organization (ie; letting them directly help in the communication between buyers and sellers).

In hindsight (like a lot of solid ideas), the whole thing sounds obvious because it could have been explained in a short text. The system we built was incredibly valuable. for me the true lesson is that being really curious about problems that are “not yours” can help find incredible opportunities for impact and alignment that help big things get done.

Though, if I had to do the whole thing again maybe “Buyer-Seller” wasn’t the best name. By the time Sebastian Gunningham pointed out we were naming something the “BS messaging system” it was a a little late to change. FWIW we called it Communication Manager up until that point which was a bit less problematic.

Thanks for reading!

1. A special thanks to Yuzheng Sun for making me aware of “The Mom Test.” I feel it approaches a similar key insight that you cannot convince anyone of anything unless it solves a problem in their world as Start with No, but from a very different, constructive perspective.

2. Jeff Wilke had several moments contributing to my growth that I’m sure he’d not remember. The other really important one was when I was explaining other “important” metrics we used beyond ODR to assess the market - including cancel rate. He asked me the cancel rate for the US market and I had to look it up. He remarked rather pointedly that if I didn’t know the current number then it wasn’t something I actually felt was important. It stung at the moment but I quickly realized it was a very, very valuable point. Also - how could I stay mad at the guy who saw my inclusion of my immature FAQ question in an OP1 doc (“What is the airspeed velocity of an unladen swallow?”) only to later throw in his own movie reference “We want you on that wall, we need you on that wall!” in his best Jack Nicolson impression.

3. It turns out I get a lot of good ideas when staring off into space, a number of them are at coffee shops, many of those are Starbucks. So if anyone from #Starbucks is reading this and would like to sponsor a post, or this newsletter let me know. In all seriousness your establishment is the 2nd or 3rd most likely place for me to have a breakthrough idea. And in terms of sponsorship budgets I’m sure you’ve got more to spend than “big shower” or “big walking” conglomerates.

4. Crazy, I know, Lawyers!!!