How a Product Owner Should Respond to "Bugs"
— 17:19 reading time
All software has bugs. From Microsoft Office to joeschmoe.com. Bugs are a reality. And a major part of representing a product is being a first line of defense against “bugs.” If users are allowed to, they will go and tell a developer every time they think they’ve found a bug. That’s the fastest way to a resolution. If what has been found is indeed a bug, the fastest way to fix it is to have someone who writes code be aware of it and write code to fix it. And in a perfect world, where users were wildly competent at diagnosing a system, this behavior might be okay. But this is not a perfect world. And it isn’t fair to assume that someone who casually uses software is going to know the ins and outs of each piece of functionality. And if every user had access to developers, nothing would ever get done. So if it’s your responsibility to own a product, it’s your responsibility to be the first responder to “bugs.”
The first and most important thing to establish here, is you need users and people to tell you about “bugs” instead of other people. This can be done in many fashions. I like to be very melodramatic and make a big show of standing up in front of a user base and saying, “I am responsible for the form and functionality of this product. If there are any issues with it, know that I signed off on them. They are my fault, and so I need to know about any and all bugs as soon as they occur.” The next tactic is to tell the development team something like, “If anyone tries to tell you about a bug or anything at all, tell them to leave you alone. Then point them in my direction.” Do those two things four or five times over the period of a month or two, and people will start telling you all about their problems. It will be terrible and painful at first. It will make you regret being a Product Manager. But eventually, it will become one of your greatest tools in building a relationship with users and understanding the weak points in your product.
The Four Types of “Bugs”
Now you’ve got everyone telling you all about their “bugs.” Pretty soon you will notice a pattern in the things they’re telling you about. You’ll recognize that there are Four Types of “Bugs”:
- Something in your product that used to work, but doesn’t work anymore.
- Something in your product that needs to work, but doesn’t and never did.
- Something in your product that should probably work, but doesn’t and never did.
- Something in your product that behaves exactly as expected, but the user doesn’t understand how or why to use it.
Here’s the ultimate reason that the Product Owner needs to be the first line of defense: Only half of the those “Bugs” are actual bugs. Numbers one and two are things that need to get to a developer. If something in your product used to work, but it doesn’t anymore, then that is a real Bug. And if something needs to work, but doesn’t, it’s probably a weird edge case that wasn’t anticipated when functionality was being developed. That’s a Bug. It needs a developer. The third “bug” type above isn’t a bug. It’s a feature request. This does not need to go to a developer, it needs to go into a backlog for consideration and development. And “bug” number four isn’t a bug either. It’s user error. That needs to feed into documentation and training.
How to Triage a “Bug”
The first thing you do when a “Bug” comes in is ask the user who reported it, “What steps did you take that created the bug?” Sometimes this is something really obvious: “I clicked on the ‘Publish’ button and nothing happened.” Sometimes, it’s really elaborate: “I used the 'Select All’ button, then copied all of the files in the system, then renamed all of the original files, then used the 'Move’ button to put the copied files into new folders. Next, I went to the original folder and copied the renamed original files. After that, every time I try to edit any of the original files, my computer grows a mouth and tells me it hates me.”
“Great. What is your username? What’s the urgency? What is the original name of every one of those files that you touched? Are you sure you did the things in the order you just described?”
Your next step is to take the answers to those questions and reproduce the bug. If you can’t reproduce the bug, either the steps the user gave you are wrong, the bug doesn’t exist and the user is confused, or the bug is transient. Try again if you can’t reproduce it. If you can’t reproduce it, it’s probably type 4: user error. Skip to the section on user error to understand what to do next.
If you can reproduce it, ask is it Type 1? This used to work and doesn’t anymore? Yes? Put it at the top of the backlog, tell the user it will get picked up as soon as someone is available. And get ready to explain the issue to the team.
Is it Type 2? Someone has found a weird little edge case, and now they can’t do their work? Again, top of the backlog. Tell the user it will get picked up as soon as someone is available.
If it’s Type 3 or Type 4, tell the user you understand their frustration, but that the product is functioning as expected so the request is going to go into the backlog and get prioritized against the rest of the feature requests. Show the user how to acheive their objective without the functionality they’re desiring.
Wait, Real Bugs Go to the Top of the Backlog?
Okay, there’s more nuance if it’s Type 1 or 2. We’ve established that the bug is a Bug. Now what to do with it. Is there a workaround? If there is, notify all end users that there’s a Bug and it isn’t getting picked up until someone is available. But tell them to be aware of it in the meantime. If there’s no workaround, we look to urgency and criticality.
Is the Bug critical to business getting done? Is the Bug a BUG!? After you’ve put it at the top of the backlog, go grab a member of the development team. Tell them to drop what they’re doing and take a look. Explain everything you know about the BUG! and why it’s important. Tell them which user was involved in the BUG’s discovery. And do whatever Q&A is necessary to get the developer started in diagnosing and troubleshooting the BUG. (Note, this is for critical stuff. i.e., someone can’t publish something that needs to go out; something crazy happened in production; etc. I’m not throwing all caps around lightly.)
Stay on top of the BUG, and communicate, communicate, communicate. Remember, as the Product Owner, you are responsible for every “bug,” Bug, or BUG.
At this point, if you do nothing else, making yourself the first line of defense against “bugs” has just saved developers from having to look at 50% of the types of “bugs” that come in. As a product matures, “Bugs” 3 and 4 make up 90% of the “Bugs” that come in. So you’re shielding developers from 90% of the potential distractions from their development. Hooray!
And if you’ve triaged properly, you’ve saved them from having to weed through the vague conversation the user is probably having with you. Believe it or not, lots of bugs get reported as, “This thing doesn’t work.” You can’t reproduce that. Neither can a developer. By nailing down exactly how to reproduce the bug, you’ve given the developer a killer starting point in sniffing out a root cause. You’ve also established a rapport with your team. And you’ve given them a common reference point for user behavior: you.
But there’s more you can get out of this, too. And it starts with how you respond when the “Bug” comes in. Let’s take a deeper look at each of the four types and how you can respond to each to get tons of value out of even the “bugs” that get reported.
Type 1: The Bug (Regression)
Something in your product used to work, but it doesn’t anymore. You’ve done triage. You’ve gotten it to a developer. It’s been fixed. Back to normal. All done here, nothing to see folks. Right? WRONG!
If there’s a regression, ideally that gets caught before the regression gets deployed to production. This type of bug is telling you that something in your products testing layer is lacking. Was there a unit and/or feature test covering the functionality that had a regression? If no, the development team should fix the bug and put the appropriate tests in place. If there were tests, the development team should fix the bug and look at the tests that were in place to see if they’re testing the right thing.
These things can be tricky. If it happens once in a while, bummer.
If you’re seeing regression after regression after regression, there’s something else that these bugs are telling you: stop prioritizing new features and start having a serious conversation with the team about regressions and test coverage.
Type 2: The Edge Case (Also a Bug)
Something in your product doesn’t work given some elaborate way that a user is using it. You didn’t anticipate anyone using the feature that way. But there it is. And it doesn’t work. It is obvious that it should work. It needs to work. An example of this would be you put an 'Undo’ feature in your product. And you made sure that 'Undo’ works for 9 out of 10 actions. But you forgot to think about that 10th action. You can’t tell people, “We now support 'Undo’!” And then tell them your intent was to support 'Undo’ for 90% of their actions. That’s an edge case that’s pretty obvious, not very edgy. But it’s the same as the more elaborate edge case of the user who copied a file, deleted a section, and now the UI still tells me that the section is in the file. But this problem only exists if you copy first, and then delete a section. Your intent was still that the UI should always tell you reliably what sections are in the file. It’s an edge case.
Again, you triaged. You fixed. Back to normal. What else is this bug telling me?
Look at the UAT/Release Process?
If it happens once in a while… “whoops.” It happens. Take responsibility for the goof and move along. If it happens and it’s a really elaborate, deep edge case, take it easy on yourself.
If it happens a whole bunch, take a look at what you’re doing before you sign off on pieces of development. Especially if it’s not a very deep edge case. Are you spending adequate time doing UAT? Are your acceptance tests comprehensive enough? Are you spending adequate time talking through the features with the development team?
This is a tricky one, because ideally you’ve got a QA analyst to sniff out test cases. That is, you’re not doing acceptance testing and QA. But that’s not always the case, and it’s not always possible. And maybe given your time constraints, you are adequate enough. It’s just a reality that the edge cases will get through. And maybe that’s not terrible. But those are the first questions you’ve got to ask when you see these getting reported over and over again. “Is this okay?”
Look at the Users
The second thing you can learn from edge cases is user behavior. Is the edge case extremely complex? Yes? Okay. Is the user who is going through the complex workflow and finding this bug compensating for a shortcoming in another feature set? i.e., they’re taking five actions to compensate that there isn’t an easy way to the objective in a single action? Maybe the edge case is priority one, but priority two is prioritizing a feature that makes the edge case unnecessary. Are these edge cases being discovered because you’ve made it too hard for the user to do what they need to do? Have you lost touch with the user’s reality?
Conversely, are the edge cases always being reported by the same user/user group? Does that user/user group have different use cases than every other user/user group? Does prioritizing a feature to support that group take away from the 10 features that everyone else needs? Is that user/user group part of your target audience?
Is that user/user group isn’t using the product in the optimal way? Is there an easier way to accomplish what they’re trying to accomplish? Have you coached them on it? Are you sure you understand their workflow?
The point here is that these bugs tell you more than just that something is wrong. By being the first line of defense against bugs, the Product Owner stays close to the user base. You can continually ask yourself these questions, stay on the front line of your product, and keep gaining richer and deeper understanding of your user base. And that’s going to inform your roadmap, your backlog, and even your release and development process.
Type 3: The Feature Request
Some feature in your product doesn’t exist. The user thinks it’s a bug. It’s not a bug. It’s just that you have limited resources. You thought about spending two more sprints on making one feature super robust. But you knew that the robustness would be used by 5% of the users once a year, and would make it mildly more convenient for the user to achieve their ultimate goal. At the same time, you knew that the next feature would be used by 100% of the users every day. You had to move on. Now a member of that 5% is telling you they’re trying to accomplish their goal, but the widget they’re using doesn’t make it as easy as it could to accomplish the goal.
First and foremost, you’ve got to ask yourself, should you have made that feature more robust? Do a gut check. No? Okay, explain that.
Now, sitting in this position gives you the opportunity to ask all of the user/user group questions mentioned immediately above (see 'Look at the Users’). But it also gives you the opportunity to explain to the user/user group how you prioritize things. Give them a peek at how you think about things. Give them an opportunity to tell you that you think about things wrong. If they’re really angry and adamant that things are stupid, and they mention that other people think so too, you’ve just uncovered a gold mine. You know that there’s a problem with how you’re doing things. If they say, “Yeah, I get it. That’s fine.” Then maybe you’re mostly aligned with your users. Fantastic. You’re making yourself approachable and invested in your users. You’re building trust.
Maybe the user/user group doesn’t know about your roadmap. Or your backlog. Or how to submit a feature request. Tell them. Maybe they don’t understand the difference between a bug and a feature request. Or they don’t know there’s a process or feedback loop for these types of requests, so they’re waiting until they’re super frustrated and filling out a “bug” because that’s the only way they know to get something to you? This is a great opportunity for education on those processes.
Type 4: The User Error
Some user clicks on some button, and doesn’t agree with what happens next. Or they report a “bug” and you can’t reproduce it, but then you watch them do something and you realize they’re misunderstanding terminology or the product’s visual vocabulary.
Again, this is great feedback for you. If the user doesn’t understand, you’ve got to assume it’s your fault. Always be hard on yourself when it comes to your Product. Subscribe to Leffert’s Law.
Is the terminology you use terrible? Is the UX total garbage? Do all users have problems with this? Yes? Great, you’ve identified a problem again. Now you just have to fix it.
If no? Do you have documentation? Is it good documentation? Does the user know it exists? Is it the first time this user has had this type of “bug”? Sit down with them. Coach them through it. Then follow up with sending them the documentation.
Is the process/feature more complex than it needs to be? Ask all of the questions under “Look at the Users.”
And finally, is it the same user/user group all of the time? Maybe it’s not you. Maybe it’s them. Note that this individual is a squeaky wheel. Take their feedback with a grain of salt. There are key users whose feedback will be more valuable in informing the product. And that’s another good thing to know.
The Wrap Up
Put yourself in the middle of the path between users and developers. The worst case scenario is you do some level of triage on incoming “bugs” and protect the development team from distractions that take away from the next feature’s development. The best case scenario is you identify where you’re failing. You establish, build, and maintain a valuable relationship with your users. You’re gathering constant feedback. You’re staying close to users’ workflows. You’re identifying key adopters to inform new feature sets. You’re seeing patterns in usage and segments in the user base. You’re constantly learning about your product and your development process. You’re turning the “bugs” in your product into gold mines to keep your Product optimal.
Director, Product Development