Transcript

Rich Ziade You know, you’ve arrived when your engineers are saying: ”we can make this a little bit better.” That’s a huge moment. Right? And that’s not QA in a classic sense. [music fades in, plays alone for 15 seconds, ramps down]

Paul Ford Rich, we’re lucky to welcome back Chris LoSacco who runs product at Postlight. 

Chris LoSacco Hello again.

PF Chris, not too long ago, as we, if you listened to the previous podcast, you know about this, he sent us an email saying: ”look, I want to talk about the ways, the mistakes that people make when they’re in product.”

RZ The six mistakes that people make. Yes, yes.

PF So we’re doing two mistakes per episode, cause it’s a lot of mistakes. Uh, and so this episode, we’re going to talk about one of my favorites. 

RZ Well, let’s, let’s restate the first two mistakes. 

PF Okay. Being vague about your timeline and commitments and number two is waiting too long to ship. Those two are kind of combined. 

CL Yeah. They go hand in hand. 

PF Now we’re going to get into the nitty gritty, less, less about scheduling and more about actual tactics and things to do. One is… and so we’ve got two more. Number three: neglecting your tooling. And number four: single sourcing your QA or not doing it at all. So let’s talk. Tooling, I love. I love a good tooling conversation. First of all, let’s define tooling. 

CL I take a very broad definition of tooling here. I don’t think tooling is one specific, you know, project management suite or where you store your code. It’s kind of all of it. It’s all of the software that your team uses to get its work done. And I would even say it’s partially about process, making sure that you have a clearly defined way to do things. That’s how I define tooling. 

PF So not just Jira, not just GitHub, not just JavaScript. 

CL That’s right. 

RZ Give us an example of tooling that’s been neglected like, Oh, and then the damage it caused. What’s a good example of, of that?

01:57 

CL Here’s one: error reporting and debugging. How do you know when something is going wrong in your platform? Obviously when something’s going wrong in production, you want to know about it. But even when something is going wrong in your staging environment, or as you develop locally, making it really clear that you know how to trace through the stack and understand problems is very often just not something people think about because they’re not planning for those worst case scenarios. 

PF I’ve got, the example that always comes to mind for me is that there’s always a hot path when you’re building a system and you’re trying to like get the data on screen. It’s coming out of the database and so on. And somebody hacks together basic auth at the beginning of the project, like I’m going to log in. I got, look, it’s all working fine somewhere along the line that breaks [yeah] and now, and nobody pays attention. Cause nobody’s really logging in. They’re like they’re simulating logging in or they’re automated logging in. And then it’s just utterly busted and like 20 other things have changed. And now you’re two weeks out and no one can log in. 

CL Yes, exactly. Here’s another one that I love: not using production data in your staging environments or in your local environments. You need to be building with things that are real. And part of that is building, you know, investing a little bit in your tooling set up your infrastructure so that you can test with real data, build against a real data. Those are the kinds of things that are very often just never handled. You know, it’s fine for now. So we’ll just leave it and then sure enough, it inevitably bites you when you’re six, not even six months down the road, sometimes it’s, you know, the first, first release that goes out there and it doesn’t work like you expected.

PF We’re fine for now is the, is the big card right there, right? [exactly] You got to listen for that.

RZ Well I think what’s unusual here is this is often, these things you’re talking about. I would say most product managers in the world don’t feel like they have the authority to have an opinion about these things. I think they feel like they are more on the customer side. They’re more on the ask side and tooling is solely the domain of engineering. And I think what you’re highlighting here is really important, which is the product won’t be as good. 

04:09 

CL It affects what actually gets to real users. And unless you’re ahead of it, unless you’re defining those tools and those processes to make sure that things are caught. It’s not even everything being caught by the way, it can be things like a design review process. That’s not engineering, but it’s still important to make sure that it’s clear how things get signed off and moved ahead. We’ve, we’ve joined teams at clients where things were taking, you know, two, three times as long as they should just because it wasn’t written down who needs to look at something before it gets in the hands of the team.

PF Classic narrative here, right? It’s like, cause everything can, you can stand up a new development environment in a, in a well constructed place in like a minute. [mhm] And then you’ll go into a giant organization and it takes a month and or the design review process, everything has to roll through marketing and then marketing goes on vacation. Like it’s so looking for the now what do you do when you don’t have power, when you don’t? Cause we often go and we go into these environments in which like we don’t have full control.

RZ Yeah. I mean, I, I wanna, I want to ask it in a more pointed way. Essentially this question you’re asking is like, how do you approach this, right? Cause to be that forward or that I’m going to use the word controlling, right? I mean, let me be the salty engineering lead. It’s like, what are you doing here? 

CL But flip it on its head. I think it’s the responsibility of that engineering lead and really the whole team to come together on what the process should be. So it’s not just the product person who’s coming in and saying, I need to make sure that you’re using Century for error reporting and you know, Tickets and Clubhouse to do your design sign off.

RZ Listen, listen, Chris, Chris, my guys are not in the mood for all this, with all the ceremony. We’re good. We, we work a certain way. 

05:56 

CL It’s not about ceremony, it’s not about process for process sake. It’s about optimizing your setup so that you, tech lead, you, engineering team can be more productive and also catch things before you have to deal with them on the other side.

RZ See I could keep going with this because you’re touching on like, Hey, you know that garden hose you bought, that’s my toolbox, right? You know, people are into their tools and you’re coming in here and telling me I’m using the wrong tools. And this is a very sacred place. This is a very sacred box for me here. 

PF No, we have to call this out because this is part of the job of the product manager. And it often doesn’t get discussed. Right?

RZ This is why I’m pressing this. [yeah] Because I think most people don’t think product managers have this kind of power and authority.

PF A lot of product managers kind of stumbled into the role because they seem to be good meta thinkers. And they’re coming sometimes from engineering, sometimes from design, sometimes from the moon. Like, and the last thing that they’ve thought about is engineering tooling. It’s never, they’ve never deployed a, you know, they’ve never had somebody launch into them about Kubernetes for four and a half hours [right] in the middle of an all day all-hands. So it’s also, we take this for granted. We have deep, long standing relationships between the leadership, the leadership of product design and engineering at Postlight. So like it’s not antagonistic. But Chris, but when Chris goes into an environment he’s not afraid of engineering or design, they can’t really tell him ”no, no, it can’t be that way” because he’s seen something different. And he also just knows it. He did the homework. Right. And I, a lot of PMs in the world don’t want to do that homework and don’t want to care too much about Jira or Clubhouse or all those things. They want to defer those cause thsoe are boring, tricky questions that kind of suck. [yeah] And that’s what we’re hearing is like, you have to go in and do your homework. 

CL You have to do your homework a little.

RZ Let’s, let’s acknowledge something here. The engineering holds all the power, right? I mean, they’re the ones, without them, if they go salty on your, if they pull back and disengage and get defiant and whatnot, you’re not going to get your thing. Right. I think this is a thing to highlight that Chris, I don’t think he knows this. This is a skill I don’t think he knows he has. Where he just asks for the thing again and again and again. And he’ll say, well, why don’t you do it this way? And I think it would be better if you did it. And then he’ll give a reasonable answer to it. And I think for most product managers, they don’t feel like they have the authority and the, the, the domain knowledge to, to step. They ask.

08:20 

PF Well, in particular, in a lot of orgs, the CPO rolls up to the CTO. Like it’s, it’s actually encoded in the structure. That product is subordinate to the larger tech org.

CL I mean, the advice though that I give to, to product leaders out there who feel kind of powerless to do this is to ask those questions and to reframe the conversation a little bit so that it’s not the product team dictating here’s what you need to do. But rather the product team saying, how do we want to do this? How do we want to optimize our flow? So that, so that we’re super efficient, right? I mean, it’s not about, to go back to the analogy before about the toolbox. It’s not about the, you know, the product lead saying, you’ve got to use this hammer and this screwdriver and this wrench, it’s more like, which hammer do we need here? And let’s have that conversation and pick the hammer and then commit to it and move ahead. 

RZ Look, I’m going to, I’m going to highlight a quality that you have Chris, that you don’t even know you have, which is you use the questions, the line of questions is almost like a weapon. Which can see can land kind of passive, aggressive and annoying and nagging and probing and overstepping. But boy, is it powerful! [Chris laughs] Once LoSacco sees something, he can’t unsee it. 

PF I’ve never been able, you can’t convince Chris of anything. I learned that very early in our relationship, [Rich laughs] and it’s not…

RZ This is becoming about Chris now!

PF No, but it doesn’t, no no, cause it’s, it’s fundamental, right? Like his quality bar is there. And when he sees something that isn’t good, he needs to understand why and what happened, and debug it. And what I’ve seen. I’ve watched people over and over go up against it and be like, well, this reason, and this is why, and this is why, and this is why. And he’s just like, no.

09:59 

CL Yeah. And what I’ve tried to do with our team here, and what I hope others can do is, is not again, not say ”you need to be doing this” in the, in the way, in the exact way that I would do it. Right? You need to use Clubhouse to track your tickets and Basecamp for your documentation and GitHub Actions for blah, blah, blah, blah, blah. It’s more that you need to care about this. And all of our PMs, all of our product people care about making sure that the tooling is right. And working with engineering, working with design saying let’s make sure that we’ve got a clearly understood process that we’re not just doing because we want to go through the motions, but we’re going because we’re doing it because it’s valuable to our team to make things run efficiently. 

PF Right. So we’ve been talking about building, but we also need to talk about QA, [yes!] right? So mistake number four, single sourcing your QA are not doing it at all. Alright. So quality assurance, what is single sourcing? What does that mean?

CL Relying on one person to do your QA. We see this happen all the time. There’s either, there’s a team with, with no QA resources. And so they just hope for the best, the engineers write some unit tests and then they put it out into the world and hope that there’s no problems or you’ve got one QA person that’s on your team or sometimes shared across multiple teams. And you say, Jane, it’s your responsibility to make sure that we go out with no bugs. It just not the way to do it.

PF So we get asked all the time, when people, big orgs asks us to pitch and we go in and we tell them about Postlight, always comes up. What’s your QA team look like? Or do you outsource QA? What’s your answer to that?

CL Yeah. My answer is we spread out the responsibility for assuring quality amongst the team. Everyone is responsible.

11:43 

RZ Everybody tests. I mean, if you could sum it up in two words and people, the quality that, I mean, I can say this stepping outside of Postlight, as an executive, the quality level of our software is very high. It’s exceptionally high in the world of software. And we don’t have a QA department. We don’t have a QA sort of services group that you can hand software to. Everybody has to test and that, and when we say test, that’s a deep pool in and of itself, I don’t mean just click around, but everybody’s got to do it. How do you, and this is again, you’re, you’re, you’re waiting into the domain of tech right. Of the technology group. [okay] Let’s call them a group for a moment. Is that something, do you think that the product manager can instill culturally? Or is that something you just got to hope exists within, within the tech group that everyone tests? I mean, do you walk around as a PM and say ”Hey, could you please test? You just wrote this piece of code, could you test it validated for me?” 

CL I don’t think it’s about, ”you wrote this piece of code. Can you please test it?” I think it’s more holistic than that. I think the, it is the PM’s responsibility to say we are going to draft an open ended test plan, and we’re going to make every person go through that plan on a weekly basis or on a monthly basis. We’re going to have the whole team contribute to doing the QA on our product. One of our product managers, who’s working on one of our large projects right now. She did this, she drafted a test plan. She shared it with the whole team. And every week we have a rotation that gets shared in Slack. And when it’s your QA day, you go in and you go through that plan and you’re using the software to try and accomplish [yup] what you need to accomplish as a user.

RZ Yeah. I think you’re highlighting something really important here, which is that I think the idea of QA as a discipline, as a siloed group is a byproduct is sort of a concession that software is just so damn hard and there’s just going to be problems, no matter how good you are or what you do. But I think instilling quality thinking [exactly] in everyone from designers to PMs, to engineers, leads to not only more stable software, but just better software [yes] generally, because they start to care about the quality of the thing beyond just the output was wrong. [yes] But also the thing could be better.

14:07 

CL That’s right. The whole team gets invested. They’re not just making sure that there’s no bugs. They’re saying, how do I make this experience great?

RZ Do you believe in a more formal QA process or do you think it should be integrated into the, into the designing and building? It’s just everywhere. Is it in your mind?

CL It’s just everywhere. There are some environments where you need to have a formal process as well. It is critical that you have, you know, we worked with a large investment bank. They had requirements around a very formal regression test suite to make sure that what the software that was going out was not exhibiting any issues that had had previously, that’s required. Some situations require it. But that even that does not mean that you can’t also say I want to distribute a more open ended plan to the whole team. And by the way, I keep saying open-ended because that’s important too. If you just have a set of wrote instructions, that people are just clicking the same buttons every time that’s not nearly as effective as saying, I want you to go into this CMS and post an article, or I want you to upload and crop this image. Tell me what you felt. It’s a little ambiguous. 

RZ So a little ambiguous. Yeah, yeah. So the person is hunting around. Yeah, this is great. Okay. That was tip number four. I want to, this is part of a series called Six Mistakes You Make When You’re Building a Product. I want to recap in, in episode one, we had the first couple of mistakes, which was really around getting out there and shipping often, [music ramps up] and being clear about your timeline and your commitments, both to your team, into your stakeholders. Today on this podcast, we talked about neglecting your tooling, also single sourcing your QA, to being a little too prescriptive about what QA is and boxing it off rather. And I’ll say the two words again: everyone tests. Everybody tests. From product design to engineering. On the next podcast, we’re going to have the final two tips, which I’m going to not mention them here, Paul, just to create enthusiasm and excitement.

16:02 

PF No, a little excitement, but they’re about relationship building. 

RZ They’re about relationship building.

PF Relationship, building relationships around your product and mistakes. 

RZ Yes, Chris, thank you again. You’ve been incredibly generous with this advice. Chris LoSacco is a managing partner at Postlight. We’re a digital strategy design and engineering firm based everywhere. Capital E. Right now we build big platforms. We design big platforms. We solve big problems for clients. We’re at postlight.com. Hit us up at hello@postlight.com. Thanks again, Chris. 

CL Thanks!

PF Can’t say it any better than that. Thank you, Chris. 

RZ Alright. [music ramps up, plays alone for 3 seconds, ends.]