פרודקטיבי 56: הפלייבוק לבניית קו-פיילוט (Ben Erez - English)

I think when we think about other PMs listening to this, is we try to frame it as either a challenge of how do you scale yourself?

And how do you personalize your answers when you're not there? And when we think about this framework of personalization and scale, I think this is the initial questions when we need to ask, when we could think about maybe a co-pilot is right for me at the situation. I think today what we're going to do is we're going to break down both through your use case, but also to expand it to other, let's call it Ben's playbook for a co-pilot, to really understand the nitty-gritty and everything that we need to think about when we think about launching a co-pilot to either scale ourselves, personalize ourselves, or both of them. Okay, sounds good? Yeah, let's do it. Okay, so let's take your case study and let's start. You said it was about a year ago when you started it. Maybe walk us through the steps you took to get it off the ground. Yeah, I would say the very first thing was just figuring out if it even works, right? So I wasn't thinking about marketing. I wasn't thinking about packaging or pricing. The original idea was I was wondering if I could build something that helped me do something as like an internal tool as an instructor. Then I started with a product sense question. In hindsight, I probably would have started with analytical thinking. I think it's actually kind of a little easier to codify analytical thinking interviews in this way than product sense. But nonetheless, that's the first step is just play with it for yourself. Build something that you could use that adds value to you and gives you leverage for something that you find yourself doing a lot. Then you started working on cloud projects, right? Yeah, yeah. Okay, what was that looking like in the real time? What did you do? Yeah, so the cool thing about AI tools at that point, cloud projects at that point, is there weren't that many levers to play with, right? So you create a project, and then there's like instructions, and there's files. That's it. Okay, so that's basically all it was, and it was doing rag to kind of try to come up with answers. And so the first thing I tried to do is I just didn't put in any files. I didn't put in any instructions. I just asked an interview question. So I said, you know, how would you, I think I might have asked like the most common question that I've seen over the years, which is something like, how would you build a product for volunteering? Yeah, common one. I may have said you're a PM at Meta, how would you build a product for volunteering or something like that. And I just kind of saw what it came up with. And obviously it jumped right into a solution. Like a bad PM, right? Like a bad PM. So I was kind of like, okay, I guess if I just, it would have no way of knowing that that's not the right way to do this. So what if I go in the instructions and I say, hey, like every chat in this project is going to be me asking you an interview question. And before you jump into solutions, I just want you to frame the context around why this matters. And then, you know, walk me through who you might build this for and then tell me what their problems might be. And then I want you to land on a solution at the end. So I just started very simple like that. And then I went back to the chat and Claude Projects has this button called retry. And you could hit retry with no changes. And what that does is it basically has the AI take another stab at generating the response, but it does pull the latest kind of instructions from the project. And then I got to see what it did, you know, with the latest instructions. And then I was like, okay, that's a little bit better, but this part is off. And I basically just did that like a few hundred times over the course of probably a couple of weeks just to kind of get the output from the response for product sense and analytical thinking to be what I would kind of expect to see in any passing answer. So once I had that all up and running, I pinged a few people that I knew who were calibrated interviewers from Meta. I'm like, hey, I built this thing for my course. I'm about to give it to my students. I don't want to take them off track and lead them astray. Could you just like ask it a question, ask it this question, right, or pick any product sense question that you like to ask in interviews and just like play the role of the interviewer. And just if you agree with what it's saying, say yes every time it checks in with you. And if you think it's not making sense, ask it a question like you would ask any other candidate. And then just like let me know when you're done and let me know kind of like your general assessment of, you know, like how was that? And one of the first responses I got back was in slightly different word. I'm not going to say who it was, but it was someone who was definitely someone I worked with at Meta who was a calibrated interviewer. It's not at Meta anymore. He's like, hey, like can I hire this co-pilot? Like I feel like this would be like a passing score for me. Yeah. Wow. So what happened after that is I gave it, because this was all happening again over the course of like a two-week period. So before I put it in the hands of my January cohort students, which was like a couple weeks out, I emailed all of my quote-unquote alumni.

who took my July and my October cohorts. And I said, hey, like, this is free for you. I'm just giving this to you. It's like, I took all your feedback. I listened to all the questions you had. I'm thinking ahead to my next cohort. Some of you are still in interview prep. Some of you already landed offers. That's cool. But like, regardless, it would mean a lot to me if you could just like play with this and tell me how helpful is this? What could be better? So that was kind of like wave two is like putting in the hands of the alumni from the course and getting their feedback. So I think here, if I highlight two things that are important is first of all, you gave it to the interviewers essentially to make sure that the bar is high enough. And then you gave it to the actual, let's call it the persona who is going to actually use it to actually use it and see how this goes and to get that essential feel for whether this gives personalization at scale, right? Totally. Okay. And then what did your alumni say? I mean, it was very positive, but I'm trying to kind of frame this in a way that doesn't sound like I was just… Like I genuinely was looking for what's wrong with it. And I did get like the overwhelming kind of feedback I got is like, this is incredibly helpful. And like, I felt like the, you know, the segmentation just felt a little kind of like shallow to me or it didn't really follow like what you said or it picked the supply side of the ecosystem when I think you generally recommend for this kind of question, maybe to pick the demand side or when it got into problem identification, it didn't do like… People were like, yeah, you always talk about how you should try to do a journey before you pick a problem, but the co-pilot wasn't doing a journey. It was just kind of going from like, you know, a segment to like the problems. And so I actually want to get more practice with the journey piece, et cetera. So I was starting to get kind of like specific pieces of feedback or like the mission statements were too verbose. So people were kind of like holding me accountable to like the stuff that I told them. And they were like, look, like I'm confused. I'm confused because you just gave me this tool and like I'm trying to figure out whether what you told me is the source of truth or what the co-pilot is telling me is the source of truth. And that created an urgency for me to be like, shit, like I can't put this in… I can't like give this now to a bunch of new people in January if I'm going to contradict myself, like if there's going to be contradictions in the system. So that's when I kind of went into like intense tinkering mode for like phase two of tinkering, which is like, okay, now I'm going to literally go section by section in both of these. I'm going to look at the last five questions and the question bank from Lewis Lin. I'm going to drop them into new chats and I'm not going to basically like be done with this process until like, until there's no contradictions between what the co-pilot does and what I would do. But you didn't stop there, right? So even after you tweaked it a lot and did a lot of feedback after your students' feedback, you continue to improve it, right? You had the ultimate test of making sure the co-pilot works. And I'd say right around the time where I was like comfortable giving it to my January students, that's when I felt comfortable increasing the price of the course to reflect, to like actually put co-pilot in the name. So that allowed me to go from like $600 to $1,000 for the course price. And I said, okay, like if someone doesn't want to take my course, but they want to get access to this tool, do I want to let people get access to it? And I was like, okay, I'll sell it separately and then I'll just bundle it in the course. But for me to sell it separately, to your point, I did have to be, that was like my first time ever selling a digital product like this. So I didn't expect anyone to buy it. I just, but then it started to happen because I linked to it from the Maven course page because I was like, hey, like this thing is included in the price, right? So people would go and check it out. Maybe they didn't want the course, but they bought the co-pilot. And I told people if they buy the co-pilot, then I'll just give them a discount in the amount of the co-pilot, which the original price was $299. So basically like if you buy it, I'll give you a $300 promo code on the course. So you'll be made whole if you end up wanting the whole thing after. Okay. So when was the moment that like this co-pilot exploded? It was probably April when I had my, April 2025, when I had my Lenny guest post go live about ProductSense interviews because that linked to all my things. That linked to my course, that linked to my templates, that linked to my co-pilot. And that also basically established my number one spot on like, you know, SEO. If someone looks for ProductSense interview and my stuff just pops up at the top. Yeah, I think I did like $10,000 of co-pilot sales that April. Wow. Yeah. Amazing. So we touched a lot of things about your case study, but I want to help our listeners maybe think about like Ben's playbook for building a co-pilot. And I want us to try and think about like the main principles that we talked about here and try to help our listeners think about their problems and whether a co-pilot is good or bad for this situation. So if I had to ask you, what would…

You say would be like the biggest principles you see. I would say the first one is to like really sensitize yourself to what you find, what kind of advice or what kind of expertise you find yourself using the most, where you would be comfortable saying, I might be like the best person at explaining this thing. Or there's other great people at explaining this thing, but for some reason, the way I explain it or the way I tackle it seems to really resonate with people. So trying to like really sensitize yourself to where you might have a superpower of some kind. And the reason that's important is because everything we talked about relied on my ability to evaluate the quality of the output. And if I was trying to do something that I was not an expert on or that I wasn't extremely opinionated with real proof points that my opinionated beliefs were backed by real results, then I would be shooting in the dark or I would have no ground to stand on as like a credible voice. And if you're just trying to build something for like for fun or just like, you know, as a hobbyist or you're not trying to like take like a high stakes thing like interview preparation, which is super high stakes, then, you know, maybe you could like loosen the requirements around that a bit. But if you are trying to bring your expertise into some kind of packaging that people would pay for and maybe even command a premium price for, then I would really try to like narrow in on that thing that you're uniquely positioned to do. So I'd say like once you know what that thing is, I would try to break it down into like what's the workflow in which that expertise fits. So for interview prep, it's generally like, you know, someone's getting ready for an interview. They may take my course. They may not take my course. They're trying to get ready. There's probably going to be mocks at some point. So where in that process does this thing fit? And where does your tool fit? Do you want them to use it during an activity, before an activity, after an activity? So trying to really be opinionated about how it might like when does it fit? Yeah, I would say that's probably step two. Okay. So step number one is like understanding what good looks like and the fact that the results that you're bringing as a human being actually produces great results in the real world and that you know how to determine whether this is a bad answer or a good answer. That's step number one. Step number two is actually mapping the user journey of your users and understanding where exactly in this process does this fit. Like is this a specific step before, during, or after? And like really nail down like the step that you need co-pilot or you could use co-pilot, right? Yes. But let me clarify what I'm… Okay. So like the thing you build with a product like this, it's still you could think of it as like a solution to a moment in time. You could think of it as a solution that allows people to do something repeatedly. If it's just a one-time thing, don't build a tool for it, right? But if it's like a repeatable thing that they have to do a lot of times, then you need to understand when that thing happens. So the iteration you do, like the step-by-step thing, is not about the moments in time in which they use it. It's about when they need it, breaking down the specific steps like within the factory. It's like there's a conveyor belt and making sure you're QAing every step of the conveyor belt. I would say if the thing that you are thinking about productizing in this way does not lend itself to like linear progression, then that way of approaching the development will be different. Like I probably don't have as much advice for someone who's trying to build something that does not lend itself to like a linear progression because the way that the LLMs work is like every, especially with interviews, every step builds on the previous step. So the solutions you come up with at the end of a product sense interview are built on top of the shoulders of the problems and the segments, etc. So you can't really just, you know, I've had people, why can't I just get right into the solutions? I'm like, because in a real interview, you wouldn't be able to get right into the solutions. You have to work your way towards that. Okay, so I think that's really important. So we said, okay, understanding what goes looks like, being able to actually build it bottom up and improve each step of the way, understanding the user journey and exactly which point in time does this bring value, and also mapping that within this step, it's a linear process of progression, right? I think that's about right. Okay, what else do you think is really important to think about when you think about a co-pilot like this? I think the scoping is a huge part of it. Like what we talked about earlier with like the, just trying to understand how much you're trying to bite off, because I was teaching a course about both of these interviews, but breaking them down into separate ones, we talked about that, that was an important thing. And then I would say like interviews kind of were a nice use case to get started with something like this, because they have clear evaluation rubrics that you could help someone understand.

what good looks like or what weak looks like. But a lot of life doesn't have evaluation rubrics, right? So I think like great management in general, like being clear about expectations and being clear with what good looks like and what good does not look like or what bad looks like and being able to articulate that in words, I think will be very important too. Like you can't, I don't think you can get anything out of this kind of product unless you also take the time to kind of create some quantifiable way for it to evaluate its own performance. Because that's the other thing I would do at the end of the, the copilot will also evaluate its own performance at the end of the interview. Okay, so I think evaluation is another key principle is like the fact that there is some sort of feedback loop between iterations. And I think that's critical, right? It's like, you don't just do it once. We said, if it's once, don't build the copilot, but you have to also, when you finish that linear progression to have clear feedback loop into the entire process. Yes. And then, and the other reason that helps is because I knew there was actually two kind of workflows I wanted to bite off. One was how do I work through a question I've never seen before? Like, so that was like the main one. But then also I just filled out this template. Like, what, can you give me feedback on my filled out template? So to serve the second use case, I had to build the evaluation rubric. And then I realized that if I could solve the second use case with the evaluation rubric, that the natural performance I got for the first case was, got significantly better because I had codified what the evaluation rubric looked like. So yeah, so my learning from that is you don't necessarily need to be building a product that has like a copilot that can evaluate some filled out template or something. But my learning from that was codifying the rubric for evaluation is something that would strengthen the main use case, the main workflow. Great. So Ben, we are about to wrap up and I would like to ask you one final question is like, what would be like a key insight you want our listeners to stay with after hearing our conversation today? I think that people, a lot of people dramatically undervalue how unique their knowledge is about specific things that they might be the experts on. And I think that in this new world we're going into, there's just like a lot of slop and there's a lot of, like I think it's getting harder and harder to know what content to trust and what content is reliable. And so my hope is that by talking more about my copilot, I don't want people just being like, oh, that's so cool. Like, I mean, I think it's cool, but that's not the point of this. The point of this is I want people to be like, there's nothing particularly special about me. If I can do this, you can do it too. But I can't do the heavy lifting for you as a listener or as an audience member for what that thing should be and how you do it. That's like the hard part is obviously you could lead a horse to water, but you can't make it drink. But I do want people to kind of potentially consider building a new type of product that they hadn't considered before. And just because you could vibe code software products now doesn't mean that that's like the best way to maybe package and monetize your knowledge. Maybe there's like a faster path to getting people what they need. And then as a consumer of your knowledge, like if I'm looking for expertise on how to improve my website or how to improve conversion in my funnel or how to better, like these guides that we're doing with insider loops, if someone has like spikes and they're like the top 0.001% when it comes to like information architecture and like guides and like written guides, and I could like buy something from them that just like allowed us to level up our guides. Like there's so many things, areas in my life, and I'm sure for you too, where you're looking for like what would the best person in the world do at this point with this thing? And I just don't think there's like a place to go and discover a lot of that right now. And I would certainly be a buyer for very specific things that fit into very specific workflows for very specific expertise. That's kind of, I guess like maybe we're moving into a world of kind of like micro expertise insertion points. And that would be really cool if someone chose to build something out of this. Amazing. To me is like when we think about whether I want to scale myself and personalize the answers, that's a good starting point. But the main takeaway from my perspective is resist the urge of building this top down. So don't go and create tons of instructions, but build this bottom up. Start with the baseline and then add gradually more instructions and make sure that you're controlling the output. And I think that's key to getting a result that would be gold standard and you can put your name on it. So Ben, thank you so much for this conversation. It was really, really interesting. Yeah, thanks for having me. This was fun. And for everyone listening, thank you for listening. You can follow us on all of the podcast software to make sure that you know when a new episode comes in. And again, thank you so much Ben for your time and thank you everyone for listening.

.