Ben has been an entrepreneur for 15+ years, with some of his companies being sold or acquired, and he is also a founding partner at Year One Labs, a lean accelerator, and most recently he co-authored Lean Analytics with Alistair Croll.

TOPIC BEN YOSKOVITZ COVERS

→ His background as an entrepreneur for 15+ years

→ What is Lean Analytics about

→ His motivation to be a part of a book

→ His thoughts on KPI

→ What is the importance to get creative

→ What are the changes in production would look like

→ What’s the difference in a segment cohort

→ What does that mean for analytics

→ How many startups get lean analytics

→ What is VP about

→ How important is a VP of product to a company’s growth

→ And a whole lot more

LINKS & RESOURCES

→ Lean Analytics

→ Ben’s blog website

WATCH THE INTERVIEW

READ THE TRANSCRIPTION

Bronson: Welcome to another episode of Growth Hacker TV. I’m Bronson Taylor and today I have been yuskavage with us. Ben, thanks for coming on the program.

Ben: Yeah, thanks for having me.

Bronson: Absolutely. Now, Ben, for the people that don’t know a lot about you, you’ve been an entrepreneur for over 15 years. Some of your companies have been sold or acquired. You’re founding partner at Year One Labs, which is a really interesting lean accelerator. And most recently, as if that’s not enough, you coauthored Lean Analytics with Alastair Kroll. We had Alastair on the program a few weeks back and he talked to us about some aspects of the book. He talked to us about the one metric that matters. He spoke to us about some vanity metrics, about causality and correlation and the difference, those kind of things. So I wanted to kind of pick your brain about some of the things that it didn’t cover that I know were also really important to kind of lean analytics, but to start out for our audience. Tell us kind of at a high level, what is Lean Analytics?

Ben: Sure. So for me, Lean Analytics is about using data in an effective way to make better business decisions faster. And in most cases, we think about it in terms of startups, but I think it applies much beyond that. So it can apply to large organizations and entrepreneurs, as we call them. I think it can apply or we. Get a lot of interest from from marketing people as well as product development people. So at all stages of business. So for us, if you if you’re familiar with Lean Startup and the concept of sort of build machine learning, which is a recent sort of way of doing things and iterating through the cycle. Really quickly, most people tend to be really driven to build. So, you know, we all have good ideas are what we think are good ideas. So and we can find really easy and cheap way to build things. I mean, really on the measuring in the part that we tend to fall down. And so I think that’s what Lena Robotics is really helping to do, measure the right things at the right time and then figuring out what to do with the data and how to make good decisions.

Bronson: So, Ben, what kind of gave you the motivation to be a part of a book like this? Was it just seeing the failures of people’s understanding or what was it?

Ben: Well, I think we learned a lot. So I mean, I’m always learning lots from from my years as an entrepreneur, from my own experience, through other people’s experience. Year one Labs was really the point where we had this lean accelerator and we were trying to help these companies really in a hands on way up to 12 months guidance from idea generation to ideally to product market to get out the door, raise, follow on capital and be sort of on solid ground. And so a lot of the ideas came out of it from that. But, you know, really at the end of the day, it was Alister who, you know, has a working relationship with O’Reilly and and I was the publisher and, you know, they were talking about writing something together around lean startup, around his experience. And and he said, Well, Ben, do you want to write this book with me? And I said, All right, sounds like fun to me. So, you know, I mean, we had a lot of experience learning about Lean and, you know, you’d see the struggles that companies were having with the measuring graphs part and really understanding where they are, what state. But a lot of it for me, it was just I think we had some good ideas and we did a lot of research. You know, it’s not just our and I sort of coming up with stuff and yeah, we think this is important. Let’s put it on a piece of paper. A lot of it is research driven and a lean startup to date has been theoretical in nature. There are clearly practitioners of this stuff that really know what they’re doing, but a lot of the stuff that’s been published has been theoretical in nature. And we said, Well, we wanted to get into the guts of it a little bit more and prove the efficacy of these processes in a way that would make sense for everybody.

Bronson: Yeah, no, that makes a lot of sense. Now, I found something that you taught before called the Lean Analytics Cycle. Right. And when I looked at that, it just made a lot of light bulbs kind of go off in my mind as I’m looking at this lean analytic cycle. So I want you to walk us through what that cycle is, and I want to put up the slide on the screen here. And people will kind of see the flowchart that you’ve created. So you say it begins with picking a KPI. Explain that part of the process to at the initial point there.

Ben: Sure. So the first thing you need to do, really, and then this is a this is not the very first thing. I mean, if you have a product or I bought some some data and you’re trying to buy some information, really thinking the KPIs, the key performance indicators, it’s the number that you think you want to change that’s going to drive some kind of solve some problem or drive some business value that you’re trying to recall. Right. So you’re not picking this number at random. Hopefully you have a sense of what numbers matter to you. And I mean, that’s that’s a lot of what the book try to help you understand is what number might matter to you at a given point in time. So this is really your metric that matters. So you pick this key, you say, I want to move this. I want to move the needle on this number. And so just to turn as an example, right, which is, you know, a common number for SaaS businesses, also for mobile businesses, most most businesses have simple churn with the percentage of customers typically sometimes users that abandon your service over a period of time. So that’s not what I want to effect churn, because I think if I can solve churn, it’s going to make my business school.

Bronson: So. So then tell me about drawing a line in the sand. What does that mean as your second step there?

Ben: Sure. So so this is basically. Picking a target that you want to hit for your number. So if we use churn as our example, we’ll say let’s say we know, first of all, you have to know what your number is. So let’s say churn is, I don’t know, 5% or 6%. And you’re fairly certain that’s not good enough to keep going? You know, it’s not you know, basically, no matter how many people you put through the top of your funnel, your bucket is too leaky, so you just can’t scale. This year, the engine is not going to run well enough. And so you want to lower churn. So you have to draw a line in the sand, which is pick a target. And this is very hard to do because there aren’t a lot of public benchmarks out there and we try to put some in the book. But these are just guidelines at best, right? This is not an exact science in terms of what is the absolute perfect number for your business. But but they’re definitely our targets. And for us as a business, typically a subscription model. So this is let’s say it’s about 10% churn per month is actually not bad in in most cases. Right. These are all caveated, but, you know, 2%, 2% percent. Yeah. So you have to draw a line in the sand because otherwise if you don’t if you don’t pick a target, how do you know if you’ve been successful or not? And the reality in most things that you do are not insanely successful or insane failure rate. They’re usually somewhere in the murky middle. And the goal of this process is essentially to say, you know, I’ve moved this this metric far enough in the past, and I want you to tell me that I can go on to optimizing the next thing in my product. And so that’s what we’re talking about here, is you pick this KPI, you draw this line in the sand, and I’ll point out that it’s in the sand because it’s a number that you can actually move. Okay. So let’s say let’s say we use churn as a stick with churn. So let’s say you get you know, you get from 5% to 3% or let’s say you get it from 5% to 2.5%. Now, you said two is your target. So you have a place that you keep optimizing and optimizing at closing. Or do you say, you know what, it’s good enough that I can keep going? And so you have that, you know, it’s so that you can say, you know what, for now, maybe I don’t over optimize this now because you can really invest a lot in optimizing for diminishing returns. Absolutely.

Bronson: That point five that you’re trying to reach may take as much as it took to get that initial 2% off. So it may just not you know, it may be diminishing returns, just like you said. Absolutely.

Ben: Exactly.

Bronson: So after you draw that line in the sand, you have a KPI, you’re trying to reduce your churn, you put a line in the sand, we’re trying to get to 2%. The next thing is you try to find a potential improvement. So tell me a little bit about that.

Ben: So so now you’re at the stage where you say, okay, I know what number I’m going after, I know what I want to get it to. Now it’s about running experiments, right? So I know you could run multiple experiments at the same time. Right. So, but, but really the point of this and, and the point of the one metric that matters is that everybody is aligned, focused on this thing for a period of time, not forever, but for a period of time. So. Now you’re looking for improvements. So you’re digging into the numbers and other things that might affect churn, for example, customer complaints. And you might say, hey, you know what? Our customer complaints are pretty high. What about customer complaints? I wonder if that would impact churn. Or maybe we need to look at our marketing strategy for reengaging customers when they’re about to spend it. And can we sort of predict when people are going to abandon our service and can we recapture or do we just pick up the phone and start calling people and figuring out what problems they’re having with the product and then maybe try to address those through product. So there’s a whole host of things that you can do with your business. And it’s not just about feature development.

Bronson: Yeah. Now, is this where it’s really important to get creative? Is this where you sit around and you brainstorm and you think through like, what could be these potential improvements because it’s not just going to be handled on a platter, right? I mean, this is where you got to just play around and think and be creative, right?

Ben: No question. Because the truth about metrics is that they don’t give you the answers. They tell you that there’s a problem and they give you a sense if you can get to the number and the place, keep going. But they don’t give you the answer in terms of what to do. So I just described, you know, three or four things that you might do related to on a phone. And there’s a whole host of other things you could probably do as well. I mean, most Web businesses, they fall down just on the onboarding process alone. Right. So, you know, you sign up for a new service. We all do this. We get dropped into the service. We look around. We’re not sure what we’re supposed to do and just we just walk away. And that’s like one churn, right? One, we’re not even we’re not even at, you know, weekly or monthly churn or six months or a day. So absolute we this is a whiteboard with a whole bunch of ideas and probably a whole bunch of other numbers that you’re looking at. Like, you know, how many people have come in every day and you’re looking at things like customer complaints, you’re looking at all sorts of different numbers to tell you where there may be problems that you fix that all end up resulting in improving. Sure.

Bronson: Yeah. No, it makes a lot of sense. And now the graph kind of splits, author flowchart splits off. You say without data, you make a guess. And with data, you find a commonality. Explain those two options for us.

Ben: So if you’re if you don’t know data about what’s going on, so you know what certain looks like or maybe you haven’t really measured it effectively, you just know it’s bad and you don’t you don’t have any other data. You’re not tracking customer complaints, you’re not tracking churn at different points in time. You’re not tracking the use of certain features. Maybe just things are frustrating people and they’re abandoning the service. So you don’t really know. So you make a guess and you’re making multiple guesses. You’re running multiple experiments. If you have data, then you’re looking for commonality in the data. So this is a scenario where you’re saying, okay, we’ve been measured by the usage of all feature. Which filters do people seem to well, not well or we are tracking customer complaints, we have data and we can look at the data and look for these threads of potential truths within the data. But ultimately, whether you have data or not, you have to do something. And part of it is all best guess, right? Even with data, you’re like, Oh, well, it still doesn’t tell you an answer. It gives you some insight into where there may be problems. And then you have and then you have to start working on actual experiments.

Bronson: Yeah. And so whether you have data or not, it kind of leads you to a hypothesis. So what does that stage.

Ben: Sure. So the hypothesis basically says and really this should be something literally write down somewhere. Okay. Just not write it down so you can look and see if it’s if it’s a good hypothesis, if it’s a falsifiable hypothesis, which basically means it could be proven false. Right? If your hypothesis can only be proven right, then it’s not really going to give you a lot of information. But if your hypothesis is basically if we do X, the result will be Y, right? So if the lower customer complaints by 20% turns out a drop and then look, look a little further, it’s like, okay, how we lower customer complaints? Well, if we hire one more customer support person who can handle more support calls of a day that that we believe that is what do they drop customer complaints which will drop. So it’s really this sort of scientific if then kind of statement that you have to write down and everybody says, okay, we understand that. And for example, let’s say we think hiring somebody that our staff is overburdened or you hire that person and then you have to see what happens and maybe it does work, maybe it doesn’t, so it’s viable.

Bronson: So then after you have a hypothesis, then you have to either design a test or make changes in production. Right. An example of a test and maybe you just did, but give me an example of what a test is or what a change in production would look like.

Ben: Sure. You know, basically what we’re saying is that a change in production basically says you’re just going to do something to your product. Let’s say you might change the onboarding experience. You might change a feature. You might add, like I said, sort of drip marketing mail campaigns to try to reengage users and you just push that live and you just push it live to everybody. A test could be a little bit different. Right. So a test could be maybe we’re doing some of those changes to our product or our technology, but maybe we’re only giving it to a portion of people. Or maybe we’re only giving to new people who sign up. So we’re doing some form of cohort analysis on new users because our product is now changing. We say, let’s say onboarding. You know, we change onboarding to try to engage people a little bit more because we believe that, you know, show them the value more quickly in our product that’s going to result in culture. But of course, you can’t do that for old users. You can only do it for new users. So that’s that’s a little bit more like designing a test. And in some cases, those might be throwaway tablets, right? They may be things that you experiment with. You might turn a feature on for for a certain group. Of people. And then turn it off and start to measure what happens.

Bronson: Okay. So you mentioned measuring what happens and that’s kind of where we go from there. So after you design a test or you made changes in production, you just got to see what the results are, right?

Ben: Yeah, exactly. And remember that. I mean, ideally, you’re running tests that have very short cycles, but that might not always be the case. Right. So some of this stuff may be on going in May. So, you know, you know, churn can happen months and months later. So there may as a as that, that’s a particular job. Like in other apple you may run really quick to change things very quickly like will produce customer complaints right now or let’s change, you know, a feature in the product that’s really frustrating people and see if that keeps people. I will change your onboarding. It might be months before you really know if they’re going to churn out at any different stage. So some of these things can take a while to, you know, to tell you if your test or is working or not working.

Bronson: Yeah. And then after you measure the results, you have to ask, do we move the needle? Is that where you look back at the KPI and see where it was and then where it’s at now and then that’s considered moving the needle or not?

Ben: Absolutely. So that’s where you’re going back and you say, okay, you didn’t move needle. And then you’re sort of asking yourself, I didn’t move it enough or not. Right. I mean, it’s pretty it’s can be somewhat binary, but of course, it’s never perfect. Right. So, you know, if we went from 5% churn to two and a half percent churn, so then you have to ask yourself, well, do we draw a new line in the sand and just say, you know what? Look at this line for now. So, you know, now 2.5 is our new churn. It’s a lot better. It didn’t hit our target, but that’s okay. Do we try again? Maybe we get a three and a half or do we just go through the whole cycle again? Or do you decide that, you know, this is just not working and we have to change something more fundamental in a business which is really we sort of pivot or give just a little bit and Glu maybe.

Bronson: You know when I was reading this part of the flowchart I, I smiled a little bit on my pivot to give up. All right. Well that’s the way I put it.

Ben: Right, exactly. So but it, it’s really just to give that sense that at some, you know, you can go through this and go through this and go through this. You know, by virtue of just doing this, you don’t automatically guarantee success. Right. And it’s similar to lean startup in general, you know, just by virtue of following the methodology. But framework doesn’t mean you automatically win. So there’s a point in time where you saying, you know what, I just figured this out. And maybe to pivot because you’ve learned something about your users and your customers that tells you something that you can do or something you can adapt in your business or your product. Or maybe you just say, You know what, this isn’t for me. I have to go do something else because I cannot I can’t practice. Not so it is a little bit doom and gloom. But you know, if you go through the times and you can’t figure it out, you know, I’m picking random, you know, if you keep going through this and through this and you have to figure out how to get the numbers to get to where you know, you need them to be, then you have to figure something else out because going through the 11th time or a 12th time or 13th time, sort of the definition of said. Right, yeah. Doing the same thing over and over again. Expecting a different result. Exactly. And so it’s really there to say most of the time it’s probably gonna be try again because it’ll probably won’t get to the level of optimization or improvement that you’re aiming for going through this cycle once but twice or three times. The hope is that you would get those diminishing returns we talked about. You know what? That’s my new line of single.

Bronson: Yeah. Now, I love this cycle because it gives some rules and some structure, not rigid rules or rigid structure, but just some loose rules and structure to something that could just be a lot of discussions and a lot of bullet points and random to do list when instead it’s like, okay, let’s run it through the cycle and see what happens. And it just gives ordered a little bit of chaos. And I think that companies that do this with a lot of their KPIs, they’re really going to have measurable results on a. All over, which is so important.

Ben: Absolutely. So, I mean, the whole book is is prescriptive, but not to the point of saying, if you follow these steps, I guarantee you you’re going to win. It’s not like a diet book, right. That says, you know, do these ten things I guarantee for your life, your waste. We can’t do that. Startups are messy, right? The data is too complicated. Everything. But there are many variables. But. But the goal is to give. I think one of the areas where startups fail so often is around the focus because it’s a crazy business to be in. And so the hope of something like this is to say, I don’t have all the answers. But I do know that if you can focus your whole team on this process, if you can focus on churn for a few weeks and everybody is focused on it, I’m pretty certain you’re going to push the needle significantly. Or are you going to or are you going to learn how to know what else you how to change it or where you might have to pareto as opposed to everybody just sort of running around wildly trying to try to solve problems.

Bronson: Which is usually what a startup is.

Ben: Exactly. And that’s and that’s exactly the challenge. You. Right. Is is the focus is so important. You know, the lean analytic cycle in the book is really about I mean, the whole thing is really about focus. Mm hmm. But you can’t tell someone because that doesn’t, you know, that doesn’t. What does that actually mean? So for us, it’s really about focus by doing this and follow this process. And hopefully by doing that, I’m going to learn a lot about your business and also learn how to focus.

Bronson: Yeah, no, I think it’s great. I think the way you described is perfect. That’s exactly what I feel looking at is like I have a playbook now that helps me focus and make sense of this. Now, another thing I found online that you’ve talked about before, which I thought was really interesting because it gives such a high level view of analytics, is the trends in analytics kind of where we’re moving from and what we’re moving to. So let’s kind of run down this list a little bit and tell us about some of the trends in analytics. The first one you mentioned is that we’re going from aggregate to individual. What’s that? Right.

Ben: So again, basically, it means that we’re now capable of the analytical tools are getting better all the time. Right. And I mean, it’s a quite a and the reality is that we can basically track everything at this point. Anything imaginable is trackable. And if a tool doesn’t exist that allows you to sort of grab their tool and use it, you sort of you can instrument the stuff on your own pretty easily. And so so I think where we’re getting is the tracking of individual users or you know, in this particular case, I think of it mostly as customers. But we can we can track an individual now through a funnel, through a product. And it’s interesting, I think through a product or even an e-commerce site, the person comes back, they buy this, then they come back. Six months later they buy this. So instead of just looking at these high level aggregate numbers, we can now look at individuals and track them, but also group.

Ben: Them and do different, interesting things with them, which I think is when we start to think about how we segment different groups of people.

Bronson: Yeah. And then the next kind of trend is segments two cohorts should explain to us. We’ve heard before in the show, but people need reminders. What’s the difference in a segment cohort and why the trend going from one to the other?

Ben: Sure. So a segment is really just a users with some common characteristic in your among your users in your database. Right. Which can definitely be interesting. And they’re looking for what different kinds of people are doing. So, you know, a good example of that. Is when Mike Greenfield from Circle of Friends had this product back in 27, 2008, which was basically Google circles for Facebook. And here, about 10 million users, they had a huge number of users. But but engagement was terrible. And so what he found was I found a segment inside of that, which was more homes. And for them, engagement was off the charts. So that’s an example of a segment, right? Looking at you’re looking at all your users and say, hey, all of these users have this common characteristic. Mm hmm. They’re doing something different in a different group, which still remains interesting. The cohorts are another way of looking at the data, because there’s basically different groups of people, irrespective of their characteristics, but over time. And so most of the time we have cohorts, just as I described it, like, let’s say my onboarding flow. I bring a new group of people in to the top of this funnel. When they come to my website, they sign up. I’m sort of measuring the funnel that way. Mm hmm. Because I’ve changed the onboarding experience. Let’s see what this new cohort does. Okay. So as you make changes to your product or your website, your marketing, messaging, or any of these things, often it affects new users more than affects old user. So it’s really a grouping into cohorts or conference over time and then measuring differences over time.

Bronson: Yeah. Is cohorts always about time? Is it possible to a cohort analysis that’s not based on time?

Ben: I think it’s mostly based on time and it’s the best way of thinking because of most of the changes. It will often only affect new users comment. And so it’s really it’s a way of saying, you know, I have an e-commerce site, for example, and, you know, or, you know, here’s my homepage to my website and I make a fundamental change on my website and I will see if more people sign up by virtue of the changes I’ve made to that homepage of my site. That’s really a new cohort of users to drive those users to the sites. For example, conversion improves as a result of something that I’ve changed, which won’t impact all users. So so it’s really about it’s really about comparing groups of people over time. And of course, the whole being that, you know, even if the users that first came in start to dwindle off, which is inevitable, right? Every business has churn. We have this concept of lifetime value, which means everybody eventually leave. But if the new cohort is more engaged, more active, buying more, doing more of will actually want because of improvements there. And the old cohorts, they’re important, but they become less important because, you know, they’re genuinely improving by looking at specific groups of people. Yeah.

Bronson: So you should be bummed out about churn rate that’s higher with an earlier cohort. If you’re improving it, you’re on the right track. But if you looked at them as a segment or as a whole, the churn rate would be off. It wouldn’t show you what the numbers really should show you.

Ben: That’s right. So you have to be careful about averages, right? Because. Well, do you have a year’s worth of data of new users coming to your website? And conversion is, you know, whatever it is, 10% of people who come to your site convert and become users. And maybe, you know, that average, maybe right now it’s 35% and they used to be much, much lower before. So if you looked at the average, you might say, oh, I need to work on that KPI, I need to improve that drastically. But if you look at the last month of new users that maybe it’s really high and that’s being consistent, you don’t have to look at that number. That’s why cohorts more.

Bronson: So that’s the trend. You know the trend toward cohorts allows us to focus on what actually matters and get better insight from the same data.

Ben: That’s right. Exactly.

Bronson: Talk to me about generic to vertical. What do you mean by that is one of the trends in analytics?

Ben: Sure. So so now we’re seeing more. So, I mean, there are some tools that can be used for almost every for every business, like Google Analytics being the main one. But we’re seeing more vertical services coming out. So that angle as an example, just focused on SAS businesses or again. Tango. Tango dot.com flurry is another example, writes Flurry is basically for mobile analytics because we you know, you see different. I mean, this is another one of the trends. But we see differences between desktop users and tablet users and mobile users and need analytical tools designed to help us, you know, throughout different behaviors. Yeah. And so there’s different ones for different verticals. Parsley is one for publishers. And so you’re starting to see these vertical tools emerge that are specialized for particular types of businesses. Yeah.

Bronson: Do you think they do better when they’re specialized? Like to tango, for instance? You know, is it going to do better than you think a mixed panel would? I mean, what’s your thoughts on that’s the trend that’s going toward is it a good trend?

Ben: Well, I think it’s I think, you know, I think it’s a I think it is a good trend. I think with analytics, it’s always going to be a multi tool kind of process, right? I don’t know. There’s one tool that rules them all here, unfortunately. So, you know, they’re running a business that you’re using to tango for specific things but are still using other tools. You’re trying to just sort of piece all of these, you know, these parts of the puzzle together. So I think it is good because it adds that specialization that you are. But it will result of that. It may add more. Some other things that you’re also interested in.

Bronson: Yeah, no, that makes sense. Now, you also say we’re moving from silos to aggregation. Talking about that, I don’t know. I don’t know what you mean by that one.

Ben: Sure. So, you know, we’re seeing tools. So as we’re seeing more tools come out, we’re also seeing things that help us aggregate all of these things together. And that’s really what we’re talking about. And also, you know, also the idea that, you know, we have tools to do specific things like, you know, the marketing guys are looking at specific things and the ops guys are looking at performance. You know, let’s just use that as a double marketing tool, looking at things like conversion and traffic and just looking at things like performance. And then we realize, oh, you know, we improve performance. There’s a huge effect on conversion, right? So now we have to bring those things together. And so, you know, I see a trend towards looking at analytics holistically, but also tools that help us say, did you know performance effects for uptick in conversion? So marketing, maybe they don’t care about like don’t care about performance, they don’t know how to track it. That’s double ops thing. Yeah, but these guys have talked together and work together. I think right now.

Bronson: That’s great. You also say we’re moving from daily to real time. This one’s probably self-explanatory, but go ahead and give us a quick anecdote there.

Ben: Well, I mean, you know, we’re able to track more stuff. More and more people are looking at real time. And I think Alacer actually has a good way of saying, you know, don’t look at real time. Look at right time.

Bronson: And what does it mean by that?

Ben: Well, particularly that, you know, just because we can track things in real time, second by second doesn’t, we should. Right. It might be interesting to watch the graphs and moving and all of these things changing. It’s fun, but but it still comes down to this core principle sort of of Olympics, which is it’s not changing behavior. If you’re tracking something that’s not fundamentally helping you change a behavior, then it’s not worth tracking. Right. So it might be interesting. It might be fun to look at. When you look at the website, of course, you’re going to pay attention. But sort of the website crashed and the real time data is kind of meaningful. Right.

Bronson: So there are.

Ben: There are points for real time. And then really, it’s about right time is for your particular business. What’s sort of the appropriate time interval for the specific metrics that matter to you? Maybe it is real time, but it probably isn’t. Yeah.

Bronson: Now you also say we’re moving from reports to exceptions. What does that mean?

Ben: So it means that so we can report everything and we can look at reports. But I think what’s most interesting about reports that we see are exceptions. Okay. So, you know, a good exposé like a spike in traffic. Mm hmm. Right. So I’m looking at a lot of data on my blog for a simple. Simple example. And then I see on one particular day a huge spike. And that’s kind of an exception to the rule. It’s like, why did dropping ten X on that day? And now I can dig into that and figure it out because. Some famous guys tweet or it might be because I wrote a blog post somewhere or I did an interview or who knows what, right? So it’s really about looking at those anomalies. Now, there’s always a risk of looking at those because it might be, why did we have no traffic that day? Oh, well, it’s because our survivors were down. Right. So you always have to understand what was going on. But it’s interesting to look at the exceptions as opposed to just rules all the time.

Bronson: Yeah, because they might give you some action items of, hey, it’s kind of like give an indicator, hey, there’s something going on here, right? In deeper, you may learn something to recreate this. It might be a one time thing. It may not be a one time thing, but this is telling you where to look, at least to figure it out.

Ben: Exactly.

Bronson: Yup. Now you also say we’re moving from pages to events. I think a lot of people watching this probably understand this one as well. But go ahead and give us the quick take on there.

Ben: Right. So, I mean, you know, in a media business. Because we still care about inventory and those sorts of things. But events are really about actions we want people to take. And that’s really what we’re talking about is a trend to work. Yes, it’s interesting that the guys on this page, but really what I want to do is click this button. Yeah. And it’s attractive that as opposed to just tracking a view, the page is really where things are going.

Bronson: Yeah. Now, the next one, I want to hear what you have to say about because you say we’re moving from funnels to influences. And in my mind, you know, funnels are clean. I mean, they’re so important. How could there possibly be a trend away from that at all? So what do you mean by funnels to influencers?

Ben: I think it’s really just the fact that funnels are getting more complicated and which is sort of in some respects is unfortunate, although it’s an opportunity to expand and to reach to improve the funnel so funnels don’t blow away. It’s really just about the fact that, you know, we get influenced by more things in the social web, right? So somebody comes through, let’s say through Twitter, but really they’re actually coming through Facebook where they saw something that was tweeted into Facebook and they go from. So it’s really about the fact that we might see something three or four different times. Before we engage. And if we are a funnel, we might say, oh, it’s all Twitter. It’s all Twitter for social traffic. But that might not really be true. There might be a lot of things going on there that are the influences of people who were then driven to Twitter. So that’s really, really just sort of unfortunate to some degree complexities of this web playing a role now.

Bronson: Yeah, so part of analytics is understanding. You don’t have all the analytics.

Ben: Right? Well, absolutely. So as much as we can track everything, we haven’t tracked everything. We’re not tracking and we don’t understand everything. So we’re still making educated guesses from data, but also from what we see happening in the real world. But you can’t just stare at a screen of numbers and get all the answers. You have to engage in what’s actually happening with your product, with your marketing, your customers to really get these things out.

Bronson: And you say we’re also moving from desktop to mobile. And, you know, we all know that. I mean, we’re all using our phones more than we used to. So what does that mean for analytics? Tell me what that actually means from a lean analytics point of view.

Ben: Yeah, I mean, I think it just again, it just keeps things right. So sort of.

Bronson: That’s what it means. This is all complicated.

Ben: It’s all getting complicated. It’s sort of unfortunate. Reality is and one of the guys we interviewed for the book, he actually said, you know what, you can’t look at mobile app. You have to separate accounts and phones because behavior is actually different on a tablet than a phone. So it’s yeah, it’s not just desktop and mobile, it’s desktop tablet phone. And, you know, ten years from now or two years from now, who knows what else it will be? So people’s behaviors are different or intense or different. The way they interact with your products are different. So all of that has been reflected in the things that you’re tracking as well as the things that you to focus on a given point in time.

Bronson: Yeah. And then this last term, I think is one of the more exciting ones going from accounting to predictive. Tell me about that a little bit.

Ben: Yeah. So I mean, accounting is very simply just reporting all the things, right, track all the numbers, report on them, spit them out, let’s look at the reports and figure out what we’re going to do with them. I think predictive analytics is about trying to, you know, as well as predict what’s going to happen in the future, how can we predict behavior more effectively? And so, you know, Alister talked of correlation and causation, some elements of that. I think, you know, I’m glad I sort of start talking about. Sure, because a lot of companies now they focus on how to predict churn out of their product or service and if you can get into the open, hopefully save the day. So I think that’s really where analytics is moving as a force as opposed to just reporting the facts, if you will, and then trying to figure out what those things mean. It really about, well, what’s my goals? You know, I need to know what’s going to happen and how I can affect that behavior down the road.

Bronson: Yeah, it’s hard to take action on accounting. It’s easy to take action on a predictive kind of thing, like, Hey, this is what’s going to happen if you don’t change course here. Now I have something to do. So it just helps us lead a startup. Yeah. I mean, I see this. In your opinion, how many startups get lean analytics? Because I mean, you interact with them, you know, you’re on labs. You know, obviously as the author of this book, you speak to many startups. I mean, is it like just a small percentage of the top they really get it or is it kind of mainstream now? I mean, what’s your take on it?

Ben: Well, I think Lean Startup is pretty mainstream, but I think it’s it’s both. So people pay a lot of lip service to. Right. So it’s like, you know, I don’t know how many times companies come to me. They’re like, I’m a lean startup and I’m like, okay, I don’t know what you want me to say about that. Like, it’s not a guarantee of success or that’s wonderful. You know what I mean? Absolutely. Yeah. So tell me what you’re actually doing here that you think is going to make a difference. Show me have startup. Exactly. So I think everybody understands they need to track stuff. Right. That’s not that’s not a big surprise. Everybody knows they need to track. I think that’s where most startups fail, not the startup, but where they fail in terms of analytics is tracking the right. Right on. Yeah. So we’re prone to vanity metrics. Were prone to tracking everything and just trying to look at all the numbers and trying to interpret all of this data. We’re prone to a lot of mistakes around then. And I think sort of similar to Lean Startup, really. It’s Oh, it doesn’t work. You know, it’s like I’m tracking everything and it’s not helping me. So I do not track stuff anymore. You know, we sort of have that response to it if it doesn’t give us an immediate value, but it only gives us if we’re able to focus on the right things at the right time. And so you invest in figuring that out. We can really get the value.

Bronson: Yeah. Now, let me give you a chance to plug your book for a second here. If somebody really gets your book, digs into it, takes it to heart, implements it, it just to me, looking at how much is in the book and the kinds of things you guys cover, it looks like they would have a massive competitive advantage if they were actually not just giving lip service to it, but implementing it. Do you agree? I mean, do you think it could have that effect if they really put it into play?

Ben: So because that would be that would be fantastic. You know, we’ve already heard from people that it is genuinely helping them. And I mean, that’s a big reason to do something like this. I know, Alison, I are always trying to help startups as much as we can and do a lot of mentoring and whatever it is that I’m writing and so forth. So it really does help. Like, that’s fantastic. I do think there is a competitive advantage in tracking the right time in these processes. I do believe there’s a competitive advantage in all of a lean startup because the goal of that is to get any insights faster. Right. And if you can do that, that’s your advantage. So what people see from the outside looking in and this is this is always this always happens, we look at a competitor and we just assume they know what they’re doing. Right. Particularly particularly bigger ones. Right. Look at a bigger thing. We’re like, oh, well, clearly they know what they’re doing, so we’re just going to copy them. Mm hmm. And but you actually have the insights, and you’re able to adapt your business. That’s really the competitive advantage. And it won’t be obvious to anybody else because they didn’t do the homework. Mm hmm. And so. So, yeah, I do think if you can follow this kind of process rigorously, adapt for your specific needs and. Learn from it, and focus on that learning component as quickly as you possibly can, that is going to be your competitive advantage. Yeah.

Bronson: Now, you know, putting yourself in the shoes of a startup, listening to this. Right. And let’s say they’re new to Lean Analytics. They’ve heard us talking about the analytics cycle. They’ve heard us talk about the trends. It’s a lot to take in. I mean, it’s like, okay, there’s a lot at play here. Where do they begin? They haven’t really, you know, dived into this world previous to this. What they want to look like when you’re trying to get your head around lean analytics and implement it.

Ben: Sure. So I think I think day one for me is you need you need systems in place that track the data number one. Right. So we would never say only track one thing even though we push this concept of the one metric that matters focusing. But I think you need to just have systems in place very early for tracking because you’ll never know when you’re going to able to go back to that data and get value out of it. So that’s that’s one. But I think also you have to look very quickly what makes a good number. So actually, it’s funny because we’re not funny, but we talk to a lot of people about, you know, what is a good metric to begin with? What are the characteristics of a good metric? So things like it’s not an absolute number. Number of users. So this is again, sort of the vanity metrics. But, you know, even number of users. Per week is not a great metric. Percent user growth is a much more interesting metric. So just understanding the fundamentals of analytics, what makes a good number versus a bad number to track is a good place to start. And so again, looking at these analytics and basically just say, what number am I track that will change how I behave? So, so that’s, you know, those basic fundamentals is where I would start and then then I would look at. So in the book we try to cover this, but I think it’s going to vary for a lot of people. We sort of cover these business models and we have these like, here’s what.

Ben: Analytic you should track based on the age that you’re at, which is.

Bronson: Super helpful, by the way.

Ben: Right. So I think you can use that concept as a as a starting point. So I understand what a good number, as I understand, is supposed to change how they behave. I’m tracking a whole bunch of stuff. Now I just want to map out a basic flow through my product, through my.

Ben: Experience, through my user acquisition. Mm hmm. So it is analytics, but with it, as I see it for my business and my business model. And then just starting to map that metric after metric after metric.

Bronson: Yeah. Now, that’s great advice. And I love what you said about knowing what is a good number. It’s so fundamental and so important. Now, let me ask you also about go instant. You’re currently the VP of product there, is that correct?

Ben: Yes. Yeah.

Bronson: So tell us real quick what is go instant, first of all.

Ben: Sure. So to instant Bill. So first I should say go. It was acquired last year by Salesforce. Yeah.

Bronson: Big win for you guys. Yeah.

Ben: Thank you. So. The builds are what we call cobras in technology. So we’re very much a technology company, and COBRA technology is to allow two or more people, but typically sales to people to browse the web together without any downloads or plug ins like you would typically have with a screen sharing application. So for us, most of our use cases are around things like complex online transactions.

Bronson: High dealer sales, kind of things like.

Ben: High value, high touch transactions is really where, you know, typically an agent or a salesperson would want to basically walk you through the experience as if you were sitting right next to them without asking you to download flashcards or anything. Just very, very quickly fully integrated into their whatever their web application or web experience might look like.

Bronson: Yeah, that’s great. Now, as the as the VP of product, let me ask you this. How important is a VP of product to a company’s growth? I know I have my own opinion on it. Would you say.

Ben: Well, so I think somebody needs to own the product in a very, very small startup that in my mind is the CEO, maybe the CTO, maybe you don’t even have a label some and it’s the founders. Right. So let’s let’s throw that see in the season the visa and say five.

Bronson: People it doesn’t really matter how you give yourself.

Ben: But somebody needs to own product, right? So I believe in that and the product vision and product roadmap and vision to the nitty gritty of how should this thing work and look and feel and talk to customers and all of that. So, you know, whether that’s the founders or as the company grows, oftentimes the founders have other responsibilities, whether it’s just operations and running the company, whether it’s, you know, going and raising follow on financing and investor relationships, you know, CEOs in particular or the business founder, let’s say.

Ben: Has all these sort of other responsibilities. And so, you know, in our case with go instant, you know, when I joined, it was very early, but a lot of stuff went on and I was brought in to say, let’s just focus. On the product here and have somebody who owns that responsibility.

Bronson: Yeah, and I think the VP of product is, you know, an amazingly important asset to a startup for all the reasons you just said. Now, let me ask you this. You know, you’ve written the book on Lean Analytics. Does Lean Analytics inform your thoughts and the way you do things as the VP of product? Because at first you may think, okay, Lean Analytics is where the growth engineers, maybe the VP of marketing, does it actually get down to the VP of product level?

Ben: Sure, absolutely. So I think there’s a. Whole. Category of discussion around analytics and product, right, and product development. And we cover some of the it’s actually in the book in terms of how to decide what feature to build and so forth. So, so, absolutely. So a lot of what I did very early on was implement our own analytics system for tracking user behavior when inside of go instance, what we’re doing, you know, some performance data, some error or data, all of that sort of, you know, is the product doing what it’s supposed to be and working well. And then we use analytics for feature development. So when we release a feature, we track its usage and that informs us as to. A lot of things, you know, is it working well the way we expected? If people say yes, it is working well, but they still don’t use it, and you can say maybe that feature just doesn’t belong there. Belong there where we actually we have in the past remove features. Yeah. So, so absolutely the sort of concept of taking qualitative customer feedback and quantitative feedback and use that to make better decisions about what to build in the product or how to change the product or what to take out of the product, I think is fundamental to good product management.

Bronson: No, that’s great. I mean, it seems like you’re actually taking on some of the roles of a growth engineer. You know, normally they’d be the ones kind of implementing the metrics inside the products. But if you have a VP of product doing that and that’s a big win, you know, if you can really focus to that level on the product itself been this has been an awesome interview. I have a couple of final questions here because I have to go, but that’s okay. Just kind of high level questions. You can take them sort of wherever you want. And you one labs. What advice do you find yourself giving to startups about growth kind of over and over? What do you find yourself just saying like a broken record that you just wish they knew already.

Ben: Right? Yeah. No, fair enough. And, you know, it’s it’s really interesting because I think fundamentally, no matter how many times you tell people things, I think the truth is that they have to experience it first.

Bronson: I know that’s a sad thing, right?

Ben: I think that’s just the reality of it. And I just you know, I catch myself talking to entrepreneurs and founders and I’ll be saying something and I’m like, I’m pretty sure I’ve told people this. And then I’ll just say flat out, I’m pretty sure you’re not going to listen. So figure it out. And, you know, not that I’m right all the time, but I think because, you know, probably you have to just experience it and be like, oh, yeah, that’s that’s what happened. But there’s a little light bulb that goes on faster. So I think fundamentally growth only comes when you have right engagement and retention first.

Bronson: So I think over time, actually, we hear it.

Ben: Yes, I think grow. Only, you know, only comes you can only focus on growth when you have the right amount of engagement and retention. So I think it’s you know, you can sort of go viral, premature and try to attempt for user growth prematurely and systems that allow you to do that. You know, and there’s ways of sort of hacking that, if you will, and then realizing that you don’t have the fundamentals there, you know, the fundamental mechanics of your business, not just about revenue, but just about using it in patterns of what people are doing is not there. And as a result, all that growth sort of tumbles. Churn is huge, engagement is low, and you can’t possibly put enough people at the top of the funnel to build a sustainable business. So I think that’s one of the biggest ones. I think it also you have to focus on a couple of things. Putting on boarding I’ve talked to already, I think a lot of companies ignore that onboarding experience and then they ignore the concept of delighting people. So I think it’s more like I think this point, you can’t really give the feedback and say this is how you delight people. Because I don’t know for your particular business what the answer is to that, but I know that or I really believe that if you delight people even in the most sort of boring of circumstances, in the most enterprise of a price off or whatever it is, you will gain that engagement and the retention. And that’s the springboard for growth because that’s where word of mouth spreads. Now you can build into your product fundamentally as opposed to trying to like stack it on pieces, you know? Yeah. But it only comes that base of delight and engagement where you can really build a growth. And then I guess the last thing I would say is that, you know, press as a mechanism of growth is pretty much a fallacy in most cases. Right. These are not fast rules. But, you know, most early stage starts, their first thing is I need to get tech press and I’m like. Okay, go ahead. And unrepeatable though it’s all repeatable and worse is it gives you this false sense of hope. Yeah, right. Because you don’t know whether it’s repeatable. Like you sort of know it’s not repeatable, but you figure if I can get into TechCrunch, I can get into TechCrunch 1000 times. I’m not going to just using it as an example, but it gives that false sense of hope because you see all this engagement pop in, you know, my big launch and then it all sort of tumbles away and you.

Bronson: Have all the trials. Sorry. Right.

Ben: Yeah. Yeah, absolutely. Absolutely. And so if you if you don’t, that’s just as a mechanism is not interesting to me. You have to figure out what. Are the behaviors that you want inside your product that are going to drive genuine growth. And if you can figure out what those behaviors are, you’re just it’s not going no matter what you do at the top, the marketing or anything else that you do, the press, it’s just not going to work because the truth will be revealed in terms of you have to change people’s behaviors effectively enough into the products.

Bronson: Spoken as a true VP of product.

Ben: Yeah, I focus on product, you know. That’s right. I think. Last question.

Bronson: What’s one of the best lessons you learned about growth that one of your own companies.

Ben: Sure. So I already before. But I think copying other people is is not bad in itself. Right. So lots of people we do it in products all the time. Right. We look at the way somebody has invited a friend to, you know, something, let’s say and we say, well, that’s you know, if Facebook does it that way, I’m going to do it that way because Facebook knows what they’re doing. Right. So that’s true. But it can only get you so far. Yeah. And I think from a growth think you can’t just assume what works for somebody else will work for your business or your product. And you also can again, you can’t entirely assume that they know what they’re doing. Yeah, in some cases, yeah. But in some cases I think it’s a fallacy to just be like, well, if that’s the new way that we’re doing, you guys on a mobile, everybody has a hamburger. Yeah, right. If you know everybody, the three lines of top that everybody moves to the hamburger. Yeah. I don’t know if that hamburger works. Yeah. So you have to measure that yourself so you can you can copy, but. But only so far. Right. And I think things like, you know, the launch pages that came out that’s a couple of years old now, right? Like, you know, sign up for my thing, tell five friends and get a huge list. You know, you can copy that. And it kind of works, but it might not be the best thing for business. There may be other things that you need to do. So I think that to me is a key copy. But use the data and use your own brains and talk to your customers about what and your users about what’s going on. But don’t just assume it’s going to automatically work.

Bronson: Yeah, that’s great advice. Ben, thank you so much for taking time out of your busy schedule to join us on the program. Yeah, thanks again.

Ben: All right. Thank you.