Tracy has been pivotal in growing Pinterest and she talks to us about their user-centered culture, the role of technology in a startup, when to focus on infrastructure, and what she’s learned about growth.

TOPIC TRACY CHOU COVERS

→ Description of Pinterest as a platform to help people discover things they love and inspire them to act on it in the real world

→ The meteoric growth of Pinterest to 50+ million active users

→ Importance of creating a product that resonates with users and iterating on it to make it valuable

→ Technical considerations, such as code quality and architecture, in scaling a product

→ Importance of data in understanding user behavior and making informed decisions on product development

→ Challenges in maintaining user privacy while also utilizing data to improve the product

→ Future plans for Pinterest, include expanding internationally and exploring new verticals.

→ And a whole lot more

LINKS & RESOURCES

→ Tracy’s Quora Profile

→ Her LinkedIn Profile

WATCH THE INTERVIEW

READ THE TRANSCRIPTION

Bronson: Welcome to another episode of Growth Hacker TV and Bronson Taylor. And today I have Tracy Chao with us. Tracy, thanks for coming on the show.

Tracy: Hey, thanks for having.

Bronson: Me. Absolutely. Now, Tracy, you’ve been a software engineer at Pinterest for over two years, which basically means you’ve had front row seats to kind of watch the rocket take off, right?

Tracy: Yeah. It’s been a pretty fun ride.

Bronson: Yeah, absolutely. I’m sure it has. We’re going to get into kind of what that ride is has been all about. But first, tell us a little about Pinterest. And I know that by this point most people have used it. But in your own words, what’s the vision of Pinterest today? What are you guys trying to accomplish?

Tracy: Right. So our mission is really to help people live inspired lives. And we want to help people discover things that they love and then help them go out into the real world and do those things. So for my personal use case, I really like reading, I really like running, I like travel. And then just it’s really great for me to collect inspiration and then actually go act on that. So if we’re running a mike, you know, look at running routes on Pinterest and then try to go out and do them for travel by collecting places I want to go to. These are the sorts of things that we want to help people like find what inspires them and then enable them to go do those things.

Bronson: Yeah. All right. I love your answer because you’re focusing on the benefits, not the features. And that’s one thing that so many companies mess up on. You didn’t say, oh, we let you pin stuff. And it’s kind of this wall of things and you can scroll through it all day. That’s a boring answer and it’s the wrong answer. Yeah, you do all those things, but it’s the inspirational stuff. It’s the what are we going to do in our life because of it? And so already you’re an engineer, but you’re speaking like someone who has a great sense of like brand and what that means. So I just want to let the people listener answer, because even though she knows a lot of engineering, it wasn’t an engineering answer and it was the right answer. Now, over the past few years, you guys have grown to 50 million plus active users. What do you think is most responsible for that kind of meteoric growth? When you look out from the inside.

Tracy: I mean, there’s a lot of I think some of it is just that we have built a product that we think resonates with a lot of users and is really valuable to a lot of people. There’s a lot of hard work and, you know, executing on the vision. It’s really important that all the technical pieces are there, but those are kind of just like the fundamentals. And then a lot of it’s just like iterating on a product and trying to understand what’s useful to people, whether it’s getting partnerships with others like, you know, big brands that are opinion content that’s really valuable to our Pinners or if it’s building more useful features like making rich pins or actionable. So we have like recipe pins we can do like look at the metadata, like the ingredients you need cooking instructions. Like these are things that make them more useful. We can work on discovery to connect people to content. There’s a lot of different things in the product, so I guess the answer is really it’s a lot of hard work and trying to make something that people find valuable.

Bronson: Yeah, that’s great. And you mentioned kind of two categories of things there. There’s the creating something that people love and there’s also the technical behind the scene scaling to make sure you can deliver that thing that they love, which comes first. Should you worry about the technical stuff until somebody loves it or should you worry about the technical stuff from day one? What do you think?

Tracy: You should worry about the product first? It doesn’t really matter if you have great code quality and a great architecture for your systems that nobody is using your product. For us at Pinterest, we start off with a really basic Django app that was just not very intelligent at all. Pretty hacky, but it was enough to get us to consolidate the hypotheses around the product. Like, Well, people find this valuable. Will they engage with this sort of an interface and use it the way we think would be valuable? Mm hmm. And over time, we’ve been working on improving all the technical stuff. So one of the first products I worked on here was a big code base refactor. Okay, we need to start onboarding more engineers, and we need to build out the technical side to support a larger engineering team. So some of that is decoupling different concerns of different services, designing better abstractions and airbases. And at that point, you know, when you have a product that people are using you, you to scale, it’s worth investing a lot more in the engineering side. You don’t want your service to be going down all the time, and you don’t want bugs to produce any type of new engineer joints. But it is very important that you have something that people care about using in a very.

Bronson: Well, let me ask you about that, because you mentioned it’s important that doesn’t go down all the time. And I think about another company that just went public and their service went down all the time when they were kind of on the way figuring out, you know, how to scale. Pinterest doesn’t seem to have those same problems. What did you guys do differently? What did you all do that Twitter didn’t? Because you guys are actually using a lot more rich media, a lot more photos. They were using text and they couldn’t keep it up. So what are you guys doing?

Tracy: Yeah. I mean, there’s a lot of things like early on we did go down a reasonable amount, but getting the right general systems architecture in place is super important. If you think of the analogy of building a big house or an apartment building, you want to get the supports in there and you want to design like the overall structure to be right. If you’re building with sticks and like duct taping things together, things keep falling over no matter what it is that you’re trying to serve. So I think in terms of the type of content that we’re serving, it’s not really so significant. But making sure that we have the basic structure in place, like we had to split our database system into a different service pretty early on just to be able to support that, that scale of like coops. And that was pretty critical for us because we’re serving a lot of data. So any time you look at a few pages, a lot of pins on there, we have to fetch a lot of data from the databases or put caching in front of that to make sure that’s, you know, efficient. Mm hmm. That sort of stuff had to be split out into a very solid service early on. The good thing about separating that out into its own service is that you don’t have a lot of people accidentally screwing it up. So it’s I didn’t do its own thing. You define interfaces very well and you can support those use cases pretty well. So those were one of the first big architectural things we did was just to move our databases into this device for service. And then as time has gone on, we’ve been splitting our things into different services. And so those have different definitions around like what kind of uptime and how we deal with failures. So for some different services, like if they start returning data, then we just don’t show the data is are critical. So we have like very severe judging on those. Like if there’s any sort of latency or sort of big availability issues, we can start responses very quickly. But often it’s just making sure that your overall system as well designed.

Bronson: Yeah, it almost seems like a catch 22 here. And you talk about it because at first you have to create something people love. So you’re not supposed to spend all this time on the infrastructure. So in a sense, you’re using duct tape or using sticks, but then as it grows, that’s not going to work. So you have to kind of rethink the foundation at some point. Is that accurate? You know, you have to start with what you have, but you will have to rebuild the foundation if it works.

Tracy: Right? Yeah, that’s actually probably the most viable strategy for a lot of startups is to build something that is kind of duct taped together and then put it over to something more robust.

Bronson: Yeah, well, how many times have you guys had to go back to the foundation? Is that just like once, or is it every six months you’re going back to the foundation and rethinking things?

Tracy: We don’t do it that often. I feel like it was around the time that I joined, but two years ago we did a really big migration. But over time, since we got a basic system in place, it’s been easier to swap out smaller components. Mm hmm. So it hasn’t been as much of a big architectural overhaul. So we had to do that one time.

Bronson: Yeah, that makes sense. Now, tell me if I’m wrong, but half of Pinterest is made up of engineers. Is that right?

Tracy: That’s correct.

Bronson: How many people is that? What’s the headcount right now?

Tracy: The total company is a little bit over 200, so engineering global over 100.

Bronson: So help me understand what 100 engineers are doing on Pinterest, because I look at Pinterest and I can see a way for there to be a lot of engineers on it. But a hundred engineers. What are they doing there?

Tracy: 100 is not even close to it is nowhere close to that. So there’s so many things that we’re working on in terms of like the backend scaling and what we always think about in terms of scaling infrastructure and ops. So infrastructure team is making sure that everything is running smoothly, that we’re able to support all the different data queries and ready to, to, to use for the site ops, making sure that our servers are running well. We have good systems for deploying, keeping things up and monitoring alerts. So there’s a lot on that. You’ll run into a lot of issues as you scale, you just run into new bottlenecks. So just to give you one example, for a while we had our database systems and we had some caches, and then we had some web servers and API servers and to boot to relieve the load on the database, we have all the caching in place, Mantashe and all the app and API server which is directly talked to the cache servers. So that was pretty good for a while. It needed to support more traffic. We just add more web servers and API servers and at some point we maxed out on the number of connections that the Manpower servers could get out. So we can no longer add more Web and API servers because it couldn’t talk to the cache servers anymore. So that’s a problem when you have say like these iconic servers and cache storage of and times and connections and that’s the number them.

Bronson: Exponential at that point.

Tracy: So then we had to figure out what’s a more scalable strategy. So putting a reverse proxy in the middle where it would be a number of servers in the middle. So you have your app and API servers are talking to P of these like more netcracker servers in the middle and those. Harper’s Rivers talked to the end. First. So you have to sometimes really do a bunch of the stuff just using fundamental constraints. Yeah. So some things like in terms of serving more traffic where you work on, which is not great for Multi-Threading. Well Turner lock in most implementations of Python. So figure out how to get around that. Like how do we improve performance while not disrupting the developers too much? Like we could just say we’re going to switch to a different language like Java, which is going to be a lot better for concurrency, but that means that it’s pretty disruptive to all and we have to rewrite everything.

Bronson: Huge.

Tracy: Big. So how do we get better gains in terms of concurrency while not completely disrupting the developer flow? So we started adding events, which is a more it’s kind of a threading, so it’s co-operative instead of preemptive. So there’s a lot of places we had to go in and patch the code to make sure that it would work that way. And we had to, for example, rewrite our then cache client because before it had C bindings and you can’t monkey patch using C, so we had to write a pure python, our client. So there’s some things we have to do there. But you know, it’s just the different challenges you run up against as you try and make sure your costs are too high.

Bronson: Mm hmm. Okay. You’ve convinced me. I can see why you need 100 engineers. Now, you throw out enough keywords and enough jargon.

Tracy: I’m convinced that’s only one part of it. So that’s like the infrastructure side of it. We also have the platform team, so like Web, Android, iOS, we have to build interfaces of the user. C And so there’s pretty high surface area in terms of the product that we’re exposed to. Users like, like what are some good discovery points? Like how do we make each of these parts more useful? We recently launched Place Pins in play sports. So you have like a new map overlay for your boards. Like there’s a lot of companies that have it on the client. We have like a, a large name for like data and data infrastructure to make sure we understand what’s going on. So when you’re a smaller product, you can just kind of use your intuition. You don’t have enough data to do any sort of interesting data analysis anyways. When you get scale, your intuition may not be very accurate anymore. If you’re serving a population is very different than yourself or just like at scale, things happen that you don’t necessarily understand. So we need good data to understand. Like when we launch new features, what is what is the effect on our user base? Interesting analysis on things like when we have people who use the mobile apps versus the website habits or how does their usage vary? So you might think that the if you use the website and use mobile apps like your ID, your time on them is just like the average of the web users, the average of mobile users time. It turns out that when people use both the web and mobile apps, their time on Pinterest increases. So there is a synergistic effect. And so we have a lot of hypotheses around that. Maybe just that become such an important part of your life that you start using the service more because it’s more available in all different scenarios. You use the service more, but we wouldn’t have known that without doing the data. It’s really hard to see like, okay, so if you use the mobile apps, it increases your usage of the website as well. I think we should be pushing people to do that more. So direction and in addition to just our use of usual like intuition.

Bronson: Yeah.

Tracy: These are just like a sampling of some of the engineering things we need to work on, but there’s a lot more going on as well.

Bronson: No, it sounds like it are. If you were to have 20 more engineers today, what’s the hard problem? You would put them on that everybody on Pinterest knows you have to deal with, but it hasn’t been dealt with yet. Yeah.

Tracy: That’s a really great question. So I think right now we’re actually really constrained on all the different clients. So we don’t have enough web and mobile engineers. So we actually have a lot of really great work happening in the backend, like improving our feed services, improving search and discovery. And we’ve been working with a lot of like rich pin to get pins to be more actionable and useful in getting data from partners. But we don’t have enough resources to build out systems in the client and build out these different experiences. So I wouldn’t say necessarily it’s like a really hard problem that we just need people to think really hard about is we have stuff that we need to build that we don’t have enough people on right now.

Bronson: Yeah, absolutely. Now, what’s the engineering culture like there? Because you’ve worked at some high profile places, you’ve been a coral, you’ve been around Facebook. How has the culture different? How is it better? How is it not as good? Tell me about the culture a little bit.

Tracy: Yeah. So one thing I would say about Pinterest is that we value Pinners a lot, so we talk about being a pinner led culture of some companies like Google are very clearly engineering driven culture is Facebook is kind of engineering and product driven. Quora was very engineering and design driven. I feel like a Pinterest. It’s not so much driven by one of these teams as it is very collaborative. So we talk about knitting as one of our values where we say like the best things happen when a lot of different disciplines knit well together. And there’s a really big focus on collaboration, understanding all the different aspects. So when we are launching new products or thinking about new products, we are not just engineering but the product side, product management, design, community data. So we have a lot more voices represented and I think it’s really awesome that we do that because it makes sure that, you know, the things that we’re building as engineers are useful now for the user experience and as engineering. I trust that’s something I’m going to be building will be well designed and that’s a little bit different than some of the other places I’ve seen. In terms of like the collaborative element. I feel like I learn a lot more about a lot of different things. So when we’re thinking about new features, I’m thinking a lot more from the aspects of from the perspective of what does the community team see? Like they’re responding to Pinners and seeing the sort of tickets are coming in. So the points of confusion, like what? I can work closely with the community. I try to understand that user perspective a lot better and it actually helps me to be a better engineer as well because I think about what are the potential use cases or what are the right abstractions to design that we need to build out more features. Like, I might just be building one thing right now, but I’m going to be extending it and scaling out to more use cases like what is the right type of abstraction to build? I think it makes me a better engineer to have knowledge and perspectives like that. But yeah, it is definitely some people prefer to be in an engineering driven culture. If you’re an engineer and you want to call the shots in some places, maybe better for you.

Bronson: As I’ve heard you talk about kind of the the culture, the people, the technology, it seems like a theme is kind of emerging, which is when you first start, you know, everybody does everything because you just have to. That’s the duct tape stage. Everyone’s just in there. You’re a part of probably analytics and the feed and the emails and everything else. But then as things grow, but the technology of the silo of things. So now we need a database just for this. We need database just for that. We need processes just for this. And then also with the people yet to silo them off, we need a group just for community. We need a group just for infrastructure. And then the goal is how do you let the technology to speak to each other and the people speak to each other in really beneficial ways? Is that seem like what you’re describing? Is that accurate?

Tracy: So yeah. So there is organizational challenges and scaling where you can’t just have everybody talking to everybody else anymore. These are two need, but are structures around who is the point of contact, who is the owner for some project or a feature or some piece of infrastructure. So yeah, it is very important that we start defining these roles where clearly when you’re really small, there’s like one person community and they talk to everybody else, an engineer on email and they figure out everything needs to happen there. Now we need to have better process around that. So we have regular meetings for, you know, various functions to sync up. So we have, you know, biweekly meetings with the leadership team. Like all the different teams are going to talk to them to talk about what products they’re working on, like the progress in the last two weeks, what needs to happen in the next two weeks, what’s with their blockers? There are or other dependencies, so we make sure that we’re in sync with people at that level the product managers will think of regularly across engineering. It depends on like what sort of things you’re working on, but there are regular sync points to make sure that we’re all working well together. Mm hmm.

Bronson: It’s almost like if the engineer, the culture, you know, you talk about dependencies, you talk about, you know, sinking with each other. It’s almost like there has to be some kind of language that you guys are, you know, coding in to make sure all the human pieces work together in the right way at the right time doing the right thing.

Tracy: Oh, yeah, definitely. It’s it’s, you know, really fun challenge to look at how we scale the company as well as scaling the product, just like from the engineering side, like how do we structure our code so that people can work on it? One of the best way to do that was to split out things in different services and then moving those services into their own get repos. So you’re not stepping all over each other’s terms and you’re working on your code. We can do deploy separately for the different services. So it’s a really going from engineering slightly different engineers or different teams are owning different pieces of our infrastructure or different pieces of the product. So you know who to go to, who’s responsible for something. But across the company, as well as that important to figure out, like what are the areas of ownership and who are the people that you need to talk to for various things? And there’s a lot of things that require just very creative thinking about how to scale. So an example of something I worked on recently was our BUGS program. Mm hmm. So bugs are like so many bugs all the time. Any time you’re writing software. So there’s, I suppose, to come from, like, code issues, both that come from data, the bugs that come from like servers and down. So those are pretty challenging to work with because it’s very easy to not roadmap them or not put them into your planning and the way bugs get surface like they can be tickets filed by Pinners or just community members. And so the people who work in the community team see those tickets for others, like an issue that we need to address. Then how do you route those to the right engineers to fix? Understand the severity. So you’ve had to come up with a process for that. The the way you do it when you’re really small is that you have like one person who’s responsible for routing all the bugs and they just keep all the information in their head about what they need to do for different types of bugs because an engineer they can help like dove into it and diagnose like where the issue is and send the right engineer. That doesn’t scale very well when you have like 100 plus engineers and there’s a lot of bugs that come from interactions of code where the assumptions are different from the caller and the colleagues perspectives which they get the service boundaries. Like if you have different assumptions, you end up with buzzwords. They’re just like straight up bugs as well. So it’s a bit trickier to figure out who to root bugs to. And something that shows up on the website could be something. It could be due to an issue happening at the database level, or it could be something at the cash level or maybe at the application logic level, or maybe it’s an API level or maybe it’s in the web app level. So those could be of where to assign those things can be tricky. So what we’ve been trying out here is building out some of a decision tree and a map to work somewhat. And there are some like useful tricks that you can try. So like our website is built top of the API, so you see a bug in the web. One thing we’ll talk we’ll do is like try to reproduce in the mobile apps so that if the bug can reproduce with web and mobile, that means that the bug must exist at least at the level or below. So different questions like that that you can ask to help figure out the routing of where these issues should go, and then figuring out like what is the best way to represent like what people are working on so we know how to root things to them. So, you know, that has to do with our offsite service, which is like when people are clicking off of Pinterest, we have to run a bunch of checks to make sure that content is safe. So our black ops in the town so that but there’s one person in particular, this one for that service. So we can say like, okay, if you run into issue with this thing, go to this particular engineer and in building out this sort of like routing process. We reduce the mean tag of resolution of bugs because we’ve gotten bugs to the right engineers much faster and lot of how can you see bugs reported by committee? There are new issues that come up like recent code changes. And so you can get engineers that use the bug just like a day ago clear to them. It’s very easy for them to fix that. It’s trickier if you write to the wrong person. They don’t have the context for those changes and they bounce it back to somebody, help get somebody else. So getting something, getting about the right person immediately helps to reduce that mean time resolution of bit. So it’s a process that we’ve had to build out and it’s something that we’re still iterating on. Like right now, it might make sense to have like all the all the bug that can be brought out through the system, go to one default engineer, but at some point it needs to be more scalable. So like, do you have like a protein point contact or is there some other way to file tickets? So there’s a lot of interesting like scaling challenges within organizations. So it’s like a blend of the technical and human elements.

Bronson: MM Yeah.

Tracy: One of the things that we have to work on.

Bronson: Yeah, no it’s great to kind of see the inside of Pinterest because most of us don’t work on companies that are that large, don’t have the same kinds of issues. But like I said before, if we’re lucky, these are the kinds of things we’re going to have to think about. And it just gives us a really good kind of working model of someone who’s many steps ahead of us so we can see the road ahead. So it’s just great getting inside of pictures there. Now, you’ve been the tech lead on a few different projects at Pinterest. And one of the things that you led was the the big redesign that you guys recently launched, rewrote the code redesign, rethought things. What was the driving force behind that decision to kind of redo the website in some ways.

Tracy: Right. So what’s like friction? I wasn’t the lead on that project. I actually switched the web team and started working on that. But I can talk a little bit about the reasons why we decided to do that. So on the engineering side, the first was that we just had so much technical debt that it was very difficult to develop code and be confident that things we’re building would work the way we expected and the things that we didn’t want to affect weren’t affected. So that was one thing where we just had a bunch of spaghetti code that was like duct tape. It got thrown together early on. But another big consideration was that Pinterest was started a few years ago when mobile apps were not nearly that big yet. So everybody was kind of like this web first mentality. We built our website first and then later iPhone came out and like those apps became more popular built API to support our iPhone app. But we had a branching of the code between the web and the API and API. Traffic has far surpassed web traffic by this point, but for a long time before this spear project, we had this like splinter splintering of the code where web was actually by default, prioritized for development. A lot of ways, a lot more people are familiar with web development. So it’s easier to go in, like have something up in the web which is go had privileged access to a lot of our data, just kind of reaching to grab stuff that it needed, whereas the mobile clients were gated on the API. So because like these were a split, there were some things where the web could like, you know, get access sooner that the API client or was an API development. There is sometimes inconsistencies between the two where we might just do like filtering differently on search results for example. But the two that you get different results on the different platforms, that was something we wanted to address. We didn’t want this like fracturing of our engineering tension. And so last thing was that the way we had built the website and we didn’t have a real client side architecture, so we couldn’t really be a full featured web app. Being a web app, something like Gmail, where you sit on Google, the kind of look around, but you never really leave Gmail account that is app. Whereas like something where you’re clicking between pages and doing full loads. Like, so you’re on the New York Times and you click to like another page. It doesn’t really feel like a web. I just like loading different pages. We want to be more of a web app where we can maintain your your state. So you’re scrolling down a feed and you click into a floor or something, you’re jumping all the way in and then you want to back out. We want to maintain your stay and all those different views that you were in. Want to build the cache from modules that you’ve already loaded so we can be more performant. So what we wanted to do on the client, we just didn’t have like the architecture at all there originally. So these are three major reasons that we had for the rewrite on the code side. But at the same time from the design side, we knew we want to design refresh of the entire site and so we bundling the two together to do a code rewrite while also redoing all the design.

Bronson: Yeah. Was it do you feel like you bit off too much or is that the right way to go about it, a design and kind of code. Do you think it worked out that you did them both at the same time also?

Tracy: Yeah, it worked out. There’s definitely pros and cons of doing everything at once. So one big danger is that if you bite off too much, you may not be able to get any evidence. So there’s more moving pieces, more points of failure, more reasons why you might just completely fail. Luckily, we do not. We actually get to rewrite everything in a reasonable timeframe.

Bronson: How long does it take? I’m curious.

Tracy: It took about eight months to get to a full rollout, although we were showing we had code in production, a serving in areas like three months in OC.

Bronson: Was that to see what the bugs are or to see how they’re interacting with the design? Was that kind of customer development at that point?

Tracy: Yeah, that’s what I was just like making sure that it would work at all and test it. Just like, just even like support this many requests. Like, can we show this to users at all? It’s also very gratifying for engineers and people who are working on stuff to start seeing things in production sooner. You feel like you’re making actual progress and you make sure you’re not going off in the deep end and never going to come back to did you start putting stuff in front of people so you can iterate and figure out what the big issues might be? And, you know, so we were pretty happy with the way everything turned out. There definitely were some challenges in redoing the code in design at the same time where we couldn’t isolate some of the effects. So we saw that some numbers dipped. We didn’t know why we.

Bronson: Didn’t have a control group.

Tracy: Yeah, there’s yeah. When you want to do like an experiment, you usually just want to change one thing at a time. So you know that the changes that you see are due to that one thing. So we had a number of cases where we didn’t know if a dip in the metrics might be due to code changes. Like maybe our performance was worse, like maybe we had increased latency, so that would be a problem or even bugs or maybe use of the design changed. So one example is that we saw that board, all low numbers went down as we were testing out the new version. So we didn’t know if it was because the poor follow button was now in a different place and it was now gray instead of red. Or if it was something else, it turns out we did run a data analysis and figured out that what was driving more follows on the classic experience was a few different models of one of the models that we just hadn’t prioritized implementing on the new site yet. So we took a modular approach to implementing everything. So there’s something that we prioritized, like implementing firsts. One of them was this on success, in your opinion? We show you a different board that you might be interested in following that’s related to be just pinned. And that was something that we had a prioritized building the new site. We didn’t think it was that important because that was driving a lot of the numbers. So it wasn’t a design issue. We didn’t run this design experiment to say, Oh, maybe we changed the color of the flow. But back to read a little help, maybe there was like some other things. Maybe we need to reposition. It turns out we just needed to re implement that that model that we were going to implement at some point. We just had some we and that accounted for the difference. So it a little bit harder to understand why changes in the metrics were happening. But overall it got us to the completed say with like a new code base and new design much faster than if we had done them serially. So we knew that we were going to have to rewrite everything. We were going to redo all the, the, the front end stuff. Well, with such a new design. So we didn’t want to rebuild the old design on the new framework and then have to rebuild everything again. It probably would have been really demotivating as well for people to like Oklahoma. They knew that they would immediately have to scrap. So in terms of like the end outcome, they made a lot of sense for us to do the things like both of those together, and we got a pretty good place with our code. And just like freshly design, it was good also that we were changing the design in some ways because other people who were not working on the project like they see the visible progress like that.

Bronson: Yeah, which feels good.

Tracy: It feels good. It’s super important that the team that’s working on a team that’s observing externally like our all but but into the projects are like adequately resourced and there is general support for it within the company. Yeah. So in the end it worked out really well.

Bronson: Yeah. Did you guys use the same programing language when you did the rewrite or did you have to really?

Tracy: Yeah. So let’s say programing languages, which is really key as well. So it was like Python, JavaScript, ginger for the templating and SAS for us. I think that was also really key in the success of the project that we had experience working with those languages. I’ve been on another rewrite project in the past at a different company where we were writing to a new language, and it was pretty tricky because nobody knew the target language that we were porting over to. So that introduced just like a whole other level of difficulty. Like, I know how to do this thing in Python. I don’t know how to do it in Scala. Mm hmm.

Bronson: And what’s the reasoning? Why people want to write in a new language? Is it because there’s some new trendy language and we feel like we need to do it? Is that is that the reason why your last company wanted to change languages?

Tracy: That there’s some reason to have a. We’ll say like four languages that are like Java scholar, that are compiled statically typed, better concurrency that gives performance reasons for doing that. There maybe a little bit of the trendiness, but I think usually it is driven by the technical considerations like, you know, skull is more functional. So in theory it’s a lot better for concurrency and you can reason about the code better. It’s not too verbose. And so it should be still pretty easy for developers to write in like a lot of complaints about how about Java that you have to write a lot of boilerplate all the time, but it’s not very fun to do that. But Scala like this is sort of smart type inference. Are you going to write code that is not too verbose, more similar to like Python or something? That’s but you also get a lot of the benefits of being on the Java having a B compiler. So every time you make sure that you don’t do really silly things like passing arguments incorrectly or whatever. But in this case, I was like, because we were writing to the same languages that we’re already very familiar with and made things a lot easier. We heard earlier languages and like what they were capable of. So it also gave us better intuition of what was possible and what we could do. Yeah.

Bronson: Yeah. I’ve been involved in complete rewrites as well, and I breathed a sigh of relief when it was over. Yeah. Yeah. It’s a scary process, isn’t it?

Tracy: Yeah. I mean, it’s really cool when you get to the finish you like while it’s completely clean code basic, very modular. So you got what you’d want. Like a lot of things to get into a code base and you’re like, Oh, you know, if I, if I could start, if I could do this over from the beginning, like it would be different knowing what we know now about the use cases and such. But it’s very hard to get those opportunities to actually rewrite such to the way that you would want to.

Bronson: Yeah. And then one of the other things that you worked on, Pinterest was the feed quality is the feed quality just the quality of the pens you’re being shown as you scroll around the site?

Tracy: Yeah. So there is the part of that we were working on was specifically about the public feed. So if you’re looking at categories like art or history or gardening travel, we want to make sure that we’re showing you good content that it’s thematically correct. For a while we were just using what to label as the categories, but the way that word was a little bit grid. Also, if you create a board on Pinterest, you can choose the category that it belongs to. So we’re just like it’s just like Limited said, like 30 some categories you could pinned to it. We would say, okay, based on the category that you label your board as, like we would put those into the public feeds of those categories. But there’s a lot of reasons why that doesn’t work. Like sometimes people don’t really care about labeling their boards correctly or sometimes supports don’t belong in one of those categories. Like if you have something that’s just like both travel and outdoor stuff, like.

Bronson: So how do you get around that? Because it seems like a hard logic problem there.

Tracy: Right. So what we did originally was like it’s very simple, like based on the product, like there’s no interesting like data work being done there. So we start shifting over to a more robust model where we could use machine learning inputs so we could start training to figure out what is this pin actually likely to be?

Bronson: What are you looking at? I mean, what what in the data are you looking at to decide what it might be?

Tracy: So there’s a lot of a lot of signals we have, actually. So for pins in particular, we can have the images to see like all the occurrences of the same image. If you just look at the descriptions of all of those pins. Look at the words that they’re on. Look at co-occurrence of other pins on those boards. We can look at board data so we can start to infer things about the pins on a for like we put together with that information, we can look at some of the the other features like the domains that they’re coming from. Right. So there’s like something from like art or daily, there’s probably architecture. So there’s a lot of other segments we could start looking in. And then we were starting to use more of the machine learning to just like build these classifiers to see like, what is this for? Like it would be what is this pin likely to be so we can make sure that the content was thematically correct? Yeah, but it is a little bit tricky. Like there are some things where the definitions of those categories are a little bit like taste driven. So what belongs in art?

Bronson: Yeah. It’s hard to play semantics with code.

Tracy: Yeah. What? What is considered art? Like somebody sketches on deviant art, Kaunas art. Or is this like museum quality art? What is history? I mean, like, some people are things like from their childhood, they put out a history of the wall. It’s a different definition of history, right? So there’s something in it to define, like, what do we believe this category means? So we have kind of like a target set that we’re training against. So there’s some, there’s some manual work to define, like what we thought each of these categories meant. So we can start building classifiers for those types.

Bronson: Yeah. And now as you increase the feed quality, did you see the growth increase? Did you see kind of a tangible impact with engagement and things like that? Or did it really not show a correlation there? What did you guys see?

Tracy: So we’ve actually have seen that. And also like more recent feeds, improvement projects, the quality and flow, like how fresh are all the pens and like how fast are things in the groups of the feeds actually does correlate with engagements. The number of reports that we’re seeing and like how many people are active on the service. So it is important that we get those right because these are the main points. They’re not the main, but they’re an important point of discovery. And we’re talking about our mission being to help people discover things that they love. Those reports are pretty important. So we’re doing a good job with the feeds and showing people things. It does improve engagement.

Bronson: Yeah, well, it’s like you mentioned earlier, you know, Ben Silverman, he says, you know, that you guys put Pinners first and this is kind of an example that you put Pinners first because what are they doing? They’re pinning they’re trying to find things they love, like you said. So if you make that experience great, it’s going to help everything in the company, right?

Tracy: Yeah.

Bronson: And I think that’s just a good way, you know, something for people to take away. They’re watching this is there’s a core competency in every company make that we’re going to see better and better and better and better, because whatever you do around the edges won’t matter if you lose that. It doesn’t matter how great the steakhouse looks on the inside if the steak isn’t good right now. Let me ask you this. What other projects have you worked on at Pinterest that maybe had anything to do with the user base and growth and retention and engagement? Have you done anything else that touches on things in terms of the growing company?

Tracy: Yeah. One project I worked on a while back was emails, so both kind of this Android notifications type of emails as well as a weekly email. So those are super important for reaching out to people like this is your off site channel to get people to come back. So notifications are pretty straightforward, but there are some things to do there. So we first had our emails was a very basic like slack text on white background. Like we links, the bagels are like, Oh, somebody repaint your pin. But I just started building those out to be like interesting HTML emails with like images and then we could use that as opportunities to just other awesome to look at. So might just be an example of like somebody with your can we show the border, the pin that they got or the board that it got re pinned to and it’s thematically related because it is a relationship you’re interested in that sort of content you might click through and look at that authentic. Oh, I actually really like the board that someone referred back to. So it’s almost like recommendations via notifications. And then there’s also the other category of weekly emails which are more recommended content. And so those are super key for reminding people. It was like great content on Pinterest and just like another touchpoint. So here we have some cool stuff that we like to add. The things that you’ve added to Pinterest and like based on those, we have some recommendations to show you.

Bronson: Yeah. How how much would draws traffic take a hit if you stopped seeing that emails tomorrow would it just be a drastic decline?

Tracy: It’s they’re pretty important.

Bronson: Yeah. How did you guys deal with like deliverability and things like that? Do you do it all in-house because email is like a whole other engineering feat? Or do you guys rely on third party services to get the emails delivered?

Tracy: So we use great pretty heavily. We also have a lot of in-house infrastructure to make sure that we’re if we’re weekly emails, we have to generate all the emails, which is like pretty substantial work, like hitting all of the different databases to get the data that we need, like queuing up the emails correctly. So there’s a lot of in-house restriction needs to be built out, but we do rely on third time services like SendGrid for things that we don’t need necessarily to build in-house.

Bronson: Yeah, and that’s good to hear because if you guys built that in-house, I was going to be really jealous of your talent because I would just get into the absurdity place. Well, this has been great. I got a few kind of final questions because you have been responsible for and a part of the teams that have really put the pieces together for Pinterest. I mean, you’ve been building the building blocks of this company that’s grown so fast. So I want your take on this. How much do you think growth is luck and how much is skill? Because obviously you have a very skilled team. They’re doing what they do. But how would you how do you shake it out there?

Tracy: There’s always a component that’s going to be luck. But you can get really good at growth and you can build out better engineering practices, like better abstractions, like better experiments, frameworks that you can run a lot of experience simultaneously and learn from those and start building things. But this you can make growth the competency of your company as well. So, I mean, there’s going to be that kind of luck, but you can be very prepared for good luck and increase your chances of luck, which is, I think, what you need to do in order to get a lot of growth.

Bronson: Yeah. Now, let me ask you this. What lesson have you learned about growth in the last couple of years at Pinterest that surprised you? Something that happened that was like, oh, that’s how growth works now. You just kind of always remember.

Tracy: There’s a lot of lessons. One interesting one is that growth is actually going to be a lot of little things change together. And, you know, this new beta launch might just bump up your metrics 5.5%. The next one, like up it up by like another point 5%. And when you train a lot of these small things together, they can have a really big impact. So it’s important to understand, like, how everything contributes to your overall system. Another thing that we’ve that I’ve kind of learned about is understanding user state. So starting to model out how people are engaging with your service, like if they’re new users, that they’re very active users or if they’re users stale. Whereas like categorizations, you can have a new model of how people are transitioning between the different states. You have a good idea of where you want to start putting in those incremental gains. So, for example, if you have a lot of people signing up for your service and then a lot of them just don’t become active users, then what you need to do is focus on user experience. Like, well, do sign up showing them how to be active, or if your problem is more that people are not even signing up, they need to focus on acquisition like the landing page or sign up. So there’s a lot of places that you can get drop off or you can focus your attention to maybe reactivate users. You need to have like a good mental model of how everything is working, where you can figure out where to put your engineering focus and your product focus.

Bronson: Yeah, if you don’t segment your users, you don’t know where the funnels broken, you don’t know where to focus. Yeah, that’s great. So last question again, this is an awesome interview, but you know, people watching this, they’re building their startup. They’re a small team, they’re a middle sized team and they’re trying to grow. What’s the best advice you have for any startup is trying to grow? No pressure.

Tracy: I think just kind of going back to the last point, understanding your growth model is really important with being disciplined in like instrumenting, understanding what your metrics are, where the drop offs are, where you’re getting growth and understanding all these different sources will really help you to focus your attention. It’s really important to focus on something when you’re very small. Aside from limited resources, you need to know the highest leverage things to work on. So you need to have a good model that you can fall back to understand. Like, if I were on this, it’s going to have this sort of gain instead of just like focusing on small things that don’t matter.

Bronson: Yeah, focus on what matters. Traci, again, thank you for coming on growth narrative.

Tracy: Yeah, thank you.