7. Judd Antin of Airbnb
We kick off the second season with Judd Antin, the Director of Experience Research at Airbnb. Judd and I speak about their model for embedding talented generalists with product teams, skill-sharing among researchers, and just what exactly makes research “sexy.”
I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. – Judd Antin
- Judd Antin
- Follow Judd on Twitter
- Alex Schleifer
- Kathy Lee
- Janna Bray
- Steffen Kuhr
- Natalie Tulsiani
- Brian Chesky
- Joe Gebbia
- Katie Dill
- Adrian Cleave
- Improv and user research
- French Culinary Institute
- Your Data Are Wrong: The Hype and Reality of Big Data
- The Incredible Hulk
Steve Portigal: Welcome Judd.
Judd Antin: Thanks, Steve.
Steve: It’s great to speak with you. Maybe let’s just start as we always do with just kind of the broad strokes. Do you want to introduce yourself? Tell us a bit about you and what you do, what your role is.
Judd: Sure. Sure, happy to. So I am the Director of Experience Research here. So experience research at Airbnb is a team that was formerly called Insights, but what we really are is a UX research team. We are embedded in our design organization so we have designers, content strategists, product managers, engineers, data scientists as our partners and we’re a team of 17 folks at the moment. We have a pretty diverse group of people on the team which is something we do on purpose. And everybody is sort of embedded in their teams, working directly on the day to day of products for guests and hosts all the time. So, yeah.
Steve: That’s good. And of course those intros, as we know from research, like we could just follow up on all the things that you said. That would probably fill our whole interview. Let me go back to some things. Can you explain Airbnb?
Judd: Sure I can. Let’s see if I can do this in a nutshell. So Airbnb is a marketplace that let’s hosts who have space come together with guests who need it. So…we provide an opportunity to travel in a much more local way. So I think you know, it’s – you know Airbnb has been one of the poster children for the sharing economy but I think for us the way that we think about it is that Airbnb is an opportunity to connect hosts and guests together, to have much more genuine local experiences. Local in the sense that you’re traveling to a place that is pro-, potentially just in a neighborhood. You may be staying in a spare room or a place where your host is either there or they’re on vacation and you’re sort of experiencing their city from their point of view and you know getting recommendations from your host about where to go in the neighborhood. So I think that is the thing that makes Airbnb really beautiful and unique is that it’s kind of this view of travel which is to say yeah you show up in San Francisco, the Golden Gate Bridge and Alcatraz and Coit Tower, those are important things that we know from research that everyone wants to see, but if you never left Union Square and Fisherman’s Wharf it would be hard to say that you got a real sense of what San Francisco is like. So Airbnb really lets that happen.
Steve: And you started off at that great explanation by contrasting sort of a common or public perception around it – you said “poster child for sharing economy.” Is there something that – and obviously everyone in the company – everyone in any company understands the contrast between how they’re perceived and – I’m just thinking about you in the role that you have and what your team does. Do you deal with that – I don’t know if it’s a gap or delta – the world thinks of us as this. You know we go out and people to sort of understand. I don’t know…
Judd: Oh yeah
Steve: You’re nodding so I’ll let you answer.
Judd: All the time. We spend all of our time talking to guests and hosts on Airbnb. We make a concerted effort through travel and remote research to get outside of the Bay Area bubble for example because, you know, Airbnb is very much present here. But what that allows us to do is see what the everyday experience of traveling or hosting on Airbnb is like for people and I think that means that we get a huge dose of real talk. You know we have a mission – you know sort of our tagline as a company is belong anywhere which is something that we all pretty deeply believe. We want people to be able to have a travel experience which lets them have a really genuine welcoming experience of a place that they can start to feel at home there and belong there. But we learn a lot about how that vision sort of translates to reality in the everyday life of a host, or the everyday life of a guest who is traveling. For host that might mean we expose the diversity of hosts. So for a lot of hosts we talk to the – the money is really an important reason that they do it, but they also feel a strong connection to hospitality. They like to experience people who are visiting them from all over the world and they talk about that as a motivation for doing it. For travelers, similarly, we find out that finding a cheap place to stay is often – I mean let’s – however much – that’s not going to be a part of our vision at Airbnb, but it is a reality of the way that people think about Airbnb when they travel. It is a part of their motivation for getting on the platform. And so I think one of the great things about the research team is that we can be that dose of real talk. We can say like okay this is important. This vision is important. The mission is important. But let’s talk about the real lives of guests and hosts and what matters to them.
Steve: How do you characterize – I don’t know, I can only ask it through suggesting an appropriate metaphor that – and I know the synthesis or the ping ponging, if you’ve got – you know here’s public perception, here’s what real people involved – here’s what the real talk says. Here’s what our internal aspirations or you know Kool-Aid or whatever it is for any organization – I mean how – what’s the sort of – tell me about how those things come together. You have to deal with what’s believed and what the aspirations are, with what the world says and with what the real talk from real people is.
Judd: I think we have a pretty happy balance between the Kool-aid, or mission driven aspect of what the company does, and the real thing, the real talk of people. What we do most consciously is probably to remind people in this building that we are nothing like the vast majority of people on Airbnb. And that’s where I think the real talk matters because if you make a bunch of assumptions on the basis of what seems obvious or intuitive to us here in this fantastic building in San Francisco in SOMA, like we’re going to get it wrong. We’re going to get it wrong for California, for the United States and certainly for the world, the vast – the majority of our business doesn’t happen in the United States. So I think that is one of the most important roles of real talk is just to say… It’s not that difficult. I don’t think we feel a real tension between the mission and the values and the everyday lived experience of guests and hosts. We just have to remind people that most people’s daily lives are nothing like ours and so we just have to keep that in mind when we make design and product decisions.
Steve: I love that that’s not a tension. Those things can work well together. Not every organization is the same and maybe elsewhere you’ve observed that as more of a tension.
Steve: But, why? How does that work well here? What’s going on?
Judd: Here’s my theory. My theory is that it’s because we’re a travel company and here’s a list of things that we don’t have to worry about – advertisers, monetizing engagement. What that means is that to the degree we can get people to find the perfect place for them, where they want to travel, get them out of our digital products and out into the world to travel, that’s success. You know what I mean. And so I think we don’t have to make a lot of assumptions about what motivates sort of digital behavior. We have to get people to travel and I think it attracts a certain type of a person who – and you know that combined with our mission for belong anywhere, we think it’s easy to have that kind of empathy for the world because traveling is such a – as an experience is kind of full of empathy. It’s like understand the world from someone else’s point of view.
And as a research team we kind of focus on that too. We’re like hey product team, let us help you understand the world from another point of view, the point of view of guests and hosts that are nothing like you.
Steve: That’s wonderful. So the company that’s about getting out of – giving people the chance to experience something differently than what they’ve experienced, in other words travel, is one way that a culture gets created that there’s a hunger to understand – a willingness and a hunger to understand.
Steve: The world is different than what we’ve assumed.
Judd: Yeah. And I think that’s a big selling point for researchers who want to work at Airbnb because you know they think – travel is such an evocative experience. You know if you look back at the – I think the average person who thinks back over their life to the 10 experiences that were most educational, most instructive, most inspirational for you, a good chunk of them are probably travel experiences. So people’s eyes are open. Like it’s emotional. It’s intellectual. And so as a topic for research, you know facilitating that kind of travel experience is really sexy for most researchers we talk to. It feels really good. It feels really real. You know I think sometimes working in the domain of digital products you can think what does this all add up to? You know what experience I’m really making. Like we don’t have that problem because travel, we’ve all had experiences of travel that are truly sort of transformation.
Steve: So what are the experiences that researchers have? Can you give some context or situations where researchers are involved in – are researchers traveling to study travel for example?
Judd: Yes. So we have done a fair amount, not as much as I want us to do, but we have started to do travel, primarily as a way of understanding – getting outside that Bay Area bubble. We spend a bunch of time in the homes of hosts. That’s one thing I think we do a lot of. So one thing we haven’t done yet is kind of like I think a travel along project. You know if you research at Uber you could like ride along with an Uber rider or driver. We haven’t done a travel along. I don’t know, that seems a little invasive and weird. But we spend a bunch of time in hosts homes and one of the reasons we do that is because that’s one of those things where it’s very difficult to get a full picture of the day to day of a host, the things that challenge them just in the flow of trying to provide hospitality and use our products without really getting into it with them. Find out what’s their routine of sort of messaging with potential guests – you know scheduling, cleaning, doing key exchange, all of that day to day stuff. It’s really hard to find out the mechanics of that without being there. And it can be very difficult to get the context and depth and empathy we need about motivations for hosting, let’s say, without a really in depth face to face conversation and where better to have that conversation than in the home.
Steve: Right. It’s just funny sitting here that you refer to it as their homes. Of course it’s their homes, but I’ve only ever been a guest, never been a host. So the unit that you’re referring to, this environment, it’s a place that I stay. I forgot that it’s somebody else’s home. So to hear you say that, even the language, which I think is probably a key – that’s probably key for you guys to be talking about these as homes.
Judd: Yeah. No, I think it’s true. And you know I’ve been thinking a lot about the language we use. You know I think in research – you know I don’t know many people in user experience who use the word subjects. We mostly talk about participants and we use – you know sometimes we talk about users. You may – you know we can talk about whether that’s a good thing or a bad thing, but I kind of always want to turn to guests and hosts – the things we do for guests and hosts because it’s a reminder of – it’s similar to the way you said oh right it’s someone’s home. It’s like right, this is a guest, this is a traveler and this is a traveler who is showing up in somebody else’s space, in their home where they live and sharing that space whether or not the host is physically present at that time – they’re sharing that space. And on the flip-side like this is a host. Not only is hospitality and hosting something that we care a lot about as a company, something we try to promote with our products and look into with our research, but it’s a home. You know it’s really personal. Again, like that is a sexy thing for researchers because it’s like one of the most important things in your life is where you rest your head. And the idea of letting somebody else into that home can seem – you know that’s where the issues of trust come in that are so interesting to look at because you have so many dimensions of that. You have the sense of trust that’s instilled in Airbnb as a brand, Airbnb as a platform. The way we facilitate relationships between guests and hosts. And then you have this face to face personal think where I am letting you into my house if I’m a host or I’m going into your house if I’m a guest. And you know I have a lot of people when I talk about Airbnb that ask about trust and safety and things like that. And the only thing I’ll say about that is it gives me a huge amount of faith in humanity how rarely there’s a serious issue. You know. The number – like millions and millions of nights are stayed in Airbnb’s and the number of serious issues that happen when people trust each other and guests go into homes is just minuscule. From our point of view it’s too many – one is too many.
Judd: But it’s really sort of restoring a faith in humanity and I appreciate that we are constantly on the lookout for doing research that can facilitate that. Like creating wonderful rich, trusting interactions between guests and hosts.
Steve: And another reframe for me that’s helpful to think about this is these two groups of people that are being connected as opposed to a bed and a toilet for the night.
Judd: And to the end of real talk right. There are some guests who do see it as a bed and a toilet. You know, it’s a cheaper bed and a toilet then they can get at a hotel. But not all guests, probably not most guests. You know the aspect of hospitality, the aspect of getting into a neighborhood, living like a local, it’s really powerful and I think it’s a real motivator for Airbnb.
Steve: I want to dive into some other language that you used and maybe get you to unpack that a bit. You said a couple of times that certain kinds of problems or areas are sexy for researchers. So that suggests some insight into what appeals to researchers. Like what we like doing. So what does sexy mean for researchers?
Judd: Yeah. So in my experience, when I say sexy – when I say sexy I mean two things. One, a topic for research which is really meaty, which you can really wrap your arms around and spend years at a time diving into. It has a lot of dimensions. It’s not easy. It’s probably a challenge. I think most researchers find the challenge sexy. For example it is something that requires us to deeply understand a problem and figure out how to translate it into design and product and communication. So that’s one aspect of sexy. And then the other thing that’s sexy I think for a researcher is the fact that they can be set up for a direct line to impact product with their work. I think when I talk – I talk to a lot of candidates and when I think about how research is structured in their organization what I hear is well they had to advocate for budget, for a study, and it took them several months to do that. And then they completed the study and then they had to present it up the chain three or four or five levels to get that VP to advocate another VP to get that PM to put it into the roadmap. And it hurts my heart. You know it’s like the worst way to do research. And so I think researchers find it sexy when they don’t have to do that which is – which is one of the beautiful things of Airbnb which is that we don’t have a perfect situation but we have researchers set up as partners, with designers, with PMs, content people, data scientists, engineers, at every stage of the product cycle. The way that they have impact is through being there every day for direct conversations about what we should build? How should we build it and why? Are we doing it right? Iterative development, launch and learn, repeat, you know. And that is sexy to a researcher.
Steve: Can we talk more about that staging or how people are set up? So there’s a number of individuals with different roles that are partnered together.
Judd: Sure. So, yeah – so we have an embedded model for research where – which I would contrast to the more service-based model where there’s like a central organization that sort of fields requests and sends out a researcher into the field. Our model is an embedded model where a researcher gets situated into a product. Let’s say that product is search. That researcher sits physically next to, hopefully between the PM and the designer, the engineer, the data scientist. And I make a big deal of that physical presence because I think that’s where a lot of the action is. There are a lot of – I’ve had conversations with researchers before where they say I need to get into that meeting. I go, what meeting? And they go that meeting where all the decisions happened. And usually I look into it and find out that meeting doesn’t exist. That meeting happened like in between
her desk and his desk at 1:45 and it happened because – like I’m elbowing a shoulder right now.
Judd: Like that’s why it’s so important to physically have that person there, physically present for everything that’s going on in the product cycle. And that person finds partners in all the other functions. So certainly with design – we have very close relationships with design, product managers. We have a strong and wonderful data science team here. Engineers of course. Content strategists. Everybody is kind of at the table, one very collaborative team trying to make great product.
Steve: So I have a dumb question and we can say there are no dumb questions.
Judd: There are dumb questions.
Steve: Thank you. Thank you for affirming my dumbness. You know product cycles, development process focuses on different tasks are hot and heavy at different points. So I’ll just ask this in the dumbest way possible, what’s the researcher doing at different stages of that? What does it look like for them?
Judd: Sure. Yeah. So I guess it would start with an exploratory or formative stage where we’re trying to figure out what we should build. What are the problems? What do they look like? Why are they problems? So very often that’s a place for in-depth qualitative research. We love to have a model where we do in-depth qualitative research and then we follow it up with really rigorous larger scale survey work to check prevalence. So that kind of one-two punch is really powerful for us so we can say here’s a deep, contextual rich understanding of these problems and we know a lot about how prevalent they are and we can segment them a little bit in ways that matter to us. So cool, we did that. Okay now maybe we have a product road-map. Or we’ve decided to build a thing in a certain way and the designers were hopefully participants in that research from the very beginning. Very often they are in the observation room and they’re coming on home visits. They’re participating in feedback on interview guides and surveys, all that stuff. But then I think they really kick into gear at that point where the implications for design kicks in from the study and we are starting to do sketches and ideation and visioning work and I’m a huge advocate of low fidelity prototype testing as a way of sort of pointing the ship. So at that point the researcher might be doing sort of a rolling study where at first mocks maybe Framer prototype, sort of golden path, Wizard of Oz-y things that are a little bit more rich and interactive. Doing repeated user studies with those. And then at a certain point in our cycle we’re sort of iterating and we end up passing off to engineering. Again, like the engineers are often there through the whole process, but then the sort of hot and heavy engineering kicks in. At that point there might be some more design refinement that can go on, or if there’s a concurrent project that the researcher is working on. But we get towards a stage where we have a functional prototype we can test we tweak that before launch. We launch. Now we can talk to real users in the wild and we get all that great ecological validity and then we’re back to square one in a way which is okay. Maybe we worked on a thing, we improved it, but a whole new set of problems reveal themselves. Like no design solution is perfect and so the researcher’s job is never done.
Steve: That’s awesome. I feel like applauding that narrative. So everybody’s busy doing this stuff. Again, I guess that’s the embedded model kind of brought to fruition. In a service model you probably wouldn’t either have the resources or wouldn’t choose to do it that way.
Judd: Yeah, I mean – so here’s – like the way that I think about that trade-off is that – the greatest thing about a service model that I can see is that you can really provide the person with the perfect expertise for a product – for a research project that’s being requested. We have to have a model that’s a little bit more like a generalist. So people on my team are generally expert at something. So maybe they’re just more passionate and experienced with diary studies, or surveys. They do little stats. You know we’re a very multi-method team. But they kind of have to be a generalist because whatever – if you’re embedded, whatever research need is kind of thrown at you, you need to handle it. You need to handle it. You need to collaborate with another researcher who’s an expert. If it’s sort of at the edge of your skill set you need to work with the data scientist. But it demands this generalist model.
And then the other thing that I think is important is that in the service model – like I – we don’t spend a lot – like the amount of time we spend on reporting varies a lot, but it is generally not a huge proportion of their time. I think of a huge amount of time spent on beautiful Keynote or PowerPoint decks as the price you have to pay for a service organization rather than an embedded organization because when a researcher is embedded presumably that PM, that designer, the engineer, they were along for the whole thing. They’ve seen it at every stage. They were in the back room and you debriefed after every session or at the end of every day. They participated in the affinity diagramming and so we may still do a video reel, cut together some clips. We may put together a Keynote, but that’s not the primary deliverable. They’ve been getting it the whole time. It was a constant back and forth. I just think that’s so much more efficient and it lets a researcher develop like real product expertise in that area rather than parachuting in and out all the time.
Steve: Are there ways that – as you said they have to work as generalists. But you have diverse staffing, people with different backgrounds and different experiences. Is there any sense of kind of community practice that you get very specifically in that centralized model? Like the researchers sit together and they share stuff, but what’s the gestalt you can create in this model?
Judd: I think we have to work just a little bit harder for it because the peripheral awareness isn’t necessarily there, the way that it would be if you had one team and sort of everybody knew who was allocated to which project at which time. And honestly like we don’t have a perfect solution to that yet. I’ve never been a part of a research team where we didn’t have a problem with that type of information sharing and it’s really important and we’re trying to do better at it. Actually we’re talking about it right now because number one we want that community of practice. We want feedback to flow. We want people to find beautiful, like synergy of research questions. We don’t want to have people doing the same project in parallel or reinventing the wheel. And all those things require that kind of peripheral awareness of what everybody is working on, or the ability to get that information quickly. On the other hand you have this sort of classic, collective action problem where if it’s too onerous for any one person to provide that information, or consume it for that matter, then the whole model falls apart because it’s sort of only as strong as its weakest link. So we have to figure out how to create a lightweight model. And to be honest the way we’re doing that right now is with Trello. I don’t know if you ever use Trello? It’s a pretty low tech tool in the sense that it’s like the closest analogy – Alex Schleifer, our VP of Design and my boss, he calls it like a digital whiteboard. It’s got columns and cards and columns. So a product team has a column and a card is an active project. And it might have a link to a research brief or an interview guide, but it’s just that simple. And that’s it. And people can check that to find out what’s going on. They can update it. They can move a card into the done pile. I don’t think it’s a perfect solution, but you call that a real problem that we work hard at.
Steve: So everyone can see what’s being researched? Or see what has been? You can look in the done pile and see what’s been done?
Judd: Yeah. So we’re still working this out, but the model is that things that are recently completed stay in the done pile. Eventually they kind of get archived and they’ll live in kind of a research repository elsewhere. But things that are active or recently completed, it’s sort of a one stop shop to view them all. We’re going to try it out and see how it works. Like I don’t think there’s a perfect solution for this, but it’s something that I hear from the team that we need to work hard at. Partly because we find those collaborations and cross-pollinations and partly because it helps us feel like a team and like we have that community of practice and that’s something that we work on actively promoting.
Steve: That sounds like another sort of – that’s the tension of the distributed – I’m sorry, the embedded vs. the centralized model. Not that the centralized model has it figured out because you know creating deliverables – I don’t know how many times I’ve encountered the like – I’ve had – I’ve been the vendor outside an organization and had people in that organization come into a new role and not know of the project and have to reach out outside the organizational walls to get that collateral or those deliverables from a vendor from a year ago, me. So it’s not like if you – it’s not like the centralized model always works for that. The deliverables just have a shelf life and they disappear.
Judd: For sure.
Steve: So is this knowledge management we’re kind of talking about here?
Judd: Yeah. You have somebody with institutional memory and they’re always there. Although of course moving from team to team to sort of cross-pollinate is a good thing, but we probably do that on the order of like 18 months to 2 years. I think everything is a trade-off. I choose to optimize for perfectly positioning a researcher for impact and to be a voice, a constant voice in the product process. The trade-off of that decision is that we have to work harder at the community of practice bit. Thankfully we have a really passionate and collaborative group of researchers at Airbnb who are devoted to that, actively seeking that and we’re small enough that we can have – you know small enough physically in that we can all sort of see each other in the space of 30 seconds and that we can meet in a reasonable sized room and share ideas on a regular basis.
Steve: Are there – do you have regular meet-ups or certain types of activities that are meant to make that happen?
Judd: Sure. You know we are in the midst of a concerted effort for information sharing. So we meet all the time. We meet every Monday morning for a standup which is sort of go around the table and everybody say the two things they’re working on that week. We meet every Tuesday for a kind of what I could call more like a team meeting. Like we get into it. We talk about recruiting and hiring. We often have special topics. We talk about growth. We have – growth in the sense of team growth and individual growth. We talk about sort of big picture product and design context stuff that’s going on. We also have something that Kathy Lee (one of the researchers on the team) put together which is Shop Talk. So it’s like a Friday event in which people can have a beer and give a presentation they would have given to their product team to the other researchers. And we’re also instituting skillshares. I sort of demand that everybody on the team – so my model for this is sort of, in my memory as somebody who got a PhD – so when we get a PhD you become the world’s foremost expert at some tiny little thing that nobody cares about and then you write a dissertation which if you’re lucky your advisor will read and then you’re kind of done with it. But, so clearly that’s why I’m not in academia, but one of the great things about that model I think is that you carve out this niche for yourself which is your calling card and that’s a thing that I ask every researcher on my team to do. Yes there’s this baseline that we must be rigorous, beautiful generalists, but what is the thing that you’re uniquely good at as a researcher? What is your calling card? What is the thing that you want to at first sort of really consume all the materials you can find about, go to seminars, take workshops and read books on and then before you know it you’re writing the books and you’re teaching the seminars and you’re giving brown bags to the rest of the team. I just think if you have a team where everybody has that unique team, an aspect of methodology or practice, communication or whatever it is, that they are passionate about developing into a unique skill and they’re all committed to sharing it with each other, that’s my model of a team that grows individually and grows as a team. Like that sounds awesome to me and we’re work- you know I don’t think that we are perfect in that regard, but we have that as an aspiration.
Steve: What are some types of things on your team – topics for different researchers, that’s there thing?
Judd: Sure. There are methodological topics – just as simple as survey best practices. So we have somebody on the team, Janna Bray, who has made it sort of a thing she’s really vocal and generous about to raise everybody’s game at surveys. We have another researcher, Steffenn Kuhr who is really passionate about rigorous evaluative research in which you are trying to bring as much objectivity to the sort of usability and user testing process as you can and he has some really interesting ideas about how to do that that he’s honed over his career. There are also folks who want to bring aspects that are a little further from sort of core research methodology as their special sauce. So for example, Natalie Tulsiani is a researcher on the team who recently gave a workshop on moderating the observer room. So the topic is as a researcher moderating the group of people who are observing a study. That just kills me. I’m like that is so great. You know what I mean. I worked with a researcher at Facebook who her thing was improv comedy. You know she thought improv could make you a better qualitative researcher and I believe she’s right. So that was her thing. She was passionate about that. I think it could be all over the map, but I just love the idea that everybody on the team has their thing and they’re just chasing it and bringing everyone along with them.
Steve: Those are great. Those are good examples. So the idea of the individual passion is a driver here.
Judd: Yeah, I mean look, the research shows just in general that when somebody stops growing in their job and they feel like they’re in a rut they’re going to leave. So just from a purely practical point of view, it’s like in my interest as somebody trying to build a team to keep everybody growing around things they’re passionate about. But I also think that’s the shortest path to everybody building everybody else’s game to be stronger. Everybody raising everybody else’s game. We will – that is the path to becoming a world class research team and I think it’s both demanding of an individual to always be raising their game and demanding of that person that they be generous and collaborative with the researchers around them to do the same.
Steve: Right. So as a member of that team I am getting energy and reward from being able to follow that passion, but also I’m receiving from others.
Steve: From them, so I’m always growing and developing as part of that team.
Judd: And if you do that I think developing that community of practice is easier. You know it’s easy to see where it comes from and how it grows.
Steve: So you’ve talked about growing the team. Can you go back in time and describe a bit of the history, the evolution? Where you’ve come from? Where you’re looking to go?
Judd: Sure. So my history at Airbnb is admittedly a little bit short. I’ve been here since the middle of May, so what amounts to about 6 months. I was the 10th researcher. As of this week we have 17. So we’re growing quickly, but we’re not growing for growth sake we’re growing because – I think because the embedded model we have means that awareness of research is high, the team is producing great work, and it’s just I’m getting more demands than I can reasonably service right now. So I need more researchers and that’s a beautiful problem to have.
Steve: So there are teams that don’t have an embedded researcher that…
Judd: That’s right.
Steve: That team needs somebody.
Judd: That’s right and we’re in this moment of accelerated hiring which is necessary in my mind because there are enough teams that have no point of contact, that getting – I mean going from zero to one is sort of technically an infinite improvement, right. And in the research that we do it is really dramatic at that moment. And I think this is – so we’re trying to take advantage of that inflection point. A lot of research teams in industry go on a ratio, right. So a ratio of designers to engineers, researchers to designers, for example. So they might say like 3 to 1 is the right ratio. One researcher to three designers, or one researcher to three designers and PMs together. And what we’re trying to do is just get ahead of that and then back off. And the reason that we’re trying to get ahead of it right now is because we’ve – I think we’re at that moment where having a research voice in all of these teams that have sprung up and don’t have any research representation is going to be disproportionately valuable. So we’re trying to concentrate growth right at this moment and then we’re going to slow down.
Steve: So there’s a context here of sort of shift in product development – let me see if I can ask this properly. You’re describing the teams are springing up so that says that what the product is is evolving and growing – do you call the thing a product – what you as a company provide and the different pieces you’re creating to do that…
Steve: …is evolving and growing so there’s a thing that’s being made that maybe wasn’t being made at some point and that’s a new team?
Judd: Right. Or you know – and I don’t – I’m not an expert in the growth of companies or product teams of course, but in this one, you know I think as the product organization matures there are two things that happen. One, new products spring up. The other, existing products, the pie grows, right, so that the amount of work is – becomes specialized enough to carve out a new team. It’s just too much. So where there was just one team before that for example looked after all the things we do around host dashboards and host tools, for example, well now we’ve a reached a scale and a complexity of the product such that there are several important work streams. And then they break them out and PMs and designers and researchers and data scientists specialize in those areas. And so I think – that’s a thing that’s kind of been happening, like spawning teams at Airbnb, and in that moment having a researcher at the table is super crucial. We don’t want to lose that momentum and there’s enough important work there – a lot of those people have come from other teams where they had a researcher and they’re like hey I’m working on this new thing, where’s my researcher? I love that. I love getting those requests. Why isn’t there a researcher in the room for this. I means I have to say no more than I want to, but I’ll take that problem any day.
Steve: Are you trying to hire with specific team roles in mind?
Judd: Mostly no. Mostly what I want to hire for is talented generalists and that’s because of what I think of as the virtue of that embedded model and because I don’t want to create – well I don’t want to create an environment in which there’s a bunch of people who are really only either happy or well suited for one type of work because stuff changes. The product moves. The demands of research shift such that okay I need to move some researchers around to tackle this new product area and I don’t in general want to have researchers who I think I’m going to make dissatisfied or their skills aren’t going to be a good fit if they need to slot into a new areas in 6 months or a year.
Steve: So you’re using the demand to figure out how big your team needs to be, but it’s not sort of slot filling, it’s capacity building.
Judd: Yeah. Overall capacity building. I think if we chase anything at all as a team we chase diversity. Like I want a team full of people with different methods, different perspectives, different backgrounds. We just hired this talented researcher named Andrew Sweeney who is going to be our first researcher in Portland. Portland is where a lot of our Customer Experience team is and so Customer Experience, those are the people who answer the phones. If you have a problem before or during your trip that’s what they’re for and they have a whole suite of internal tools that they use to handle these problems, to track information about cases that are open. Those tools are not as good as they could be and we have never done systematic research on those tools. So in comes Andy Sweeney who has – Andrew has a background working on complex, information rich tools, right. So his approach to the problem is different than the one that I would take. He goes to system usability scale for example which is not the first place that I go, but that’s brilliant for this situation because we’re talking about understanding information flows in a complex information environment. Efficiency is important because we want to enable Customer Experience agents to be going as fast as they can and not blocked by the fact that they have the information tool, like a CRM, and three spreadsheets and a notepad and their workflow is just broken. So chasing that diversity as a team is something that I think goes to the earlier point about raising everybody’s game where everybody is helping each other out. I want to learn about system usability from Andrew.
Steve: And so Andrew’s – he’s tackling a new problem space. That’s not hosts or guests? That’s the new organization and being able to deliver a good experience.
Judd: That’s right and I think he – so in a way we’re taking on a new sort of constituency which – as a research team – which is our internal CX agents who we call crewbies. So we have hosts and guests and crewbies.
Steve: Say the word again?
Judd: Yeah, crewbie. People who work in CX, manning the phones and doing Chat and email. They call themselves crewbies. But I mean they’re the most talented, empathetic people on the planet. Like we – many people who work here do these shadowing sessions where we’ll sit with crewbies and sort of just observe their day to day and we do that to gain empathy for the problems that guests and hosts have during and before trips and after trips. But these are some of the most patient, amazing people on the planet and we’re, you know, in the process of working towards better tools for them.
Steve: Okay. So that sounds like a marked evolution in what – in how the work that your team is doing is impacting the company.
Judd: Yes, it’s a more internal focus than we’ve had before. I think it’s fundamentally the same sorts of things which is like do rigorous research that is deeply understanding the problems and needs that people have and understanding potential solutions, but it’s in the context of making our business more efficient and providing better service.
Steve: So will Andy have access to the designers and the engineers that are creating those tools for the crewbies?
Judd: Oh yeah, there’s an entire products team that’s spun up in Portland and in San Francisco just to work on this.
Steve: So once you have a product team then that’s – in your embedded model – that’s when the request for the researcher comes?
Judd: That’s exactly it. And I think I look at a lot of things that way. So I could have hired Andrew 6 months ago when there was no product team, but the reason I didn’t is because he would have had nowhere to land. I think that’s setting a researcher up for failure. You know I want to know – you know a researcher can do the best work in the world, but if there’s no landing zone for that research then they’re not going to be able to have a lot of impact most of the time.
Steve: So at the risk of re-asking a question I brought up earlier, we were talking about slot filling or not, is Andrew someone where you saw his particular – what made him a diverse candidate? Can an individual be diverse? I don’t know. What was unique about him that contributed to diversity? It sounds like the thing that he was brought in to do is perfect. For the company, perfect for him. Six months ago were you saying that the right project for him wasn’t there? We talked before about capacity building versus slot filling, but I’m wondering is there sort of a nuance to that that goes with the diversity aspect?
Judd: Yeah, I mean – I guess I would be lying if I said that there was never an appropriate time to look for a specific skill set and that it was always just general capacity building, looking for great researchers. This is a case where absolutely I thought I’m sure we can find an incredibly talented researcher who has experience working with information rich internal tools and making them work better and understanding complex workflows and efficiency and all this stuff. I’m sure we can find that person and low and behold we did. At the same time I think, you know Andrew has done a bunch of things in his career, not to emb- I’m sure he’s going to be embarrassed about this and – and he just started on Monday by the way, but – he’s done a lot of things in his career and he would be perfectly capable of filling almost any other slot. He just happens to be a great fit for this role.
Steve: Let’s go back in time. As you said you’ve been here “x” amount of time, but you must have some sense of the history. You were #10 you said.
Steve: Do you know the story of #1? Like how did research become a thing that was hired for and done?
Judd: Well research is in the lore of Airbnb. One of the stories that gets repeated internally is about one of Airbnb’s early investors who at a very early point in the company turned to Joe and Brian (Brian Chesky and Joe Gebbia, two of our founders) and he famously said go to your users. And so they spent several weeks in New York City with some of the first hosts on the platform, deeply understanding what they were going through and what their experiences was like. And Joe and Brian are designers by training. They were at RISD and I think that having designer founders – having you know two of our founders be designers has sort of infused this kind of design and research sensibility into the whole thing . And that’s been beautiful for us as a team because I don’t – let’s say to the degree anyone cares about buy-in from the top, which I think realistically every team needs to care about, relatively speaking we do not have to worry about that. We have people like – we have a whole bunch of leaders saying where’s the research. Let’s look at the research. Go to your users. So it’s really embedded into the lore of the company. I wish I could be more specific about the history of this team in particular, but honestly I can’t.
Steve: So with that caveat I’m going to ask another specific question which you can then throw back and say you can’t answer. Especially for this podcast, I think an implicit topic – I think it’s implicit – is this idea of research leadership. And the trajectory often in organizations is a researcher is brought in. Sometimes they’re a junior person and they kind of work for a UXer design person, depending on the industry. And then some organizations there’s a point at which it’s acknowledged that this is a specific thing and it needs to be led, not by someone who’s a designer, but by someone who that’s what they do. That’s my sort of general sense of the pattern and I wonder, do you know when did that happen at Airbnb? When was research a leadership – when was there leadership in research?
Judd: Yes I do.
Steve: See! You said you couldn’t answer it.
Judd: I can answer a few questions. So in the beginning of the team it was just one or two researchers and obviously the design team was much smaller and it was just one big team. Everybody reported to Joe Gebbia. And over time what happened is the design team grew. I think the research team, which was at that time called Insights, was not growing at the same rate, and partly because I think of a – the fact that it’s hard for somebody who’s not a researcher to know how really best to leverage and grow that organization. But the partnership was there. They were doing great work and then at a certain point Alex Schleifer showed up. Alex is our VP of Design. He’s been at Airbnb for about a year and a half and he took over leadership of the organization and almost immediately recognized the need for research leadership. And it took him awhile – it took me awhile to meet him and I think to his credit he gives me a lot of freedom and says like Judd I want you to build a world class organization and I want to support you in that, what do you need? And he gives me a huge amount of context and feedback, but he’s set me up with his team such that my peer now is Katie Dill who runs sort of the organization of designers. Adrian Cleave who runs our design operations organization – but we are Alex’s first team and he treats us like peers and so there is no sense that even though I’m in the design organization I don’t have the sense of – that some – of research being – the sole value of research being to feed design for example. I feel genuinely like we are equal partners in a product process and I give Joe a lot of credit and Alex a lot of credit for recognizing that that was a useful and valuable thing to do with research from a pretty early time.
Steve: So the position that you came in to take was a new position?
Judd: It was a new position which was to lead the research organization. Yeah. And I was and have been given like a fair amount of leeway to say how should we do this. And that feels great. I don’t have the answers. You know when he said that I was like great I don’t know how to do that, but I’d love to work with you and figure it out.
Steve: That seems really key and I don’t know if that’s just part of the overall culture or something that is unique to you, but – and I’ve had this conversation with people a number of times about knowing the solution versus knowing how to get to the solution and the comfort with the ambiguity. You’ve mentioned a number of times in this conversation that we don’t have this figured out. We don’t have the best solution but we’re trying this.
Judd: Yeah. I think anybody – you know it’s interesting because we were recently having some conversations about the structure of product teams in general and how hard that is and going why hasn’t this been figured out by now? Shouldn’t there be one canonical way to set up a product team? No. There is no one canonical way because every business is different, every product is different. Values are different and that implies different things for structure. Well I think it’s the same thing for research, for design, for product overall. I feel everything that I do is to make good principal decisions in response to the realities of the situation at Airbnb. To try to build a research team which is uniquely responsive to Airbnb and at the same time embody some qualities that we really care about – world class rigorous research, perfectly positioned researchers every time. Those are kind of our two mantras. Center of excellence, perfectly positioned. And those are sort of – there’s no playbook for that. I think we apply those principals. We apply them systematically and we communicate a lot. And I don’t – it sounds stupid when I say it out loud, but I don’t know of another way. I think I’ve been really, really hard with my team and they are embracing of the idea that feedback is the most important thing we do. Feedback for each other, feedback for me, feedback for me with them – because I say to them I don’t know of another way to do things better than to give and get feedback. It should flow like a river. And I think that can be hard, to be open, to focus on strengths rather than weaknesses. To be humble is a thing we’re always seeking to be better at, but that’s how I approach the task of building a team. I welcome any and all feedback. I don’t pretend to have all the answers. I just think I have a principled way to get there.
Steve: I’m going to ask sort of a clichéd research question but I think it follows from what you’re saying. In a purely speculative way, like if we were to be talking in five years let’s say, what do you think – what could you imagine the state of research at Airbnb being?
Judd: Five years is a long time. Well, hopefully we will have scaled to the point where we have what we would consider full research coverage. To me that means being sort of lean and agile, but having a researcher represented kind of at every level, from the ground level up to the leadership level and we’re definitely not there yet. It takes time to build that capacity. You want to build it from within too so we’re growing that capacity at the same time. I also think in 5 years we’re going to be a company that has already hopefully long since already embraced the idea of like a really global/local product so that we have product teams that are staffed all over the world and I have researchers embedded in those teams and they are connected back to San Francisco and they are providing this very difficult thing which is how do you create a product which is sort of you know 80 or 90% core, but that 5% or 10% on either edge is the real local bit where the rubber meets the road. So how do we create a uniquely Singapore product or a Germany or Brazil product where we represent belonging in a way that resonates culturally where we have tweaked our onboarding process to highlight different value propositions. Where we do special things that facilitate trust between guests and hosts that are unique to an Indian market for example. And I think that’s going to take an organization which is global. Like I don’t think that sending researchers from here out to those places is going to cover that. So that I hope sooner than 5 years will be true. And then the last thing I’d say is that I hope that by that time we will have created not just a product organization, but an entire company which is totally driven by research and insights in which – you know I don’t think that we need to do all the work, but whether we’re talking about marketing or legal or policy or local operations or customer experience there are research questions in all of those places and I want us to be able to have an organization where somebody on my team is the glue to every one of those teams.
Steve: So taking embedded beyond product?
Judd: For sure. Because I mean – for example name a – it’s a little bit stupid to me that most organizations have a product research and a market research organization. Name a product question which doesn’t have an influence on marketing and advertising and vice versa. I’m sure somebody could snark and think of an example, but that would be the exception that proves the rule, you know. I’m not saying – there’s no need to have one big kumbaya team, but there is a huge amount of value to saying like look fundamentally we’re just researchers applying our expertise in different disciplines and in different ways, let’s be more together than not. And in my experience most organizations are not like that. So I don’t know. That is a big aspirational goal for this company.
Steve: I love it. Just so well put. So that’s kind of the looking ahead. It’s a lovely vision. I want to go back. You mentioned a few things about yourself – PhD and some other places that you’ve worked. Maybe you could just talk about – I’m just curious about sort of parts of your background, whether it’s professional, personal.
Steve: Go back and talk about some of the things – what are some of the things that are in your background, experiences or education or whatever, that are really present for you now in the way you’re looking at the world and the kinds of things you’re trying to make happen?
Judd: Yeah, I mean, I guess if I – that’s a really – you’re forcing me to be introspective. Well I began – as an undergraduate my major was cultural anthropology and so I began my career as an anthropologist. You know very focused on meaning and understanding and writing culture and the idea of culture and what it meant. And I feel like that’s really important to me now, especially at a company that’s focused on travel. It gives me a lot of empathy. You know I feel like I studied a lot of kind of epistemology that gave me like a fundamentally subjectivist approach to research which I think makes it valuable – makes me a better multi-method researcher and leader in the sense that I think objectivity is a myth. That I think everything has a cultural and social lens and all we’re doing is like seeking confidence and that confidence is built through multi-method research, through looking at the same problems from multiple directions and perspectives. So cool, if you do that there’s no territoriality. We just need all the methods. They’re all flawed and they’re all powerful. So I think that was influential for me.
My PhD was in social psychology and information systems so I took a real jag into experimental social psychology and data science. That I think informs me because I, you know, have had now very deep experience with both the most ethnographic qualitative work there is, you know in which I spent six months at an after school arts and media program in the Bayview learning about informal learning outside the classroom. And I’ve done a huge amount of experimental social psychology and data science and I can appreciate – you know I can hold my own with basic statistics and write R and Python and PHP and talk to an engineer in code and stuff like that. So I think having had that experience is deeply influential for me.
And the last thing I would say is that I think – so after I graduated from undergraduate I went to culinary school. So I spent six months getting a degree at the French Culinary Institute in Manhattan and then I worked as a chef for awhile. I actually – I was not destined to work as a chef. That was an extremely difficult life and I found that out and was grateful to have kind of something to go back to which was anthropology, and graduate school. But having been a chef, having been somebody who worked with my hands and really embraced that kind of creativity and appreciated flavor and the craft of perfect taillage, like knife skills. I think that helps me have like a little bit of a window into a design world. I don’t think that I have a lot of a window compared to the amazing people I work with on the design team, but I think it gives me an appreciation for craft, for art, for the details, for the menial but beautiful handwork that makes great food for example. I think I can apply that to research and to design.
Steve: That’s awesome. You know we talk about researchers as kind of translators and even this – the diagram you drew of somebody elbowing somebody they’re sitting next to. It’s kind of like a translation thing and it seems like the experiences that you’ve had have given you a lot of vocabularies to translate between and maybe even the translation was sort of an ingredient in those.
Judd: I agree with that. I think – you know recently I gave a talk – I’m not sure I should say this, but I’ll say it anyway. Recently I gave a talk at a learning lunch here at Airbnb, mostly to a data science crowd, and the talk – this is also a talk that I gave at a design conference in Philadelphia. And it was basically a social psychology talk. It was about the ways that we are all biased. It was about confirmation bias and minimal group bias. It was about post hoc bias. And like the point I was making is research is human and research requires human – humans. Every type of research does and humans are flawed. We cannot avoid our biases. The best we can hope to do is counteract them. By hanging out with each other. By trying multiple things. And that talk got me into a little bit of trouble only because I think it seemed like what I was doing was being unduly negative about different methods, and I was. It was kind of one of those talks where you crap on everything and that was my point. It was like okay everybody spends time exulting the virtues of their particular method and that’s great. They’re all strong, but they are all weak and we are weak. And so I think part of the translation for me is to be able to speak to everybody and go okay I get the strengths of your method and I get its weaknesses and this one too and this one too. So let’s just get past that and work together.
Steve: And that’s being human.
Judd: How can you avoid being human?
Steve: Yeah. Is there anything we didn’t talk about in this conversation we should cover?
Judd: I guess one of the things that is on my mind a lot is, because I’m growing a research team, is the idea of responsible growth for a research team. And what does that mean? I’ve been thinking about it because we are growing and I want to make sure that we do it in the right way. I’ve experienced both types of growth, responsible and irresponsible. And the way I think growth gets irresponsible is not even really in a headcount way or a budget way. It’s probably more like a communication and culture way. So the things that I want to promote in the content of responsibly growing this team at Airbnb is radical transparency and communication. The worst outcome in the world is if new researchers look at old researchers look at managers look at me and go I have no idea what these people are thinking. Or I don’t know how to plug into this decision making process and I don’t feel like I have a say. How did this get to be this way? I don’t think it’s right but I don’t have a voice. Okay, terrible, terrible. Need to avoid that at all costs. So the way we do that is by being fundamentally transparent and collaborative. Like everybody from me on down, we talk about what we’re thinking and why we make the decisions we make. We open them to feedback and maybe we form a committee in which everybody has a voice. Okay, committees, that sounds kind of bureaucratic, but what I mean is like involve everybody from the intern to the senior manager in doing something like how should we build a skillsharing system? Cool, we can all be involved in that.
And then the other thing I think is responsible growth is making sure that everybody has a path forward in their career because other than feeling like you’re stuck in a rut, the other thing that I think is a recipe for a researcher being dissatisfied is feeling like they have nowhere to grow. And so making sure that everybody feels like they have a path is really important to me as the team grows. That path is building out your unique niche, working on your core skills and expanding them, taking on more stakeholders, taking on more senior stakeholders. And you do not have to become a people manager to be a lead. That is another thing I feel like is really important because people management is a set of skills that is unique and crazy difficult and learned. And you choose to focus on them. Not everybody wants to do that. It is absolutely not the case in this organization that the only path forward toward seniority as a researcher is through people management. It can be through research leadership, through product leadership, not just people leadership. And so as we grow I think the responsible bit is making sure that all of those paths for growth are open to everybody and everybody knows about them.
I spend a lot of time thinking about that because it’s scary to think that we were 10 when I started, we’re 17 now. I don’t know, we’ll be 25 or 30 in the next year or year and a half. That’s a lot of growth and everybody thinks the next person in the door is going to be the one that changes the culture, but I don’t think that. But I think we should be deliberate. It should happen on purpose not by accident. That kind of growth, the planning around it. So that’s what I seek by doing it this way.
Steve: That’s fabulous. Do you have any questions for me?
Judd: Why are you doing this podcast?
Steve: Why am I doing this podcast? You know it’s self-serving I think. I’ve been around long enough that the best work was being done by vendors. Let’s just say that. And I still am one. I hate the word vendor…
Judd: The “v” word.
Steve: …we all know what that means. You know people who work outside organizations. Teams like – companies like this didn’t exist. Teams like this didn’t exist. Leadership roles like yours didn’t exist. It’s a big change that’s happened in the last few years where now – I mean the kinds of vision for what research can be and how it can impact. You know it has to be done inside. It’s not to say that – I’m not saying that my work doesn’t have value, it’s just different. If you work with organizations the context has shifted. There are people inside organizations that have roles and titles and responsibilities that didn’t exist before. So my professional life has changed. So it’s self-serving because this is a chance for me to learn about this shift. It’s fun to be able to do that. It gives me an excuse to have conversations with you and learn things I wouldn’t otherwise learn. And you know I think much of the best work – whatever the percentage is – amazing things are happening inside organizations. So that’s the place to kind of look and learn. I’m not – I like working outside organizations so when you’re a consultant or a vendor you journey from place to place, like in The Incredible Hulk. You know you have these adventures and then you have to leave at the end which is an obscure reference for people that didn’t watch the T.V. show in the 80s or 70s or whenever that was.
Judd: I didn’t get it.
Judd: I liked it though. It’s a good reminder of how young this field really is.
Steve: Especially in the form that it’s in. The conversations we’re having about change in design, insourcing and acquisitions and so on. I feel like research follows design and kind of we’re trailing by a couple of years. But we’ve seen research firms get bought too, not just design firms, you know in our recent history. We’ve seen that happen. So I’m curious.
Judd: As much there is no one canonical way to build a research organization there’s also no information out there about how to do that. And so to have a resource where you can learn from people who are trying to build those organizations is really valuable to the UX community so thanks for doing it.
Steve: Alright. Well it was great to speak with you. Thanks so much for being a guest and for being my host here today. I’m throwing those words out in a really confusing way now.
Judd: Nicely done. Thank you, Steve. It was a pleasure.