8. Aviva Rosenstein of DocuSign
In today’s episode I speak with Aviva Rosenstein, the Senior Manager of User Experience Research at DocuSign. We explore how to make all types of research actionable, the benefit of doing your own recruiting, and the evolution from building a usability lab to having an in-house research capability.
They weren’t going to limit themselves to telling us about what was going on with sending. And I’m not going to shut those people down. I’m going to listen to what they have to say because it might be something amazing or new or super important. And of course it’s important to the customer so you don’t want to shut them down. So sometimes the scope of the study can get very quickly past where you wanted it to be, but it’s still going to be valuable so you still listen. – Aviva Rosenstein
- Aviva Rosenstein
- Follow Aviva on Twitter
- Snowball sampling
- DocuSign University
- Sean McLeary
- Practicing What We Preach: designing usage centered deliverables
- UX Process Improved: Integrating User Insight
- Advanced Communication Technology Lab (ACT Lab)
- Allucquére Rosanne Stone
- Radio Television Film Program
- RE:DESIGN/UXD 2015
- Erin Malone
Steve Portigal: Well thank you. So let’s start broad and big as we always do and have you tell us about you, your role, your organization.
Aviva Rosenstein: My name is Aviva Rosenstein. I am the head of user research at DocuSign and I’ve been here for two years. DocuSign is a rapidly growing pre-IPO startup that’s been around for over 10 years. So it’s different on a lot of ways from your usual startup. We are interested in making signing documents and digital transactions easier for everybody from consumers to large enterprises. We are very concerned with making sure that we’re trustworthy, that you can trust your digital documents with us, and so we care a lot about security. And we are more than just signing. We are looking at all kinds of ways of making that digital transaction easier. So what people think about us is oh you’re just an e-signature, but we’re actually a lot more than that.
Steve: Was signing what the company started off as?
Aviva: We started off as e-sign, that’s absolutely right. Electronic signature. We actually started off with a lot of real estate agents who, if you’ve ever bought a house, you know how many papers that you have to sign as part of that process and so that was a really easy set of use cases to start out with of a very motivated group of users who wanted to make that easier on their customers. But now we are doing e-sign in the whole gambit of industries and we’re moving into digital signature in other parts of the world that have different laws and policies related to electronic signatures. So we are expanding our technology to accommodate those other countries.
Steve: Is e-sign and digital signature, are those different technologies or standards?
Aviva: They are different standards, that’s correct. And different technologies.
Steve: Are those terms that are created elsewhere? I’m trying to sort of unpack the language.
Aviva: They are created by their standards bodies although DocuSign has been active in those standards bodies, at least with the development of e-signature and getting e-signature accepted in North America and other commonwealth countries. Digital certificates and digital signatures are something that came out of the European market and South American market and other markets. So we are accommodating to those standards as we move into those markets.
Steve: In some ways it just seems so – I don’t know, Silicon Valley, 2016 – sort of hilarious, silly and awesome that there’s a business whose legacy is about signing stuff.
Aviva: It’s less – I think it’s really awesome. If you think about how much paper had to move physically from place to place to place using things like FedEx or UPS and people had to sign – let’s say you got a job offer. You had to wait for the job offer to get mailed to you, or delivered to you on a truck, with a dead tree. And now none of that is necessary. And then there was – in between that there were faxes. So if a fax came to you, you still had to print it out, sign it and then fax it back or mail it back. With e-signature there’s none of that dead tree. There’s none of that carbon footprint. So we’re actually doing a lot for the environment here.
Steve: I mean your example of the real estate one rings true to me. I mean every time I’ve signed or refinanced it’s easily – I mean I’m not being hyperbolic, it’s an inch thick of paper that you’re just signing and signing and signing and signing and signing.
Aviva: So here you put your signature in once and you agree that that’s your signature and that’s going to represent you. So there’s that handshake that you have with the technology and then it’s a click. It says sign here and you click on that and your signature appears where you just clicked. So that’s our standard way of doing things in e-signature. It’s a little bit different than the other technologies.
Steve: When I think about – yeah, just sort of reflecting on – it is starting to become a more ubiquitous kind of thing. I bought something I think with Square yesterday and I signed with my finger and I didn’t think about it. And I got an email. And I walked into your building today and did the normal sign-in with reception – fill out a form, it prints out a badge and there was a nice DocuSign interface that I go through to kind of sign in here and it makes a lot more sense.
Aviva: And it’s completely auditable. So we know where you were physically and what time you signed that. So if that stuff – if there’s a court case, if somebody challenges and says that’s not my real signature and tries to defraud a company, the company who is our customer has recourse to look at the information about when that signing even happened and try to connect it up with the person who signed. So it’s a lot safer and more secure than a signature with a pen on paper because anyone can forge that.
Steve: So as many things have moved in our recent lifetimes from analog versions to digital versions there’s all these things that you get and one of them is being able to go back and then trace them to audit them.
Steve: That helps me to think about this category as sort of an old school technology becoming a modern technology and what that affords.
Aviva: Like so many technologies they start out as being – you know just replacing a single medium like – oh it’s like signing something with a pen only digital. But then as you keep looking at it there’s more and more opportunities to build out on that from and to pass that information into other systems. And that’s where things get really interesting because we have a set of APIs that our customers can use to integrate our signing technology into their workflow.
Steve: Can you say more about what an API is and how someone else might use that?
Aviva: APIs are the building blocks for making an application. So it’s a way of determining, in our technology, what inputs somebody is looking for and what the system does in response to that and capturing it in such a way that somebody else can use it to build an application
Steve: So the point is though that other people can use sort of a core part of the technology without having to have DocuSign write a piece of software.
Steve: If you wanted to say have a point-of-sale that people were going to sign with their finger you could use all the technology that DocuSign has and get all that auditing and all the security and all the greatness of the technology and then put your piece around it.
Aviva: Right. So our service you can use the way we ship it, but if you wanted to build your own system and include the e-signature component then you would plug in the API and it would take you through the DocuSign interface and allow you to sign, but it would be integrated with your technology. So there’s a lot of different ways that people are using our technology in their own products.
Steve: And how is user research part of the effort that DocuSign is making to create stuff for people, for lack of a better way of putting it?
Aviva: I think that we have an impact at every point in the product development lifecycle at this point, the two years now that I’ve been here. We started really with the basics of usability studies of making sure that what the UX team was building could be used by people before it got shipped. So we did a lot of early stage prototype usability studies just to validate things before we ever showed them to customers in a production environment. But we also have done other kinds of user research from just concept interviews, finding out what people are looking for, whether our concepts are going to work for them, what would need to change about our concepts in order to work for them in their environment. We’ve done more field, follow me to work, kinds of studies where we’ve gone in and watched people use our production technology in their environment and talk to them about what was and wasn’t working or find out – sometimes we would watch them and find out, hmm, they’re using things and not understanding something correctly and we need to make that clearer. Or they’re making a mistake and they could be doing this much more easily and so we need to make that more apparent. And then that goes back into the next round of features that are up for – possibly including the product. So we’re both informing the roadmap as well as making sure that things that are on the roadmap are getting built in a way that people can use them and are delighted to use them.
Steve: I’m curious about sort of the types of organizations and the roles within those organizations that you guys are looking at, again within what you can say.
Aviva: Well I think that’s pretty clear. The people that use our products kind of fall into specific roles. There are people who are mostly just signing stuff that other people send them. So if you were the person who is buying a house and your realtor sent you something and you were the signer – so we could talk about the signer and that experience. There’s also people who are sending documents and they may be sending a single document or they may be using a reusable template where they’re sending the same kind of document over and over and so they’re customizing it for each signer and that may be more or less automated depending on their systems and what they’re doing. So we talk about different kinds of senders. They may be working in a small business. They may be at a big call center. They may be somewhere in between. They may be out in the field as a salesperson getting somebody to sign something on an iPad. So that’s a varied experience, but we can talk about senders. And we also have admins – administrators – people who administer the product for their sets of users because like any other kind of enterprise software there’s different roles that people can play within the software and they have different kinds of access and different permission levels. So there’s an interface specifically for administrators. So we talk to them a lot because they spend a lot of time with the product and have a lot to say about how we can make it better. And then of course there’s the developers who are using our tool kits and our APIs to build their own stuff and they have their own specific needs. So those are four of the most maybe top of mind end users that I think about on a daily basis. There are other people in the whole realm of our customers that are making decisions about whether or not to purchase the product and I think about them too. But the day to day end users is really what it’s for.
Steve: What have you learned about trying to get in front of those different user types?
Aviva: So consumers are relatively easy. Signers are often consumers. And so sometimes we just use Craigslist to get those folks. Or we use other recruiting services to find people who have experienced e-signature or maybe specifically experienced our product. But we also maintain our own customer panel that people opt into. So they fill out a short survey that describes them and their working environment and we can sift through our panel to find people who meet a particular kind of target and then send them an email inviting them to take a screener for a specific study and if they’re interested then they’ll answer some more questions and we can use that to schedule. And that’s been super helpful. Growing that panel has been a tremendous asset in speeding up our research process. But sometimes we need people who aren’t in our panel yet and then we use other groups within the company to help us identify the right people to talk to and get their information so we can contact them. So for example when we recently did research where we went to some of our customer’s sites we worked very closely with our sales team to identify customers they thought would be interested in talking to us. So that was hugely helpful.
Steve: So a few different approaches.
Aviva: Whatever works to be honest.
Steve: Yes, that’s how recruiting works is whatever you can do.
Aviva: Sometimes it’s a snowball. Sometimes it’s friends and family but we try to get the best sample that we possibly can because then the research is that much more trustworthy and resonant. So we’re a little bit ADD about our sampling.
Steve: How so?
Aviva: Currently we do not have a full-time recruiter. Currently the researchers are responsible for recruiting their own studies. So that means they spend time on the phone with every participant before they arrive on-site, or remotely, to participate in a study, unless it’s a survey. But if it’s an interview or a usability test they’ve often been vetted on the phone first and that means that when the team shows up to observe a usability test, for example, or listen to an interview while it’s happening, their time isn’t being wasted because we got the wrong person in the door. And we’re not stretching the process out because we have to re-recruit. So getting the right sample is really critical.
Steve: Is there a counter side to it, maybe just in the kind of work you’re doing there isn’t, where those people that shouldn’t have – you get those people that you don’t know how they screened in and once you get over being irked about that fact you – something different can happen which sounds – it’s a nice sort of hand wavy Steve thing to say, but in the real world you’re in, is that true?
Aviva: That’s especially true when you’re doing qualitative work in the field. So for example when I went into the field last summer I was particularly interested in the sending experience and I was trying really hard just to recruit against that. But our customers had so much they wanted to tell us that they didn’t feel constrained by that. They wanted – we were on-site, they had a lot to say. They weren’t going to limit themselves to telling us about what was going on with sending. So we also heard a lot about the administrative experience and we also heard second hand a lot about the signing experience. And I’m not going to shut those people down. I’m going to listen to what they have to say because it might be something amazing or new or super important. And of course it’s important to the customer so you don’t want to shut them down. So sometimes the scope of the study can get very quickly past where you wanted it to be, but it’s still going to be valuable so you still listen.
In usability testing we try to figure out in advance like do we want people who have experience and people who don’t have experience? Or what is it, depending on the complexity of what we’re testing because we want to be able to do a fairly lean, fairly tight script. So for those kinds of studies it’s maybe less important to get in the stray, but for actual behavioral observation or opinion interviews it’s a lot more interesting sometimes to let those people in the door.
Steve: I think your point about participants that have a lot to tell you and I think given that you work for a company that has customers, a lot of your research is with those customers. They know you, they know who you are, they have a relationship with you. They use the product. They have needs out of that product. What they want out of those interviews is different than some other kind of contextual research where it’s more exploratory, it’s not branded, it’s not tied to the relationship.
Steve: So when you’re doing your work you’re representing the company.
Aviva: I’m part of – I’m representing the brand. That’s absolutely true. And I like to think that we’re part of the company’s competitive advantage because we care about listening to our customers and we’re out there doing that every day. So if they have something to tell us it doesn’t matter what channel it’s going to come in through – whether it comes in through a customer support channel or through user research conversations, but it’s going to get back to the people who need to hear that information. It’s going to get back to the product managers or whoever it is that is going to get value from that information.
Steve: Can we do like a – it’s not quite a day in the life – can you kind of give a Platonic ideal project where – you know where it’s coming from, who’s talking to the people and then where that information goes?
Aviva: There are so many different kinds of projects that that’s a little bit hard to do but I guess the Platonic ideal is probably the usability test. So let’s say we’re doing a rapid iterative study with a functional prototype of a product that’s not been released yet. So the people who are involved are primarily the design – the designers that working on it and the product owner, however that’s defined – it’s usually a product manager here who is responsible, ultimately, for that feature or for that product. So we will do a round of testing – it might be 5, 6, 7, 8, depending on a lot of different factors – a lot of different sessions with time in between each session to debrief and decide what did we see? Did we all agree on something that went wrong? Do we all agree on why it went wrong? Okay let’s make that change in the prototype right away if we can and then bring the next person in to validate that. So at the end of a day or two we’ve moved the functional prototype in a different design direction, frequently, and because the stakeholders were right there observing the tests we can make those decisions really, really quickly. And then at the end of that process there’s usually a report that may or may not have video clips, depending on how quickly we’re working, or what it was that was observed, but some report that gets shared with a larger group of stakeholders. That might include more people at the VP level for example who are over the product managers. Or it might include the marketing people who are working on that product, or the people who are supporting that product, or – depending on what group of people we were talking to. And that they would then hear the results of that report. And then that report would then get mailed out to everybody who is a stakeholder that we know of and that maybe they weren’t able to attend the report debrief in person but they’ll get an email with a link to the report as well as sort of here are the top things that we found in the email, just so you know about it, and there’s more detail in the report. If you have any questions come and talk to us directly. So we’re doing this broadening out the communications circle at every point. So the people who are making the decisions about what the design looks like is a much smaller group, but the people who are hearing about the information can be pretty broad.
Steve: So that broader group, what are they going to – what are they going to do with that information? ‘Cuz your owners are right there and you’re making changes and making decisions as part of that process.
Steve: But it ripples out, so why do they – I mean not to be so negative, but like why do they care about it?
Aviva: If we’re in a usability test and we’re showing off a feature and some of the participants volunteer information about how much it will help them or what they see the value is of this, or what they like about it, then giving that to our marketing team, so they can use it to evangelize for that particular feature or product, is hugely valuable, right. ‘Cuz we’ve just said here’s what some people are saying about this that other people might resonate with. So it’s not just the usability of the product, but it’s also the entire customer experience of the product that we’re caring about.
Steve: So there’s qualitative information in something like that?
Aviva: Yes. And there’s also information about well we’re shipping this, but we know people are having some problems understanding this, one aspect of it. Or this particular concept’s a little bit difficult. So we’re going to do the best we can to explain it, but we’re also going to let support know in advance that hey you might get some calls about this. Or we’re going to let training – the DocuSign University folks – know we think that people are going to get a lot of value out of this but it’s kind of a complicated concept, maybe you should include it in your training. So there’s lots of different stakeholders involved.
Steve: And are – now I’m just grasping at straws here a little bit, but I can also imagine as you describe these scenarios, there are things that you uncover in usability testing and you can make the best design decisions you can, but they may – if there’s an underlying mental model issue or something it needs to get shared more broadly because it’s a bigger decision that’s going to have to get made that’s going to happen elsewhere.
Aviva: Absolutely. Right. Right. There’s a limit to – like if you’re on a deadline and you’ve got to ship something and you find out there’s a huge mental model conflict then the product team has to make a decision – do we keep going forward in this direction and try and spackle over it with help? Or do we step back and rethink this and take another look at it? Or do we ship this now and schedule that we’re going to take another look at this in some future release because people need this feature now one way or the other. So the product owner has a lot of decisions to make and you need to give them the best information that you can for them to make a good decision.
Steve: So I want to talk a little about how you got to this point here in this organization. You know you described to me and this – it sounds very state of the art. Sort of getting everyone together, being efficient, iterating, making decisions, sharing it. And you know this as well as I do that people will hear this or people come up to you and meet you at events and so on, we’ll say well, you know, I don’t get to do that at my job because this and that. You know there’s a lot of causes. But you’re sitting in a situation where you came into this organization – we haven’t talked about your own history here. I guess it’s about the question which I’m winding my way towards is, you know, how did it get to this point where you’re able to work this way?
Aviva: Well quite honestly it was starting in a particular focus and place which was the usability test. Proving the model, showing its success and its impact and just kind of driving that wedge in. But there’s a couple of things that are required in order to do that successfully and one is to realize that everybody is on the same team. They may have a different focus, but at the end of the day we all care about the quality of the product and we all care about the customer experience. If you don’t have that, if people in the company are not on that same page, it might be a little harder. I’m really fortunate to work with an incredible team of professionals here who really do care deeply about product quality and about innovation and about what can we do to make the customers’ lives better? So because I’m fortunate in that regard it’s easy to collaborate. It’s easy to trust. It’s easy to know that we all really want to do the right thing. We might not have the same opinion about how to do the right thing, or the timing of that right thing, or the cost of that right thing, but we can all agree that we want to do the right thing and so we have that shared goal, that shared mission. And so that has made it easier for the research team to – can I say be a victim of our own success? I mean we’re – there’s many more opportunities for us to have an impact than we have time. So we’re able to pick the teams that want to work with us and they’re going to commit to acting on the research insights that we deliver. If they just want us to do a study to prove that they’re right then we don’t have to take that project. But that really doesn’t happen here at DocuSign. It really doesn’t. It’s a great group of people.
Steve: So can you talk a little more about what was research before you got here? What was your brief? How did you sort of move it towards this victimhood of success (which is a lovely phrase)? Give us a little background.
Aviva: And I do have open positions available so we are trying to increase our size here and be able to serve more product teams. When I got here there had been a researcher sometime previously. I hadn’t met her. I wasn’t familiar with that person’s work. And we also had one person who was an intern who was doing the best he could. He was still working on his masters at San Jose State and he didn’t have an active mentor really. So it was a bit in a holding pattern. They had decided that they were going to build a lab but nobody was really looking at it really closely to see how it worked and so the lab that was delivered was not functional. So my brief was come in and build a research program, really from the ground up. Start with the basics. Start with enabling us to do user research. Start with building a lab on-site. Help us create personas because they didn’t have those. Basically I was given a bit of carte blanche to build the program and because I’ve been doing this for I guess 15 years now, 16 years now, that was great for me. I’ve gotten to build programs in other places and I’ve learned a lot along the way about what works and doesn’t work. So we just kind of got our foot in the door with some product teams and showed what kind of impact we could have and how quickly we could move. How rapidly we could do this work. How it didn’t have to take months and slow down development. How it could speed up development in some cases. And how it would just help make the product better. And fortunately everybody has been very responsive to that here.
Steve: If you can go back to when you were having those first conversations. You know they have something in mind. You can – you start to assess what you think the need is and what you might be able to do, but you know you pointed out that this is a place where everyone cares about working together and cares about creating the right kind of experience for people. How do you assess, before you start living here, can you be successful here? Can you accomplish what you all want to accomplish?
Aviva: That’s a tough one. I mean you do the best you can when you interview to ask questions about what the culture is like and what people’s goals are and how the different teams collaborate and work together. I’ve heard of companies where it’s a little bit more toxic, where design and engineering are sort of at each other’s throats and blaming each other a lot and not working collaboratively. I think that you can always intervene in those cultures and turn them around. I’ve seen that happen. But not here. That was other times in my career. I just had a sense when I first interviewed here over two years ago that this was a company that really cared about its customers and cared about making the experience for them as good as possible. And I’m pleased that that seems to have been the case. Like it’s held up. I interviewed with the Director of the UX team here, Sean McLeary. He’s a wonderful guy who is impassioned. You know he really wants to do the right thing and he’s willing to argue about it with his team, about, you know, tell me why you want to go in this direction? Have you considered these other things? Why have you made these choices? And if he disagrees he’s very direct about that at times. And that’s great because that means that people can be honest with each other and trust each other and not try to guess what they’re supposed to do, right. So it’s a fairly open environment. It’s an extremely collaborative environment. And the UX team has been very careful in hiring so that when we bring people on that they’ll fit into that open, trusting, collaborative culture.
Steve: So you’re within User Experience?
Aviva: I’m within User Experience and User Experience is within a larger customer experience group.
Steve: And so what’s the size of the research team now?
Aviva: Right now there are three of us, but I expect that to double within the next six months.
Steve: As you said there’s more to do then you have…
Aviva: There’s more to do then I have person power to do and there’s different kinds of projects that I would like to get into like doing more survey research. Right now survey research is extremely valuable at the place we are as a company. Just understanding user feedback and coding that. So that kind of part qualitative, part quantitative research takes a different kind of researcher sometimes then somebody who’s going to be doing a lot of moderating or a lot of interviewing. And so we’re opening up a contract role for that. We’re actually going to bring on a recruiter sometime soon just to give the researchers a few more cycles. And to be honest, they’re not thrilled about letting go of that time on the phone, but they’ll get used to it. But it has enabled them to deliver really good samples. We just need to be able to train the recruiter to be that good, to be as good as researchers and that’s certainly possible.
Steve: Well they’re coming in when there’s already a set of values and a set of approaches that you guys have created.
Aviva: Right, right. They’ll take over management of our customer panel and growing the customer panel and tracking participation and all the kinds of things that you need to do logistically in order to meet policy requirements.
Steve: What do you look for when you talk to people about working here? When you’re interviewing candidates as opposed to participants what are some things that jump out for you?
Aviva: Two things mainly. I want people who are passionate about research and nerdy about research. Like who love to discuss that you know – the value of doing this method over that method and can understand when you might choose a particular method over another one or know how to merge methods and create new things. But who also are incredibly good communicators and story tellers. Who can not just spend time collecting that data, but know how to then turn around and tell a compelling story and deliver compelling insights. So it’s actually more than two things because I’m also looking for people who understand not just research, but also design, well enough to be able to give good recommendations. So if somebody has a huge background in research but doesn’t know anything about design basics they’re not going to be able to translate what they’ve seen into any kind of recommendations that will be useful. So it’s helpful to have people who have some background in design as well as research.
Steve: I’m so fascinated by the relationship between design and research. You know if you look at – and you can probably explain this better than I could, but the history of a lot of the stuff is not through design. You know I came up through industrial design, before there was user experience design. And that was kind of the – at least they were the industry vanguard in insights and contextual research was about that. And so you didn’t have to understand design because designers would sort of take care of that part of it. And now user experience is kind of – maybe it’s just how I see the world and where I live and everything – but, you know, I’m not doing – I’m not in consumer package goods as much as I am in software for example. I think there’s an interesting relationship. I agree with you, if you can’t talk about the implications of your thing in the language of the people that are going to act on your thing, which is usually design, then you’re limited.
Aviva: Right. I really tend to hire generalists or try to create generalists because I’m a generalist. I came out of a background of doing demographic research and Nielsen audience research before I got into this whole web design world and most people aren’t generalists. Most people are either really good at qual or really good at quant but can’t necessarily pair them up. So that affects who I hire. But because the bread and butter of what we do always comes back to usability testing and user testing designs then understanding the context of web design or software design becomes really important. Otherwise you can say well they had problems doing this, but you can’t tell them why. And you can’t tell them what to do to fix it. So that’s why design is so important.
Steve: So is it okay for a researcher to say here’s what you could do to fix it?
Aviva: Yes, absolutely. It’s collaborative. Again, they are making a recommendation and it’s not a prescriptive recommendation, but it’s a suggestion. Like consider moving this label closer to the control. You know there’s all kinds of ways that the designer could choose to interpret that recommendation or act on it. They might change the control completely. They might – or they might just move the label closer to the control. Lots of things can happen, but it’s a place to start talking about it.
Steve: So the observation might be that, you know, people consistently didn’t appreciate that this control performed this function. So one thing you might do is move the descriptor or the label closer to it.
Steve: And then that’s – so that’s a way, and I think researchers – I think researchers struggle with the “A” word , actionable, right – being actionable. And so what you’re saying is just putting your observations or your synthesis, your interpretation, in the language of what you might do, moves the ball – I’m mixing metaphors here – moves the ball down the field significantly.
Aviva: Well, yeah. I mean part of it is tone, right. If a researcher is writing recommendations with the word “should” a lot I’m going to have a conversation with them about their tone because should is a really strong word. Like telling a designer how their product should behave, or how their product should look is not what we’re trying to do here. We’re trying to say hey there’s a problem and here’s some ideas about ways of fixing it that you should consider, but you’ll probably come up with better ways anyways. Or they might say, huh, that is a really good idea, I’m going to go ahead and do that. Mostly designers – I think everybody wants to be given latitude to make their own decisions. Just like a designer doesn’t want a product owner to tell them what design or control to use – like put a dropdown there. Well there’s all these other ways we could do the same thing that doesn’t involve a dropdown so let me be free to consider that. Researchers are the same. They don’t want to be told do a usability test because that might not be the right solution for the problem. We all want to be told here’s the problem, here’s some ideas for how to address that problem, now you’re a professional – go ahead and find the best solution. So we take that into account.
Steve: But I also I think you’re saying, if I’m not putting words in your mouth – I think you’re saying that even that dialogue about here’s the problem – I mean so a research request comes in and people sometimes say, and we all experience this, here’s the study we want you to do. And then you have to say well what’s the problem you’re trying to solve because people aren’t necessarily trained to say here’s the problem I’m trying to solve. So that’s also what’s happening – you’re trying to facilitate that happening between researcher and designer. Here’s what we’ve learned. Here’s the problem and here’s how you could solve it, not here’s how, as you said, here’s how you should solve it.
Aviva: You should solve it, right. Well and also there’s a matter of – reflecting on what you said about people aren’t trained how to ask. There’s questions that we ask when somebody comes to us and has a request for something. It’s like okay what decision are you going to make based on this information? And when do you need to make that decision by? And what’s the risk if we don’t do this? So it’s a fairly limited set of questions. It’s fairly easy to ask in a meeting or an email, just to get enough information to know huh, okay you’re asking for a survey, but what you really need to do is a card sort. They might not know what a card sort is so then you have an opportunity to explain that and to demonstrate how it’s worked well for other people and usually they’ll go that sounds really cool. Let’s do that. As long as it fits within their timeframe and their budget. But if somebody comes to me and needs a decision in two weeks I might prefer that they do a long field study that’s going to take a month to get back anything meaningful, but they don’t have the budget and the time for it so we’re going to figure out something that can work within the timeframe even if it’s only directional.
Steve: And there’s a managing expectations part of it that I’m sure you’ve kind of championed in your time.
Aviva: Yeah, I haven’t thought about it that way, but that’s actually a good way of putting it.
Steve: You know one of the things I enjoy about these conversations is hearing the comparing and contrasting what it’s like to be someone who works outside the company. Because you and I are both taking requests. We’re both having to kind of push back, clarify, go back and forth, but the context is just so different. You know the dynamic is so different. The money is different. Everything about it is different, but yet that’s – anyone that needs research done has some amount of familiarity and some amount they don’t have nor should they. That’s kind of what they’re coming to you for.
Aviva: Sure, sure. And I’m just happy that they’re asking us. You know they might be asking for the wrong thing, but at least they’re having a conversation with us. So it’s great to be asked. And then it’s an opportunity for us to maybe teach a little bit. But at the end of the day the research isn’t the product right? The research is how you get to the product. So we don’t expect to be treated with the same kind of importance that somebody would be putting into the actual development process, or shipping code, right, ‘cuz we’re there to help enable that. But they could do it without us. The outcome might not be as good. It might, but it might now.
Steve: And historically – I mean you know this ‘cuz you’ve been around, they did do it without it us.
Aviva: Yes, that’s right.
Steve: So – you know, just to sort of reflect on where we are at at this point in time and you’re in a leadership role in this function and you’re evangelizing it, advocating for it and now a victim of your own success. That this is – I think you represent a different state of the art of how this work is being done. I don’t know, it’s interesting to sort of see where have we all come from in this – where have we all come to I guess is the phrase.
Aviva: Well you know I’ve dropped out of enterprise from time to time and gone and worked as a consultant and it’s been really fun and I’ve been successful in my own way at it. I came and joined this company because I thought something special was happening here and it was the right time to do it. And I got to choose. Like there wasn’t any urgency for me about oh I need a job, I’m going to take the first one that comes up. And I think knowing as a user researcher that your work is valuable and that there’s high demand for your skills and that you have the ability to choose the place that you want to work and can choose the kind of culture that you want to contribute to is really important. That you have power as a user researcher. I think we’re in a different time right now than we were 10 or 15 years ago. The model has been pretty well proven. I think a lot of people understand the value of UX and the role that research plays in that. The role that talking to the customer and understanding the customer and the user experience plays in a company’s success. And so they’re actively looking for people in those roles.
Steve: You started off our conversation by describing how you guys are an unusual type of startup, but – does that – does the – does that business context, start up, what kind of start up, how old? Does that change some of the – you know how proven the model is among these different audiences?
Aviva: Absolutely, absolutely. You need a fairly mature management organization and fairly experienced product managers to – who know the value of this stuff. I think any MBA who’s been reading the Harvard Business Review gets the importance of UX on some level, but that may not translate into really understanding what it takes to do it. So you want to look for the people who have shipped before and who have worked with design before. I’m not the kind of person who wants to be the design team of one. There are other people who are really, really good at that. I don’t think that’s where user research shines obviously because you need a design team of at least two in order to support the kind of work that I want to do which is research. But that’s just me. I don’t want to be a designer.
Steve: Meaning a researcher and a designer. Is that what the team is?
Aviva: There are a lot of designers who do research, right. It’s a really common thing and you hear about the UX team of one where there’s one designer. That designer is going to have to do some research. A company has to be large enough to support a user research team. They need to get to that point and otherwise there’s not going to be a role for somebody who’s a pure researcher, let alone a research manager like me. So I’m going to pick those companies that are past that inflection point where they actually have enough designers on staff to support a user research team so that I can do what I like to do which is train user researchers and build teams.
Steve: So design precedes research in this example.
Aviva: You know that’s been the case everywhere I’ve worked. I think there’s been some cases where some really farsighted start ups have actually started with the research before they bring in the designers, but I’ve never encountered any myself.
Steve: I mean my theory is that just the research industry and the design industry, that we lag. There are people like you who are in-house leaders of research functions but that’s still relatively new, but not that long ago there was no one that was a leader in design. That was a very rare thing. Now it’s a very common thing. And then we’re kind of trailing them. You know what are people going to general assembly to learn? You know what is the market like out there? That those things are – that we’re always where they were a couple of years ago.
Aviva: Yeah, yeah. I think user researchers are just, user experience researchers in general are sort of odd ducks in a lot of ways. But yeah it is definitely a trailing career path in some respects. The classic example of a start up is the business guy and the tech guy, right, and sometimes you get lucky and you get a tech guy who also knows design and maybe they know research, but not usually. That’s not usually how it plays out.
Steve: I have to ask you to expand on the odd duck comment.
Aviva: Well you know we’re – researchers tend to be really fascinated by how things work. More so than we are in creating stuff that’s new. So our perspective is at the level of pattern. Like what are the patterns that are happening here and how can we evolve those patterns or how can we take advantage of those patterns or whatever it is. We’re looking at different measures of success for ourselves then a designer might. And we’re doing different kinds of things every day and yet we’re still in lock step with them when we work, at least that’s how it’s been for my teams. My teams have always been part of design. They’ve always been part of UX. And without UX there’s no UX research.
Steve: Well that tendency you’re describing seems like some of what you’re trying to mitigate with how you’re asking people to report. So don’t just say this thing is happening or this works this way, but lean in (if that’s not too overused a phrase) lean into the thing that’s not your default and say how it could be different.
Aviva: And I think the reason for that is in my time in this field the biggest complaint I’ve seen from decision makers are you’re giving me this information but you’re not telling me what to do with it – I’m not sure what I should be paying attention to. I’m not sure whether I need to get involved or not. Help me understand what I need to do next. And so part of that process of designing a report, or even a memo, or an email about a study, or designing the study itself is helping – helping the team understand what’s important and what they need to pay attention to and how they can fix things is a part of that. Otherwise you’re just kind of saying here’s some information that’s undifferentiated and kind of sending that over the wall and you don’t know what they’re going to do with it ‘cuz it’s divorced from the product experience. But I want the research team to be part of the product team. I want them to care about the user experience and that means caring about what happens as a result of their research and how it gets implemented on and acted on and built into the product, if that makes sense?
Steve: Yeah, it does. I think the whole sort of research analysis and synthesis process is – I mean here’s how I look at it I guess. It’s a series of iterations of oh wow, so what, oh I think it’s this, oh wow – you know you’re kind of picking things out, being excited about them, discarding them. It’s maybe like kneading dough or something. You pull some things out and push them back in.
Steve: And constantly kind of refining and I think a certain amount of it is sort of for your own enjoyment and I don’t mean that in a self-indulgent way, but…
Aviva: That’s true.
Steve: …you’re just trying to find the edges of the space. You’re trying to figure out what it is. Play with it, work it. But at some point you have to start to speak to someone who doesn’t speak your language which I think is a theme from our conversation here that you’re trying to communicate with other people that have other needs that need to act on it.
Aviva: Some years ago I gave a talk at a conference about walking the walk and what I meant by that is we talk about user experience and understanding the user’s language all the time, but we don’t always use that same set of skills for our own deliverables. We don’t always understand the mental model of the people that we work for and what their needs are and try and accommodate to that in how we deliver information, and that’s a huge fail. We have to not just talk the talk about understanding the user experience, but think of our own colleagues as the users of our insights and information and how can we communicate those in ways that are going to be useful and usable by them. That’s a big thing for me.
Steve: Yeah. That’s great. I think walk the walk is a lovely label to put on that. I’m going to switch gears slightly. I think it will, as it always does, it will I think maybe pull in some of the things we’ve talked about. I’d just love to hear you talk a bit about – you’ve eluded to it, but talk a little more about your background. I don’t know…
Aviva: My background is weird. My background is crazy. You know everybody gets into user research through these bizarre channels. It’s only recently that there’s programs where you can go and study user research in a master’s program and a design program and then apply it. So I came to the field because I was getting a doctorate in understanding how people communicated online which is a weird thing to get a doctorate in and I had to sort of make that up for myself. I started out as an educator and as part of that master’s program I was doing a lot of demographic work on different populations. So I learned how to do survey analysis. I learned how to use survey tools. And then at some point while I was being an educator I was a – I did a lot of work in group process situations. So it was information education and community education. So I learned a lot about how to get a group to move in a particular direction or how to run a meeting, things like that. I decided I didn’t want to work in non-profits anymore. This interesting thing was going on at the time with bulletin board systems and people creating community and talking to each other at a distance through these technologies that I thought was fascinating. I was a member of The Well at the time and some other bulletin board systems that were places where people could chat and there was this beginning of a academic area of cyber studies happening. So I decided I was going to change careers and go to school and study that. And I got into the University of Texas at Austin where they had this amazing group called the ACT Lab, or the Advanced Communication Technology Laboratory under the oversight of a really amazing woman named Allucquére Rosanne Stone who had done some of the early research in these cyber studies. And I started putting together a program for myself to understand how people communicated online. While I was doing that the Web sort of happened. This was in the early 90’s. And all of a sudden people were putting up these billboards of self-presentation on the information super highway. They were talking about themselves online. They were talking to other people online as a part of that. They were developing community through the Web. And I decided yeah I’m studying that. That’s interesting. So I started – a couple of things happened. Because my program was part of basically a film school, the Radio Television Film Program, production was really important and I wasn’t a film person so my production teaching was about the Web. So this new technology, the Web, was happening and I taught the first for credit class at the UT of Austin in designing web pages. And anybody could do it then. It was like this was in the days where image maps, being able to click on an image and have it do something was a big deal. So this was before CSS. So that’s how I got started on the Web and taught some classes in designing interface. That was part of what was going on in this lab. But when I decided what to study as a doctoral student I was studying what people were doing with the Web and how I decided to do that was to interview them in person and have them give me tours of their personal homepages. So it was kind of like doing digital ethnography. In fact it kind of was doing digital ethnography. We just didn’t really have words for it then. But sitting next to somebody over a computer and having them share their experience of it with you led very naturally into a user research role when I got out of school and decided I didn’t want to be an academic anymore. This was during the first bubble and there were a lot of opportunities so I ended up in a small e-commerce start up as a user researcher and just stuck with it and kind of learned as I went. We were all making this stuff up I think.
When I got to Yahoo, which I think was my third job in the field, that was an amazing place to learn the trade. At the time there were – it was such a big, active, busy environment with so many great people at the time and a really solid group of user researchers that I could learn from. So that’s kind of where I came up and was trusted with building a team for Yahoo Media in Santa Monica. And so left the Bay Area and went down to Santa Monica. Was just a user researcher by herself in this team and built a team. Hired more people. I think there were six of us when I left and that included some market research as well as user research because we became a customer insights team at some point. But I’ve been doing this since the year 2000 and it’s been a great career. It’s been a great ride. I would not – I couldn’t have chosen a better time to change fields because I love the research aspect of it, but I also love the collaboration aspect of it and the fact that we’re building stuff and sending it out in the world and that people are actually using it and what we do in the lab affects what they’re doing every day.
Steve: Do you think your other pursuits, and I’m thinking – I know that you play music and have been involved in music I guess since I’ve known you. Do things like that fit into the creative process that you harness for research?
Aviva: That’s a really interesting question. I think any kind of creative process is going to kick – kick start other creative processes. I think sometimes your brain just needs a break. When you’re trying to solve a problem sometimes it’s nice to step away from banging your head against it and do something else creative until you can get back some perspective.
Steve: So from head banging to head banging.
Aviva: Yeah, pretty much. That’s funny. Yeah, at some point in the past 10 years I went from being mostly an acoustic player to increasingly being a loud electric player so head banging probably applies. But I really do think of the music as a hobby. I’m not sure that I can draw an active connection between what it’s doing for me in research. It’s just a different place for my brain to be. One of the things that it does for me is it forces me to live in the moment more because when you’re playing lead guitar you’re reacting kind of on the fly and jamming with people around you and so that’s giving me more confidence with trusting my judgment in the moment. So maybe it’s doing that for me a little bit.
Steve: It reminds me of the workshop that you led that I participated in where you had people play instruments that they knew or didn’t know, kind of on the fly, and you pulled out all kinds of great lessons about being in the moment and collaborating and trusting each other and probably 10 other rich things that I’m omitting right now.
Aviva: Yeah, that was fun. And that wasn’t a workshop that I would ever have come up with on my own. It was something I was invited to do and it was a lot of fun doing it, but I honestly – I’m still kind of amazed that I got to do that, that I got to go to a UX conference and get people to play with musical instruments. It was fun.
Steve: I didn’t know it was like a – it just seemed to make sense in the audience.
Aviva: No, I was invited to do that by Erin Malone who coordinated and curated that conference.
Steve: If we were to sit down in five years and talk about some of these kinds of things – the field, DocuSign and the work that you’re doing – I don’t know, any of the things – we’ve been talking about the past up ‘til now, but if we flip it to look forward are there any things that leap out for you as either aspirations or expectations for the future?
Aviva: I think a lot about scale. How to do user research at scale? How to communicate findings at scale? How to make sure that we’re not losing insights when they’re being communicated at other parts of the organization. So I’ll probably be thinking a lot more about tools between now and then. And I also think a lot about how to train the next generation of user researchers because I know there are opportunities for them and I don’t think we’re – we’re churning out more designers I think in the system then we are churning out user researchers. I often talk to new graduates and they want to do both, but they don’t want to just do design and I want to know how to identify and train the people who would be stellar user researchers and maybe aren’t born designers, but would be great in this role and bring them up. So I’ll be thinking about that a little bit.
Steve: That’s great. Is there anything that we should have talked about, that I should have asked you?
Aviva: I love that question. I use that question all the time. If you are a user researcher who is looking for a great opportunity and has some years of experience under your belt and want to work in a place where your work is appreciated then please get in touch. I’d love to talk to you.
Steve: Do you have any questions for me?
Aviva: Is this what you expected?
Steve: Well yes and no. I mean yes this is the kind of conversation I was hoping to have. No, I didn’t expect exactly what I – the content was not expected. So yeah, this is…
Aviva: Well good. I always want to be a little bit surprising.
Steve: Yeah, and this was really good.
Aviva: Well thank you so much.
Steve: Well thank you very much.
Aviva: This has been fun.