smartphone qualitative

12 crucial tips to designing smartphone qualitative that gets great insight.

Over the Shoulder has been helping put smartphone-based qualitative into the toolkit of qual researchers and insight seekers for almost nine years now. We’re often asked by clients to list the biggest tips and “watch-outs” that smartphone qual practitioners should bear in mind to make their jobs easier and their deliverables to clients more valuable.

So we put the question to our in-house team of smartphone qualitative designers and turned their years of experience into the list of 12 crucial tips below.

  1. Design your project to be entertaining and engaging.

    Great smartphone qualitative leverages the intimacy people have with their smartphones, and the enjoyment participants get out of telling their stories and sharing their truths. Your study design should always reflect this. So “de-formalize” and “conversationalize” your language. “Gamify” your assignments. Fill your study with “Easter Egg” questions that provide moments of levity and little emotional rewards.

    Use a platform that lets you set up logic to give realtime acknowledgement to your participants. For example, if you as a participant to rate their test-drive experience from "Amazing" to "Disappointing" and they chose "Disappointing," following up with "Oh no! What do you mean when you say 'Disappointing'?" It makes participants feel like they're engaging with someone who really wants to hear what they have to say, not a machine. 

    Use emoticons to let participants express themselves, and use them in your design for standout, visual appeal and clarity.

    If you include scales, remember that rating the moment you’ve just experienced on a scale of “Best time ever” through “Major bummer” is more fun and conducive to emotional disclosure than rating it on a scale of “1-7, with 7 being extremely enjoyable.” Just about any project can be designed to be engaging to interact with, and the insights you get back get better when your project is entertaining and fun to be part of.

  2. Ask only the most important questions, and as few of them as possible.

    The biggest surprise our first-time clients get is the sheer volume of response you get from a project. And that’s great, as long as you’ve been disciplined in your design, and kept the number of audio and video recordings you ask for to a reasonable level. But if you ask too many audio and video questions, you’ll be awash in media response, and all the time we spend building tools to make your analysis more efficient will be powerless to help you. Asking for too much is the number one mistake first-time practitioners make and it can simultaneously kill your project profitability and annoy your participants while adding nothing to the insightfulness of your project.

  3. The right sample is the smallest one possible.

    Again, smartphone qualitative produces a large quantity of rich photo, audio, video and other data. Keeping your sample as small as possible reduces the amount of data you'll need to analyze.

    Most studies don't need more than 20-30 carefully-selected, engaged participants to produce great insight. Our "rule of thumb" is that each segment of your sample that you want to be able to understand and isolate from the others should have about 15 participants in it. 15 participants typically gets you to the feeling of "saturation" (where you start hearing the same themes and stories repeatedly, and the number of new ideas diminishes quickly).

  4. Choose a study length that lets you see a proper window into the behaviors you want to understand.

    Smartphone qualitative projects can be as long or as short as needed. We've helped clients do everything from single-day projects to ones where they participate in journaling for over a year (!). 

    The basic guidelines we recommend are:

    Keep your study period as short as possible (more study days = more data to go through).

    Design your study so that it's long enough to let you see the relevant behaviors you want to understand. For example, if you want to understand daily snacking behavior, be sure to include weekdays and weekends, as behaviors tend to vary between the two.

    Remember that your participants DON'T have to be doing assignments every day. If you want to understand how they find recipes, shop and prepare new foods, you can design your study length so that it spans they typical length of the behavioral cycle you want to observe. If your consumers typically do the "inspiration-preparation-serve & reaction" cycle over the course of 2 weeks, that's a good study length. You may not need your participants to be answering assignments every day over the whole two weeks. You can give them "rest days" and let them journal the behaviors as they naturally unfold. There's often no need to make up a daily assignment to understand a weekly behavior.

  5. Always participate in an on-device test of your project before you launch it.

    Seriously. We never let a study we've designed and built for a client go into the field without its designer going through it on their own smartphone. Even our most experienced Project Designers, who have designed hundreds of smartphone qualitative projects for our clients, will tell you they almost invariably learn something that can make the participant experience better and the project more successful. Walking through your project on your smartphone will instantly reveal if you’ve broken #1 or #2. Plus, knowing what it feels like to be on the receiving end of your assignments journals and questions is always good practice.

  6. Make participating easy for your participants.

    Eight years of experience in smartphone qual have taught us one important (if obvious) truth. "Make it easy for participants = Get better insight." That's why our entire platform is designed around having the simplest, clearest and easiest participant experience in the business. But the actual design of your project plays a big role as well, and there are many ways to make life easier for participants.  For example:

    Always make it clear to participants where they are in your study and what's coming up. Tell them how long the assignment they're about to to start is going to take them (and never underestimate it).

    If you have people journaling their "joys and frustrations" in the moment, make sure your journaling assignment takes 60 seconds or less from opening it to hitting "submit" and don't make them wait while their answers upload. You'll be amazed how many more moments you capture, and the quality of those moments.

    Use logic and skip patterns so that participants never have to "forward through" questions that aren't relevant to them.

    Be incredibly clear on when projects start and end. Our rule of thumb is that any important project detail needs to be communicated three times to avoid confusion. Participants should be able to get back to an explanation of the rules, dates and expectations of your project right within the app at any time.

    Ensure that you're clear up front (at the recruiting stage) exactly how much time participating in your project is going to take.

    Avoid extending studies past their original finish dates, and if you have to extend them, offer bonus incentives. Few things irritate participants more than "adding on a few extra assignments" that will require them to continue participating after the date you told them they'd be finished.

  7. Choose response media so that participants can easily and comfortably express themselves.

    One of the most exciting things about smartphone qualitative is obviously the ability to submit beautiful "selfie" videos in answer to your questions. And there's no doubt that a great in-the-moment HD video can be an insightful showstopper in a presentation. But video isn't the right capture medium in all situations.

    For example, if you've sent your hemorhoid-suffering participants into the drug store to survey the shelves and tell you about the product that's most relevant to them and why, asking for a selfie video will make them uncomfortable (or should we say "even more uncomfortable"). But they can easily take a photo of the product, then hold their phone up to their ear (feigning a phone call) and tell you about their inner monolog in an audio recording. You'll get far better insights for it, not to mention better compliance. More on choosing the right media can be found in "In praise of audio recordings."

  8. Review your results while your study is live, and ask follow-up questions.

    The beauty of good smartphone qualitative lies in its interactivity and flexibility. A good smartphone qual platform will let you know the instant participants submit responses and let you send them individual probes and follow-up questions when you need to. Over the Shoulder not only lets you send probes and follow-ups, it even lets you ask for responses in ANY media you choose. 

    We recommend you or someone on your team dedicate a block of time each day during your fieldwork so that you can take advantage of the opportunity. Someone will need to go through your participants’ submissions regardless: that someone may as well be doing it in real time so that they can take advantage of the ability to probe individuals and get to deeper layers of insight.

    Another important advantage of working in “real time” with your project (or having a Community Manager who does it for you) is how motivating it is to your participants. Remember, participating in a smartphone research project feels, well, weird at first for participants. Imagine yourself sending your personal thoughts off into the ether as photos, text, audio and video recordings, and wondering if anyone’s even looking at them on the other side.

    Giving individual participants a push notification addressing them by name and telling them that they're doing a great job is extremely engaging and will get you better insight. And it’s something that we can say with authority will increase engagement with your project and the quality of the response you get.

    Same goes for participants who aren’t performing up to grade. An artful prod from a Community Manager can often turn a marginal participant into a superstar and avoid dropout and replacement costs and delays.

  9. Ensure that you have participants who are real people, engaging fully in your project

    Getting the right participants into your project takes effort, planning and money. But unlike online quant and even some online qual, the audios and videos smartphone qualitative generates will reveal a poor recruit immediately.

    Find a good recruiting partner (we're happy to help), pay a motivating incentive, and manage your participant community actively (if you'd like Over the Shoulder can assign an internal Community Manager to your project to ensure this happens).

    Some of our practitioners even schedule a 10-minute live intro call with each participant at the beginning of the project to get them warmed up, comfortable and engaged before they download the app and start participating. 

    Pay participants an incentive that makes it worth their time to engage fully with your project (we recommend paying roughly the same per-hour rate that you’d pay for face-to-face qualitative). The recruiting and incentive tab for your job may be a little higher, but so will the engagement with your study and the value of the insights it lets you bring your client. Replacing bad participants and chasing down participants who are not fully engaged will end up costing you more than having a compelling incentive and well-screened participants in the first place, as well as keeping your project on schedule. “Pay peanuts, get monkeys” definitely applies here.

  10. Have an analysis plan, and design your study to make analysis easier and efficient.

    Our smartphone qual designers quite literally consider what kind of answers they’ll get to an assignment or question, and how those answers will be analyzed and presented as they work through their initial design. It’s like a research geek’s version of “beginning with the end in mind.” 

    You should ensure that you’re familiar with the tools you’ll be using to monitor and analyze your results well before the submissions from your participants start rolling in. If you’re using a new platform, or its your first smartphone qual project, make sure you're totally comfortable using it BEFORE your responses roll in. We actually set new clients up with access to a “demo” Project Viewing Portal so that we can make sure they’re up to speed with how it works, and the analysis tools they’ll be using before their fieldwork starts. And, we make sure that they have an analysis plan specifically based on their project’s design so they know what they'll be doing with the answers to all of the questions and assignments before the project even starts.

  11. Make your project entertaining and engaging for your clients.

    Remember that your clients are giving up the "focus group ritual" when you use smartphone qualitative. The "back room chatter" and the group focus that comes with it are a valuable part of the face-to-face research process.

    Happily, good smartphone qualitative platforms offer lots of ways to rally your key clients around the project and engage them with it so that they get great value out of it. Try "daily reports" with key submissions. "Buddy up" key clients with a research participant and make it easy for them to see their buddy participant's responses as they come in. Even let them ask follow-up questions and probes (with you as editor and controller) so that they feel involved and learn as the project unfolds.

    Use your "ripped from reality" photos, audios and videos to make a powerful presentation. Even use tools like online media collages that let your clients share and access key participant submissions long after the project is over.

  12. Respect the privacy of your participants.

    In many studies, we're asking participants to capture and share moments that are intensely private. So, it's crucial to be good guardians of the secrets of people who participate in studies.

    Ensure you're using a system that actively protects participant's identifiable information and everything they submit. If your research subject is intensely private, ensure you're working in a system that can keep participant identities entirely separate from what they submit. 

    If a submission looks so great that you can't resist using it outside of the project that collected it, make sure you can get the participant's explicit permission to use it. We find that most participants are happy to have their submissions shared if they can review what's going to be shared, and the exact way their submissions will be used.

    We've built technological, operational and organizational security at the highest level right in to the Over the Shoulder platform so that we can keep people's secrets safe, and help you make sure you're using their submissions appropriately and respectfully.

In praise of audio recordings

Smartphone-based qualitative as a methodology is growing rapidly. But as market researchers, we’re still learning a great deal about the best way to get great insight from participants. As an example, take the decision of whether to ask participants to respond using video or audio.

Capturing video from consumers through their smartphones is an amazing tool. In-the-moment consumer videos are rich, visually powerful, and a great way to tell a story and make a point. That’s why high-quality video is a core component of the Over the Shoulder platform, and a centerpiece of virtually every project our researcher-clients conduct.

But in many cases, the power of video overshadows another very powerful way for participants to express their thoughts and feelings: good old audio recording.

Recording an audio elicits a different feeling compared to recording a video. Try it yourself. Go to your smartphone's video camera, and make a “selfie” recording telling me about something that annoys you.

How did you feel doing that? For most people, the answer is “I feel a little self-conscious and maybe a little distracted.” You have to look at yourself talking, which many people don’t really like to do. Did you fix your hair or adjust your appearance before you hit “record?” That feeling of self-consciousness can get in the way of your ability to let your thoughts flow freely.

Now make an audio recording telling me about something that’s annoyed you at some point today. Did it feel different? You may have noticed that your thoughts flowed more freely as you recorded your audio. And that free-form thinking is extremely valuable in smartphone qualitative.

So when is audio preferable to video?

  • We sometimes counsel the researchers who use the Over the Shoulder platform to “use audio responses to get depth of understand of your consumer, and video when you want to be able to make a point in a presentation.” It’s not that video isn’t great, it’s simply that in many situations, audio recordings get a more free-flowing, unedited response from participants. 

    Beyond getting more free-flowing responses, allowing your participants to respond with audio recordings has other big benefits. 
  • Imagine you’ve got participants journaling a visit to the drug store. You’ve got them standing in front of the shelf, viewing all of the hemorrhoid remedies they could choose, and you want to understand what products are connecting with them and communicating something relevant. You could ask them to record a selfie video explaining their thinking, but you shouldn’t expect great a response. But if you prime your participants to answer the question with an audio recording and suggest they hold their phone up to their ear like they’re quietly talking on the phone, you’ve created a situation where they can much more comfortably share what’s going on in their heads, This will lead to responses that are free from embarrassment and self-censorship, and that means great insights for you.
  • Here’s another example: say you want participants to share their thoughts first thing in the morning, before they get out of bed. Or you want them to talk about their post-shower routine. You can ask for a video, but you’re unlikely to get it. But if you ask for an audio recording, participants don’t have to worry about their bed-head or appearing naked in a video. They can easily express themselves in an audio recording, and you can get their true in-the-moment thoughts and emotions.

And what do participants think and feel about audio?

We regularly poll Over the Shoulder participants across projects, and their feedback bears out the strengths of using audio recordings. We asked over 1,000 participants who had just completed projects on the Over the Shoulder platform “Thinking back to all of the different ways you were asked to answer questions in this project, which did you find the easiest to express yourself in?” People were more than twice as likely to prefer “audio recording” over “videos.” When we asked them why, they gave answers like “you can record an audio anywhere, anytime, without feeling self-conscious” or “my thoughts just flow when I’m able to talk without the distraction of seeing myself in the video I’m creating.”

With all of the benefits of allowing research participants to answer questions with audio recordings, it’s a wonder why the vast majority of smartphone qualitative platforms don’t include audio recordings as an option. The answer is that enabling audio recording across Android and iOS devices is technically difficult. But at Over the Shoulder, our devotion to making it easy to be a participant, and making it easy for participants to express themselves freely drove us to figure out a way to ensure that audio recordings are always an option.

So, the next time you’re designing a smartphone qualitative project, bear in mind that an audio recording may be the best way to capture in-the-moment thoughts and emotions and make sure that you’re using a platform that makes recording audios easy for participants.

Is An Emoji Worth A Thousand Words? Exploring The Use of Emojis in Qualitative Research

Whether English teachers like it or not, emojis are a huge part of how people express themselves today. To get a sense for how fast emojis - which literally means “picture character” in Japanese - are flying on the web, brace yourself as you check out this website that visualizes how frequently each one is used on twitter. And that’s just one source.

Another staggering statistic: Swyft Media reported that the world’s 2 billion smartphones shared over 6 billion emoji (or stickers) every day. In 2015.

Marketers recognize them as cultural currency and co-opt them. McDonald’s recent campaign is clearly lovin’ em.

Screen Shot 2017-01-18 at 10.03.03 AM.png

Their adoption seems set to accelerate even further as major players are actually building tools that enable emoji usage. Facebook has added emoji-based “reactions” as a feature. Smartphone keyboards in iOS and Android have launched a new emoji prediction and replacement feature that make adding them one-touch simple (e.g., tap in the word “happy” and iMessage suggests a 😀). Seen the latest Macbook Pro? Then you know the Touch Bar has effectively added emojis right to the laptop keyboard. It doesn’t take a 🕵🏽 to see that emojis are embedded in our culture, language and vocabulary.



The moment I realized emojis and smartphone qual might fit well together happened during a text conversation with my mom - a baby boomer and target of brands like Whole Foods, Subaru, Nordstrom, and Nutella. She was always very loose with the use of punctuation when texting. Our text conversations typically ended with something like, “I LOVE U TIMMY!!!!!!!!! HOPE TO SEE U SOON!!!!!!!!!”

Somewhere around 2014 she discovered emojis. To my surprise and intrigue, all that “!!!!!!” vanished... it was replaced with, “I ❤️ U 🦀🏃🏼🎶! HOPE TO SEE YOU SOON 🃏 🍻 🎟 !” Curious and confused, I called her to ask what all the emojis meant. This is what she said:

🦀 = I’m a Cancer and we share an intuitive nature (not intuitive enough, apparently)

🏃= She knows I’m busy running around the city but wishes I’d visit her in burbs more often

🎶 = She hopes I still whistle Beatles's songs while I work

🃏 = She misses laughing and joking around while playing crazy 8’s

🍻 + 🎟 = She hopes we could catch a concert before the summer ends

In that moment, I realized three things: 1) texting with emojis is an inherent part of communicating on smartphones today, and not just for millennials; 2) compared to written words, emojis are not only quick, easy and fun they’re sometimes a more accurate way to express emotions; and 3) with a little explanation, they reveal a lot about what a person is really thinking and feeling.



It wasn’t just my Mom. Around the same time, we started to see participants in the smartphone-based qualitative research projects we helped design and implement use emojis to express themselves instead of responding in text - completely unsolicited... It was just a natural part of how they were responding. With participants using them more and more often, it became clear that ignoring emojis was the equivalent of ignoring the body language of someone during an interview.



That’s when we stopped ignoring emojis and started having fun experimenting with the best ways to work them into our smartphone qualitative projects. We adjusted the Over the Shoulder platform so that emojis could be used by participants, included in questions, and even used to rate participant responses. Here are some successful ways we’ve used them so far:

1. Making Instructions & Prompts Simpler and More Engaging for Participants
Projects that make participation fun and easy for consumers get more and better insights. Our entire Over the Shoulder platform is designed around that crucial principle. So, we’re now animating the instructional text in our projects with emojis. It’s a simple way to make projects more engaging, and we’re finding that it’s already producing better insight.

2. Our New Favorite Projective Technique
Classic projective tests like the Rorschach and Thematic Apperception Tests work well on good smartphone-based qualitative platforms, but emojis definitely have something to add. It turns out that you can use them to help reveal motivations, needs and associations based on the participant’s free-association, just like the classic projective techniques.

Imagine you’re looking to understand a person’s emotional experience with fast food. When consumers use the app to document their in-the-moment occasions, why not have them type a string of emojis that represent how the experience is making them feel, and then explain their emoji selection in a follow-up video or audio response. This simple way of making it engaging and easy for people to express themselves definitely gets more insight than just using plain old words.


Screen Shot 2017-01-18 at 10.26.16 AM.png

3. Multiple Choice Emojis
Sometimes you want to have people select an emotional association from a closed-ended list. How are shoppers feeling during checkout on Black Friday vs. Cyber Monday? Happy, stressed, totally mad? The Over the Shoulder platform lets you add a corresponding emoji next to written words, so people can easily identify and select the emotion that fits their in-the-moment experience the best.



4. Beautify Reports
Our qualitative practitioner clients tell us that emojis help them make their debriefs and reports more powerful. Check out these colorful and visually-striking pages from some Over the Shoulder-inspired research as an example:


We’ll keep you posted as we experiment more with emojis in the smartphone-based projects we help our qualitative practitioner clients develop and execute. What do you think about using emojis in qualitative research? How have you been using them in your work?  We'd love a comment or question.


Using mobile qualitative to answer big data’s little questions.

As the designer and developer of a qualitative smartphone platform, the prospect of big data seemed really scary to us at first.

We imagined big data’s immense potential to capture actual behavioral data, pure and uninfluenced by the effects of observation. Massive numbers that we could manipulate and cut down to tiny subgroups without fear they’d disappear below the threshold of projectability and confidence. Getting insight into actual consumer behavior and motivation at a fundamental level (like qualitative is meant to do), and doing it on a massive, projectable scale. In short, we imagined big data rendering the qualitative we help our clients do less necessary.

But the more we’ve worked with clients who have big data at their disposal, the more we’ve come to understand big data’s ability to make qualitative better, and for qualitative to help fill the insight gaps that big data points to but can’t answer. Big data is amazing at answering “what’s happening.” But it almost always leaves you wondering “why?” And when brands go out to innovate and communicate, the “what” without the “why” isn’t enough.

So, as it turns out, big data has been a huge boon to the smartphone-based qualitative work we enable our clients to execute. Big data lets them see mass-scale behavior and lets them pinpoint crucial moments and points of influence that used to be invisible. It guides smartphone qualitative to focus on exactly the right moments where consumer motivations, perception and attitudes need to be better understood.

Here’s an example. One of our clients is an industry leader in collecting, managing and mining big data, then helping their clients build communication platforms based on the insight it provides. One of their data strategists shared a story with us about a retailer who actually used big data analysis techniques to reliably identify consumers who were pregnant based on other behavioral data they collected. Though the tactic proved problematic, the power to identify a target is impressive.

Now, imagine you’re a marketer for a company that manufactures baby furniture. Big data would let you know know exactly who is about to need baby furniture, what brands and retailers they’re likely to favor to buy it, and what media they consume so you can communicate with them about it. Goldmine.

But if you want to do anything more meaningful than send them a coupon, you’ll need to understand more. If you want to if you want to innovate and communicate effectively with people who are about to bring a child into the world, you need to understand things like: “what kind of an environment they’re hoping to create in their child’s nursery, and why.” You’ll need to understand how they perceive the different brands of furniture you’ll be competing with as they shop and what’s driving those perceptions. You’ll want to understand what’s special and different about your products, and how you can talk about your furniture as something that actually brings greater value to its users. You’ll still need to find insight into the consumer need your brand and product can satisfy better than anyone else.

Would you like qual with that? And now, with smartphone qualitative, you can get that insight easily. A simple, week-long smartphone qualitative project with 50 of the consumers your big data identified can let you complete the picture, You can ride along on their smartphones and have them show you the room they’ll be turning into a nursery, and share their plans, wants and dreams. And you can bring that richness right back to your team in photo collages, audio confessions and HD video. You can ride in their pockets and purses and have them journal the pathway they take as they decide what furniture they’ll buy, and document every in-store, online and other influence along the way. You can send them out to shop your product (and your competitors) and tell you exactly what your designs are doing that’s right, and how you’re faring at retail.

And it’s not just us wondering how to add the “why” to the “what” that big data identifies. We saw a panel discussion called “Qual VS Big Data.” at Qual 360 in Washington last week where insight leaders from Cirque du Soleil, Merck, Travelocity/Orbitz and Gallup weighed in. One of the key themes that emerged? It’s not “Qual vs Big Data” at all. It’s more like “Big Data lets me see what’s happening, and Qual let’s me understand it and make it meaningful to my internal audiences.”

Suddenly, big data and qualitative are more like Batman and Robin, and less like Lex Luther and Superman. Rather than fighting for space in the insight world, our experience has taught us that big data and qualitative are more of a dynamic duo helping to raise the bar for effective innovation, brand development and communication by working together to present the whole picture. And now instead of feeling threatening, big data starts to feel exciting.

Got a Big Data "what" and want to understand "why"? We'd love to hear from you.