Everything You Always Wanted to Know About Smartphone Qualitative *But Were Afraid to Ask

The Over the Shoulder team has spent the better part of a decade building and honing the Over the Shoulder smartphone qualitative platform and helping qualitative research moderators & practitioners use it to do industry-leading insight work.

Along the way, we’ve fielded a lot of questions that qual practitioners wonder about.

So, we decided to start posting common questions and answers to them as a resource for everyone who wants to learn about smartphone qualitative, understand their options, and be inspired by the opportunities.

Please feel free to build, comment or post additional questions. We’ll try to update this blog at least weekly. We hope you’ll find it helpful and inspiring! 

- The Over the Shoulder team.


How should I be thinking about timelines for my smartphone qualitative project? How long does recruiting take? How long should I leave for the project design & project programming process?


We’ll always do our utmost to meet your timelines regardless of the circumstances. That said, here are some guidelines on timelines:

3 weeks is ideal.  3 weeks from commissioning your project to the start of your fieldwork gives you enough time for just about any recruit, it allows plenty of time to come up with a great design, get input from your clients throughout the process, and it gives OTS ample time to get your study programmed, tested and ready for the field. The 3-week timeline gives everyone involved enough time to thoroughly and meticulously think through your project and not feel rushed.

2 weeks is comfortable as long as your recruit is standard.

1 week is doable if you can get input and approvals on your design quickly and your recruit is not too challenging.

Some studies can be put together and recruited in less than a week, but it’s probably going to feel rushed.

The big driver of your timeline is usually how long it will take to do your recruit. While a small study (20-30 participants) with wide recruiting specs can often be recruited in just a few days, highly specialized recruits can take a few weeks or more.

Designing a typical project usually takes about a week including reviews, approvals and adjustments.

Programming and testing a project is easy to do 72 hours. Shorter is usually possible, and we’ll always accommodate you if we can.


Does Over the Shoulder have a panel? Why?


Over the Shoulder does not have a panel. There are a few reasons. 

First, we committed early to building an open platform that can easily bring in participants from any source so that you have full flexibility to work with any panel provider, recruiting resource and lots of other recruiting approaches. Our mission is to help you do great smartphone qualitative, not sell you our panel.

Second, most of our clients need specialized recruits for their projects. We’ve learned through the years that “pre-recruited” smartphone panels just don’t have enough members to allow you to get good participants if you need specific age groups, regions, product usage specs or other recruiting requirements.

Third, most pre-recruited smartphone panels have real issues with participant quality, professional respondents and security.

Fourth, we respect and rely on our many recruiting partners, and having our own panel would put us in conflict with them.

Bottom line, every Over the Shoulder project gets a fresh, carefully screened, managed sample of exactly the participants you need. It takes a little longer and costs a little more, but experience has taught us that it gets you far better results and far fewer hassles.


Can Over the Shoulder do my recruiting for me?



Over the Shoulder has partnerships with recruiters that we can work with to get you high-quality, engaged participants for your project. We know which recruiters we can rely on to get exactly the right people into your project as quickly and cost-effectively as possible. If you’d like, we’ll even manage the process, write your recruitment screener, interact with your recruiter on your behalf to replace dropouts and include recruiting & incentives one simple invoice for you.

On the other hand, if you have a recruiting partner you love and you’d like to work with, that’s great too!  We’re happy to work with them to make your project go smoothly. We have support materials at the ready to help any recruiter be a great smartphone qual recruiter, and your Over the Shoulder project team will happily interact with any recruiting partner you choose to make your project successful.

And, if you have a panel of participants already, we can easily onboard them into your Over the Shoulder project.


I know my objectives and my insight needs, but I’m not sure that I can (or want to) figure out how to design a smartphone qualitative project. Can Over the Shoulder help me with that?


Of course! Over the Shoulder is proud to employ the best smartphone project designers in the business. Our average project designer has 6.5 years of experience designing smartphone qualitative (not bad considering smartphone qualitative has only been around for about 9 years!) and hundreds of projects under their belt.

We offer design support at three levels:

  1. Full Project Design

    It’s the easy way to a killer design. You start by sharing your objectives, insight needs and ideas for your project with your Project Designer. Your Project Designer carefully listens to what you’re trying to learn, and puts together an outline of how to answer your objectives, applying everything they know about the Over the Shoulder system, and proposes an outline for your design. You review it, shape it, add your thoughts and ideas, and when you love the outline, your Project Designer drafts a question-by-question, prompt-by-prompt smartphone discussion guide for you. Review it, get feedback and approvals, and presto…you have a great design every time.
  2. Design Assistance and Review

    Want to have more control and save some money? No problem. We’ll assign you a Project Designer who can run you through the basics, give you some ideas and send you off with our easy-to-use project-building template. You write up your smartphone discussion guide exactly how you want it. Your Project Designer is there to support you, and will review your project to give you any final suggestions and make sure it makes the most out of the platform.
  3. Design-it-yourself

    If you’re comfortable with the Over the Shoulder platform and smartphone qual design, you can draft up your project in our easy-to-use template.


Does Over the Shoulder do analysis, presentation building or do projects directly forend clients?


No. Over the Shoulder is here to support qual practitioners, not to compete with them.

Early in our history, we used our own platform and provided analysis and reporting to end clients.

Now, we focus on developing the world’s best smartphone qual platform and teaching practitioners how to use it. 

We don’t do analysis, but we’ll provide you with the tools and training to do it yourself. We don’t write reports or give presentations to end clients, but we’re happy to help you look like a rockstar.

That keeps us plenty busy, and ensures that the quallies we support never have to worry about us competing with them for projects. Seems fair to us.


Eight years ago, Over the Shoulder was a little group of passionate qualitative researchers who saw smartphones becoming popular and got excited about using them to get to fresher, more powerful insight. “Riding along” on the smartphones we knew people would start carrying with them wherever they went seemed like an amazing opportunity to understand their reality better.


At the time, there were no other smartphone qualitative research apps, so we designed and built the first one and started using it. We still remember the thrill of seeing these amazing moments of consumer reality come back through the app and lead to new insights and ideas.


But to be honest, that first version of the Over the Shoulder app left a great deal to be desired.


Why? Because it wasn’t easy for participants to use. Here’s an example: If we wanted to have participants capture a video, we had to ask them to leave the app, go through their home screen, go to their camera, capture the video, return to the app, tap “upload,” then navigate to the video they’d just taken, select it, and then upload it. It could take up to 4 minutes and 22 “screen taps” to capture a one-minute video.


Happily, one of the first things we intuitively did as we continued to develop Over the Shoulder was to find ways to make it easier for participants to capture their moments and express themselves. The payoff was immediate and dramatic. Every time we found a way to make capturing a moment take fewer “taps” and less time, we could see that participants pulled out their smartphones more often, and captured more powerful and insightful moments. Our practitioner clients noticed and encouraged us to continue our crusade to achieve the perfect participant user experience.


So we doubled down. We dedicated ourselves to figuring out how to remove “friction” wherever it occurred in the app user experience - no matter how minor. We filled notebooks with personal observations. We got serious about collecting and analyzing user feedback. We white-boarded, wireframed, and re-imagined what was possible. And we tested and built the best ideas into the Over the Shoulder app.


Over the last 8 years, we conceived of and added 77 significant improvements designed to make our app user experience easier, faster and more intuitive for participants. As a result, participating feels more natural and takes much less time. Participants don’t have to leave the app or wait while a photo, audio or video uploads. Answering a  photo, audio or video question  happens with dramatically fewer “taps”. Instructions are clearer and more engaging, so participants spend less time futzing with the app, and more time sharing their reality. In short, we maniacally dedicated ourselves to making Over the Shoulder the easiest participant user experience in the business.


But why do we even bother? Why does an easy participant user experience make such a big difference? Two big obvious but important factors are at play.


First, the moments smartphone qual lets us capture are often short-lived (they’re called moments, after all). Capture these moments and you’ll often find great insight. Use a tool that takes too many seconds or too many screen taps and you’ll miss the moment - you’ll get their post-rationalization, not their reality. A participant can pull out their smartphone and be capturing a video within the Over the Shoulder app in about 6 seconds (or 5 “taps” on their screen). That means they’re often capturing the experiences that lead to insight AS THEY ARE HAPPENING. The “frustration moments” they’re journaling are captured while they are experiencing actual frustration and can “blurt” about it without overthinking. In-the-emotion is often where the insight is found.


The second reason we’re maniacal about participant user experience is that people have so little patience for apps that take too much time and mental effort. Market research apps are no exception. Here’s a bit of harsh reality…if you’re interacting with people through an app, people are comparing your user experience to Google, Apple and Instagram. They spend their time with the best-designed apps in the world, and if your app experience doesn’t measure up, you’re annoying them. So it wasn’t good enough for Over the Shoulder to be better than the other smartphone research apps that have followed us into the market. We knew we had to make sharing a moment through Over the Shoulder as easy as updating your Facebook status.


But the payoff for getting your participant user experience right is massive.


When your app lets people journal a “thirst occasion” for you and it actually takes people 30 seconds to do it, they do it more often and more reliably, and you get better data. Try the same with an app that takes participants 2 or 3 minutes to capture the same data and your participants will skip occasions.


When participants can capture a photo, a video and a few key multiple choice or text responses in under 60 seconds, they can do an amazing job of helping you understand the relevant moments of their lives. And they mostly have a great time doing it. If you allow friction in your participant user experience, they won’t engage as well, and you won’t understand as well.

So call us obsessed. But it’s in the name of getting better insight. And insight is, after all, why you’re doing research in the first place.


Is An Emoji Worth A Thousand Words? Exploring The Use of Emojis in Qualitative Research

Whether English teachers like it or not, emojis are a huge part of how people express themselves today. To get a sense for how fast emojis - which literally means “picture character” in Japanese - are flying on the web, brace yourself as you check out this website that visualizes how frequently each one is used on twitter. And that’s just one source.

Another staggering statistic: Swyft Media reported that the world’s 2 billion smartphones shared over 6 billion emoji (or stickers) every day. In 2015.

Marketers recognize them as cultural currency and co-opt them. McDonald’s recent campaign is clearly lovin’ em.

Screen Shot 2017-01-18 at 10.03.03 AM.png

Their adoption seems set to accelerate even further as major players are actually building tools that enable emoji usage. Facebook has added emoji-based “reactions” as a feature. Smartphone keyboards in iOS and Android have launched a new emoji prediction and replacement feature that make adding them one-touch simple (e.g., tap in the word “happy” and iMessage suggests a 😀). Seen the latest Macbook Pro? Then you know the Touch Bar has effectively added emojis right to the laptop keyboard. It doesn’t take a 🕵🏽 to see that emojis are embedded in our culture, language and vocabulary.



The moment I realized emojis and smartphone qual might fit well together happened during a text conversation with my mom - a baby boomer and target of brands like Whole Foods, Subaru, Nordstrom, and Nutella. She was always very loose with the use of punctuation when texting. Our text conversations typically ended with something like, “I LOVE U TIMMY!!!!!!!!! HOPE TO SEE U SOON!!!!!!!!!”

Somewhere around 2014 she discovered emojis. To my surprise and intrigue, all that “!!!!!!” vanished... it was replaced with, “I ❤️ U 🦀🏃🏼🎶! HOPE TO SEE YOU SOON 🃏 🍻 🎟 !” Curious and confused, I called her to ask what all the emojis meant. This is what she said:

🦀 = I’m a Cancer and we share an intuitive nature (not intuitive enough, apparently)

🏃= She knows I’m busy running around the city but wishes I’d visit her in burbs more often

🎶 = She hopes I still whistle Beatles's songs while I work

🃏 = She misses laughing and joking around while playing crazy 8’s

🍻 + 🎟 = She hopes we could catch a concert before the summer ends

In that moment, I realized three things: 1) texting with emojis is an inherent part of communicating on smartphones today, and not just for millennials; 2) compared to written words, emojis are not only quick, easy and fun they’re sometimes a more accurate way to express emotions; and 3) with a little explanation, they reveal a lot about what a person is really thinking and feeling.



It wasn’t just my Mom. Around the same time, we started to see participants in the smartphone-based qualitative research projects we helped design and implement use emojis to express themselves instead of responding in text - completely unsolicited... It was just a natural part of how they were responding. With participants using them more and more often, it became clear that ignoring emojis was the equivalent of ignoring the body language of someone during an interview.



That’s when we stopped ignoring emojis and started having fun experimenting with the best ways to work them into our smartphone qualitative projects. We adjusted the Over the Shoulder platform so that emojis could be used by participants, included in questions, and even used to rate participant responses. Here are some successful ways we’ve used them so far:

1. Making Instructions & Prompts Simpler and More Engaging for Participants
Projects that make participation fun and easy for consumers get more and better insights. Our entire Over the Shoulder platform is designed around that crucial principle. So, we’re now animating the instructional text in our projects with emojis. It’s a simple way to make projects more engaging, and we’re finding that it’s already producing better insight.

2. Our New Favorite Projective Technique
Classic projective tests like the Rorschach and Thematic Apperception Tests work well on good smartphone-based qualitative platforms, but emojis definitely have something to add. It turns out that you can use them to help reveal motivations, needs and associations based on the participant’s free-association, just like the classic projective techniques.

Imagine you’re looking to understand a person’s emotional experience with fast food. When consumers use the app to document their in-the-moment occasions, why not have them type a string of emojis that represent how the experience is making them feel, and then explain their emoji selection in a follow-up video or audio response. This simple way of making it engaging and easy for people to express themselves definitely gets more insight than just using plain old words.


Screen Shot 2017-01-18 at 10.26.16 AM.png

3. Multiple Choice Emojis
Sometimes you want to have people select an emotional association from a closed-ended list. How are shoppers feeling during checkout on Black Friday vs. Cyber Monday? Happy, stressed, totally mad? The Over the Shoulder platform lets you add a corresponding emoji next to written words, so people can easily identify and select the emotion that fits their in-the-moment experience the best.



4. Beautify Reports
Our qualitative practitioner clients tell us that emojis help them make their debriefs and reports more powerful. Check out these colorful and visually-striking pages from some Over the Shoulder-inspired research as an example:


We’ll keep you posted as we experiment more with emojis in the smartphone-based projects we help our qualitative practitioner clients develop and execute. What do you think about using emojis in qualitative research? How have you been using them in your work?  We'd love a comment or question.


Using mobile qualitative to answer big data’s little questions.

As the designer and developer of a qualitative smartphone platform, the prospect of big data seemed really scary to us at first.

We imagined big data’s immense potential to capture actual behavioral data, pure and uninfluenced by the effects of observation. Massive numbers that we could manipulate and cut down to tiny subgroups without fear they’d disappear below the threshold of projectability and confidence. Getting insight into actual consumer behavior and motivation at a fundamental level (like qualitative is meant to do), and doing it on a massive, projectable scale. In short, we imagined big data rendering the qualitative we help our clients do less necessary.

But the more we’ve worked with clients who have big data at their disposal, the more we’ve come to understand big data’s ability to make qualitative better, and for qualitative to help fill the insight gaps that big data points to but can’t answer. Big data is amazing at answering “what’s happening.” But it almost always leaves you wondering “why?” And when brands go out to innovate and communicate, the “what” without the “why” isn’t enough.

So, as it turns out, big data has been a huge boon to the smartphone-based qualitative work we enable our clients to execute. Big data lets them see mass-scale behavior and lets them pinpoint crucial moments and points of influence that used to be invisible. It guides smartphone qualitative to focus on exactly the right moments where consumer motivations, perception and attitudes need to be better understood.

Here’s an example. One of our clients is an industry leader in collecting, managing and mining big data, then helping their clients build communication platforms based on the insight it provides. One of their data strategists shared a story with us about a retailer who actually used big data analysis techniques to reliably identify consumers who were pregnant based on other behavioral data they collected. Though the tactic proved problematic, the power to identify a target is impressive.

Now, imagine you’re a marketer for a company that manufactures baby furniture. Big data would let you know know exactly who is about to need baby furniture, what brands and retailers they’re likely to favor to buy it, and what media they consume so you can communicate with them about it. Goldmine.

But if you want to do anything more meaningful than send them a coupon, you’ll need to understand more. If you want to if you want to innovate and communicate effectively with people who are about to bring a child into the world, you need to understand things like: “what kind of an environment they’re hoping to create in their child’s nursery, and why.” You’ll need to understand how they perceive the different brands of furniture you’ll be competing with as they shop and what’s driving those perceptions. You’ll want to understand what’s special and different about your products, and how you can talk about your furniture as something that actually brings greater value to its users. You’ll still need to find insight into the consumer need your brand and product can satisfy better than anyone else.

Would you like qual with that? And now, with smartphone qualitative, you can get that insight easily. A simple, week-long smartphone qualitative project with 50 of the consumers your big data identified can let you complete the picture, You can ride along on their smartphones and have them show you the room they’ll be turning into a nursery, and share their plans, wants and dreams. And you can bring that richness right back to your team in photo collages, audio confessions and HD video. You can ride in their pockets and purses and have them journal the pathway they take as they decide what furniture they’ll buy, and document every in-store, online and other influence along the way. You can send them out to shop your product (and your competitors) and tell you exactly what your designs are doing that’s right, and how you’re faring at retail.

And it’s not just us wondering how to add the “why” to the “what” that big data identifies. We saw a panel discussion called “Qual VS Big Data.” at Qual 360 in Washington last week where insight leaders from Cirque du Soleil, Merck, Travelocity/Orbitz and Gallup weighed in. One of the key themes that emerged? It’s not “Qual vs Big Data” at all. It’s more like “Big Data lets me see what’s happening, and Qual let’s me understand it and make it meaningful to my internal audiences.”

Suddenly, big data and qualitative are more like Batman and Robin, and less like Lex Luther and Superman. Rather than fighting for space in the insight world, our experience has taught us that big data and qualitative are more of a dynamic duo helping to raise the bar for effective innovation, brand development and communication by working together to present the whole picture. And now instead of feeling threatening, big data starts to feel exciting.

Got a Big Data "what" and want to understand "why"? We'd love to hear from you. www.overtheshoulder.com/contact