Smartphone Qualitative

12 crucial tips to designing smartphone qualitative that gets great insight.

Over the Shoulder has been helping put smartphone-based qualitative into the toolkit of qual researchers and insight seekers for almost nine years now. We’re often asked by clients to list the biggest tips and “watch-outs” that smartphone qual practitioners should bear in mind to make their jobs easier and their deliverables to clients more valuable.

So we put the question to our in-house team of smartphone qualitative designers and turned their years of experience into the list of 12 crucial tips below.

  1. Design your project to be entertaining and engaging.

    Great smartphone qualitative leverages the intimacy people have with their smartphones, and the enjoyment participants get out of telling their stories and sharing their truths. Your study design should always reflect this. So “de-formalize” and “conversationalize” your language. “Gamify” your assignments. Fill your study with “Easter Egg” questions that provide moments of levity and little emotional rewards.

    Use a platform that lets you set up logic to give realtime acknowledgement to your participants. For example, if you as a participant to rate their test-drive experience from "Amazing" to "Disappointing" and they chose "Disappointing," following up with "Oh no! What do you mean when you say 'Disappointing'?" It makes participants feel like they're engaging with someone who really wants to hear what they have to say, not a machine. 

    Use emoticons to let participants express themselves, and use them in your design for standout, visual appeal and clarity.

    If you include scales, remember that rating the moment you’ve just experienced on a scale of “Best time ever” through “Major bummer” is more fun and conducive to emotional disclosure than rating it on a scale of “1-7, with 7 being extremely enjoyable.” Just about any project can be designed to be engaging to interact with, and the insights you get back get better when your project is entertaining and fun to be part of.

     
  2. Ask only the most important questions, and as few of them as possible.

    The biggest surprise our first-time clients get is the sheer volume of response you get from a project. And that’s great, as long as you’ve been disciplined in your design, and kept the number of audio and video recordings you ask for to a reasonable level. But if you ask too many audio and video questions, you’ll be awash in media response, and all the time we spend building tools to make your analysis more efficient will be powerless to help you. Asking for too much is the number one mistake first-time practitioners make and it can simultaneously kill your project profitability and annoy your participants while adding nothing to the insightfulness of your project.

     
  3. The right sample is the smallest one possible.

    Again, smartphone qualitative produces a large quantity of rich photo, audio, video and other data. Keeping your sample as small as possible reduces the amount of data you'll need to analyze.

    Most studies don't need more than 20-30 carefully-selected, engaged participants to produce great insight. Our "rule of thumb" is that each segment of your sample that you want to be able to understand and isolate from the others should have about 15 participants in it. 15 participants typically gets you to the feeling of "saturation" (where you start hearing the same themes and stories repeatedly, and the number of new ideas diminishes quickly).


     
  4. Choose a study length that lets you see a proper window into the behaviors you want to understand.

    Smartphone qualitative projects can be as long or as short as needed. We've helped clients do everything from single-day projects to ones where they participate in journaling for over a year (!). 

    The basic guidelines we recommend are:

    Keep your study period as short as possible (more study days = more data to go through).

    Design your study so that it's long enough to let you see the relevant behaviors you want to understand. For example, if you want to understand daily snacking behavior, be sure to include weekdays and weekends, as behaviors tend to vary between the two.

    Remember that your participants DON'T have to be doing assignments every day. If you want to understand how they find recipes, shop and prepare new foods, you can design your study length so that it spans they typical length of the behavioral cycle you want to observe. If your consumers typically do the "inspiration-preparation-serve & reaction" cycle over the course of 2 weeks, that's a good study length. You may not need your participants to be answering assignments every day over the whole two weeks. You can give them "rest days" and let them journal the behaviors as they naturally unfold. There's often no need to make up a daily assignment to understand a weekly behavior.

     
  5. Always participate in an on-device test of your project before you launch it.

    Seriously. We never let a study we've designed and built for a client go into the field without its designer going through it on their own smartphone. Even our most experienced Project Designers, who have designed hundreds of smartphone qualitative projects for our clients, will tell you they almost invariably learn something that can make the participant experience better and the project more successful. Walking through your project on your smartphone will instantly reveal if you’ve broken #1 or #2. Plus, knowing what it feels like to be on the receiving end of your assignments journals and questions is always good practice.

     
  6. Make participating easy for your participants.

    Eight years of experience in smartphone qual have taught us one important (if obvious) truth. "Make it easy for participants = Get better insight." That's why our entire platform is designed around having the simplest, clearest and easiest participant experience in the business. But the actual design of your project plays a big role as well, and there are many ways to make life easier for participants.  For example:

    Always make it clear to participants where they are in your study and what's coming up. Tell them how long the assignment they're about to to start is going to take them (and never underestimate it).

    If you have people journaling their "joys and frustrations" in the moment, make sure your journaling assignment takes 60 seconds or less from opening it to hitting "submit" and don't make them wait while their answers upload. You'll be amazed how many more moments you capture, and the quality of those moments.

    Use logic and skip patterns so that participants never have to "forward through" questions that aren't relevant to them.

    Be incredibly clear on when projects start and end. Our rule of thumb is that any important project detail needs to be communicated three times to avoid confusion. Participants should be able to get back to an explanation of the rules, dates and expectations of your project right within the app at any time.

    Ensure that you're clear up front (at the recruiting stage) exactly how much time participating in your project is going to take.

    Avoid extending studies past their original finish dates, and if you have to extend them, offer bonus incentives. Few things irritate participants more than "adding on a few extra assignments" that will require them to continue participating after the date you told them they'd be finished.

     
  7. Choose response media so that participants can easily and comfortably express themselves.

    One of the most exciting things about smartphone qualitative is obviously the ability to submit beautiful "selfie" videos in answer to your questions. And there's no doubt that a great in-the-moment HD video can be an insightful showstopper in a presentation. But video isn't the right capture medium in all situations.

    For example, if you've sent your hemorhoid-suffering participants into the drug store to survey the shelves and tell you about the product that's most relevant to them and why, asking for a selfie video will make them uncomfortable (or should we say "even more uncomfortable"). But they can easily take a photo of the product, then hold their phone up to their ear (feigning a phone call) and tell you about their inner monolog in an audio recording. You'll get far better insights for it, not to mention better compliance. More on choosing the right media can be found in "In praise of audio recordings."

     
  8. Review your results while your study is live, and ask follow-up questions.

    The beauty of good smartphone qualitative lies in its interactivity and flexibility. A good smartphone qual platform will let you know the instant participants submit responses and let you send them individual probes and follow-up questions when you need to. Over the Shoulder not only lets you send probes and follow-ups, it even lets you ask for responses in ANY media you choose. 

    We recommend you or someone on your team dedicate a block of time each day during your fieldwork so that you can take advantage of the opportunity. Someone will need to go through your participants’ submissions regardless: that someone may as well be doing it in real time so that they can take advantage of the ability to probe individuals and get to deeper layers of insight.

    Another important advantage of working in “real time” with your project (or having a Community Manager who does it for you) is how motivating it is to your participants. Remember, participating in a smartphone research project feels, well, weird at first for participants. Imagine yourself sending your personal thoughts off into the ether as photos, text, audio and video recordings, and wondering if anyone’s even looking at them on the other side.

    Giving individual participants a push notification addressing them by name and telling them that they're doing a great job is extremely engaging and will get you better insight. And it’s something that we can say with authority will increase engagement with your project and the quality of the response you get.

    Same goes for participants who aren’t performing up to grade. An artful prod from a Community Manager can often turn a marginal participant into a superstar and avoid dropout and replacement costs and delays.

     
  9. Ensure that you have participants who are real people, engaging fully in your project

    Getting the right participants into your project takes effort, planning and money. But unlike online quant and even some online qual, the audios and videos smartphone qualitative generates will reveal a poor recruit immediately.

    Find a good recruiting partner (we're happy to help), pay a motivating incentive, and manage your participant community actively (if you'd like Over the Shoulder can assign an internal Community Manager to your project to ensure this happens).

    Some of our practitioners even schedule a 10-minute live intro call with each participant at the beginning of the project to get them warmed up, comfortable and engaged before they download the app and start participating. 

    Pay participants an incentive that makes it worth their time to engage fully with your project (we recommend paying roughly the same per-hour rate that you’d pay for face-to-face qualitative). The recruiting and incentive tab for your job may be a little higher, but so will the engagement with your study and the value of the insights it lets you bring your client. Replacing bad participants and chasing down participants who are not fully engaged will end up costing you more than having a compelling incentive and well-screened participants in the first place, as well as keeping your project on schedule. “Pay peanuts, get monkeys” definitely applies here.

     
  10. Have an analysis plan, and design your study to make analysis easier and efficient.

    Our smartphone qual designers quite literally consider what kind of answers they’ll get to an assignment or question, and how those answers will be analyzed and presented as they work through their initial design. It’s like a research geek’s version of “beginning with the end in mind.” 

    You should ensure that you’re familiar with the tools you’ll be using to monitor and analyze your results well before the submissions from your participants start rolling in. If you’re using a new platform, or its your first smartphone qual project, make sure you're totally comfortable using it BEFORE your responses roll in. We actually set new clients up with access to a “demo” Project Viewing Portal so that we can make sure they’re up to speed with how it works, and the analysis tools they’ll be using before their fieldwork starts. And, we make sure that they have an analysis plan specifically based on their project’s design so they know what they'll be doing with the answers to all of the questions and assignments before the project even starts.

     
  11. Make your project entertaining and engaging for your clients.

    Remember that your clients are giving up the "focus group ritual" when you use smartphone qualitative. The "back room chatter" and the group focus that comes with it are a valuable part of the face-to-face research process.

    Happily, good smartphone qualitative platforms offer lots of ways to rally your key clients around the project and engage them with it so that they get great value out of it. Try "daily reports" with key submissions. "Buddy up" key clients with a research participant and make it easy for them to see their buddy participant's responses as they come in. Even let them ask follow-up questions and probes (with you as editor and controller) so that they feel involved and learn as the project unfolds.

    Use your "ripped from reality" photos, audios and videos to make a powerful presentation. Even use tools like online media collages that let your clients share and access key participant submissions long after the project is over.

     
  12. Respect the privacy of your participants.

    In many studies, we're asking participants to capture and share moments that are intensely private. So, it's crucial to be good guardians of the secrets of people who participate in studies.

    Ensure you're using a system that actively protects participant's identifiable information and everything they submit. If your research subject is intensely private, ensure you're working in a system that can keep participant identities entirely separate from what they submit. 

    If a submission looks so great that you can't resist using it outside of the project that collected it, make sure you can get the participant's explicit permission to use it. We find that most participants are happy to have their submissions shared if they can review what's going to be shared, and the exact way their submissions will be used.

    We've built technological, operational and organizational security at the highest level right in to the Over the Shoulder platform so that we can keep people's secrets safe, and help you make sure you're using their submissions appropriately and respectfully.

Everything You Always Wanted to Know About Smartphone Qualitative *But Were Afraid to Ask

The Over the Shoulder team has spent the better part of a decade building and honing the Over the Shoulder smartphone qualitative platform and helping qualitative research moderators & practitioners use it to do industry-leading insight work.

Along the way, we’ve fielded a lot of questions that qual practitioners wonder about.

So, we decided to start posting common questions and answers to them as a resource for everyone who wants to learn about smartphone qualitative, understand their options, and be inspired by the opportunities.

Please feel free to build, comment or post additional questions. We’ll try to update this blog at least weekly. We hope you’ll find it helpful and inspiring! 

- The Over the Shoulder team.


Q:

Can I use smartphone qualitative as part of a project that includes other methodologies like focus groups or live virtual interviews?

 

A:

Absolutely!

Hybrid project designs are powerful, and there are lots of ways to use smartphone qualitative in combination with other qualitative and quantitative methods. A few examples include:

Smartphone qualitative prior live sessions (in home, in facility, virtual, etc.). You might start your project with 60 participants doing a week of smartphone qualitative where they do things like share their home life, work life, usage occasions, or even visit a store to purchase and try a product. Participants submit assignments throughout the week, and you can review their video submissions, then select the best, most representative 20 participants to schedule live sessions with. It’s an amazing design that lets you get in-the-moment insight from 60 people, review and think through their submissions and be prepared to “go deep” in your live sessions. You can even show participants things they’ve submitted in the moment to spur their recall and lead to deep exploration in person.

Smartphone qualitative follow-ups to live sessions. Schedule traditional focus groups, then send your participants out of your live sessions to:

  • Gather in-the-moment insights through the Over the Shoulder app.
  • Send them on “buy & try” missions, where they'll locate and purchase specific products in their local retailer (or purchase them online), try them out in their home, and complete assignments that let you see everything from their in-store experience, purchase experience, and the moment they prepared and used the product in real0life situations.
  • Have them journal key moments based on your live discussion.
  • Send individual follow-up questions to participants based on what they've talked about in your live sessions.
  • Send a few questions that participants might have been too worn out or socially constrained from giving you an answer in your group sessions.

You can even select the best participants in your focus groups/interviews and invite only them to do a smartphone-based follow-up study. We’re happy to help make it all come off smoothly.

Add smartphone-based pre-work/homework to your live sessions. It’s easy to have your participants start your project with smartphone-based homework that they’ll complete before they come to your live sessions. Have them journal their snacking behavior, buy and try a few products, or have them share their personal story before you meet them face-to-face.

There are plenty more great ways to combine methodologies to get more insight and compelling “ripped from real-life” media to help tell a compelling story. As always, we’re happy to hear your ideas for hybrid projects and help you find a great design to deliver your objectives.


Q:

How should I be thinking about timelines for my smartphone qualitative project? How long does recruiting take? How long should I leave for the project design & project programming process?

A:

We’ll always do our utmost to meet your timelines regardless of the circumstances. That said, here are some guidelines on timelines:

3 weeks is ideal.  3 weeks from commissioning your project to the start of your fieldwork gives you enough time for just about any recruit, it allows plenty of time to come up with a great design, get input from your clients throughout the process, and it gives OTS ample time to get your study programmed, tested and ready for the field. The 3-week timeline gives everyone involved enough time to thoroughly and meticulously think through your project and not feel rushed.

2 weeks is comfortable as long as your recruit is standard.

1 week is doable if you can get input and approvals on your design quickly and your recruit is not too challenging.

Some studies can be put together and recruited in less than a week, but it’s probably going to feel rushed.

The big driver of your timeline is usually how long it will take to do your recruit. While a small study (20-30 participants) with wide recruiting specs can often be recruited in just a few days, highly specialized recruits can take a few weeks or more.

Designing a typical project usually takes about a week including reviews, approvals and adjustments.

Programming and testing a project is easy to do 72 hours. Shorter is usually possible, and we’ll always accommodate you if we can.


Q:

Does Over the Shoulder have a panel? Why?

A:

Over the Shoulder does not have a panel. There are a few reasons. 

First, we committed early to building an open platform that can easily bring in participants from any source so that you have full flexibility to work with any panel provider, recruiting resource and lots of other recruiting approaches. Our mission is to help you do great smartphone qualitative, not sell you our panel.

Second, most of our clients need specialized recruits for their projects. We’ve learned through the years that “pre-recruited” smartphone panels just don’t have enough members to allow you to get good participants if you need specific age groups, regions, product usage specs or other recruiting requirements.

Third, most pre-recruited smartphone panels have real issues with participant quality, professional respondents and security.

Fourth, we respect and rely on our many recruiting partners, and having our own panel would put us in conflict with them.

Bottom line, every Over the Shoulder project gets a fresh, carefully screened, managed sample of exactly the participants you need. It takes a little longer and costs a little more, but experience has taught us that it gets you far better results and far fewer hassles.


Q:

Can Over the Shoulder do my recruiting for me?

A:

Yes!

Over the Shoulder has partnerships with recruiters that we can work with to get you high-quality, engaged participants for your project. We know which recruiters we can rely on to get exactly the right people into your project as quickly and cost-effectively as possible. If you’d like, we’ll even manage the process, write your recruitment screener, interact with your recruiter on your behalf to replace dropouts and include recruiting & incentives one simple invoice for you.

On the other hand, if you have a recruiting partner you love and you’d like to work with, that’s great too!  We’re happy to work with them to make your project go smoothly. We have support materials at the ready to help any recruiter be a great smartphone qual recruiter, and your Over the Shoulder project team will happily interact with any recruiting partner you choose to make your project successful.

And, if you have a panel of participants already, we can easily onboard them into your Over the Shoulder project.


Q:

I know my objectives and my insight needs, but I’m not sure that I can (or want to) figure out how to design a smartphone qualitative project. Can Over the Shoulder help me with that?

A:

Of course! Over the Shoulder is proud to employ the best smartphone project designers in the business. Our average project designer has 6.5 years of experience designing smartphone qualitative (not bad considering smartphone qualitative has only been around for about 9 years!) and hundreds of projects under their belt.

We offer design support at three levels:

  1. Full Project Design

    It’s the easy way to a killer design. You start by sharing your objectives, insight needs and ideas for your project with your Project Designer. Your Project Designer carefully listens to what you’re trying to learn, and puts together an outline of how to answer your objectives, applying everything they know about the Over the Shoulder system, and proposes an outline for your design. You review it, shape it, add your thoughts and ideas, and when you love the outline, your Project Designer drafts a question-by-question, prompt-by-prompt smartphone discussion guide for you. Review it, get feedback and approvals, and presto…you have a great design every time.
     
  2. Design Assistance and Review

    Want to have more control and save some money? No problem. We’ll assign you a Project Designer who can run you through the basics, give you some ideas and send you off with our easy-to-use project-building template. You write up your smartphone discussion guide exactly how you want it. Your Project Designer is there to support you, and will review your project to give you any final suggestions and make sure it makes the most out of the platform.
     
  3. Design-it-yourself

    If you’re comfortable with the Over the Shoulder platform and smartphone qual design, you can draft up your project in our easy-to-use template.

Q:

Does Over the Shoulder do analysis, presentation building or do projects directly forend clients?

A:

No. Over the Shoulder is here to support qual practitioners, not to compete with them.

Early in our history, we used our own platform and provided analysis and reporting to end clients.

Now, we focus on developing the world’s best smartphone qual platform and teaching practitioners how to use it. 

We don’t do analysis, but we’ll provide you with the tools and training to do it yourself. We don’t write reports or give presentations to end clients, but we’re happy to help you look like a rockstar.

That keeps us plenty busy, and ensures that the quallies we support never have to worry about us competing with them for projects. Seems fair to us.