limitations of survey research

20 Advantages and Disadvantages of Survey Research

Survey research is a critical component of measurement and applied social research. It is a broad area that encompasses many procedures that involve asking questions to specific respondents.

A survey can be anything from a short feedback form to intensive, in-depth interviews that attempt to gather specific data about situations, events, or circumstances. Although there are several methods of application that researchers can apply using this tool, you can divide surveys into two generic categories: interviews and questionnaires.

Innovations in this area in recent years allow for advanced software solutions to provide more data to researchers because of the availability of online and mobile surveys. That means the people who are in the most challenging places to reach can still provide feedback on critical ideas, services, or solutions.

Several survey research advantages and disadvantages exist, so reviewing each critical point is necessary to determine if there is value in using this approach for your next project.

List of the Advantages of Survey Research

1. It is an inexpensive method of conducting research. Surveys are one of the most inexpensive methods of gathering quantitative data that is currently available. Some questionnaires can be self-administered, making it a possibility to avoid in-person interviews. That means you have access to a massive level of information from a large demographic in a relatively short time. You can place this option on your website, email it to individuals, or post it on social media profile.

Some of these methods have no financial cost at all, relying on personal efforts to post and collect the information. Robust targeting is necessary to ensure that the highest possible response rate becomes available to create a more accurate result.

2. Surveys are a practical solution for data gathering. Surveys or a practical way to gather information about something specific. You can target them to a demographic of your choice or manage them in several different ways. It is up to you to determine what questions get asked and in what format. You can use polls, questionnaires, quizzes, open-ended questions, and multiple-choice to collect info in real-time situations so that the feedback is immediately useful.

3. It is a fast way to get the results that you need. Surveys provide fast and comfortable results because of today’s mobile and online tools. It is not unusual for this method of data collection to generate results in as little as one day, and sometimes it can be even less than that depending on the scale and reach of your questions. You no longer need to wait for another company to deliver the solutions that you need because these questionnaires give you insights immediately. That means you can start making decisions in the shortest amount of time possible.

4. Surveys provide opportunities for scalability. A well-constructed survey allows you to gather data from an audience of any size. You can distribute your questions to anyone in the world today because of the reach of the Internet. All you need to do is send them a link to the page where you solicit information from them. This process can be done automatically, allowing companies to increase the efficiency of their customer onboarding processes.

Marketers can also use surveys as a way to create lead nurturing campaigns. Scientific research gains a benefit through this process as well because it can generate social insights at a personal level that other methods are unable to achieve.

5. It allows for data to come from multiple sources at once. When you construct a survey to meet the needs of a demographic, then you have the ability to use multiple data points collected from various geographic locations. There are fewer barriers in place today with this method than ever before because of the online access we have around the world.

Some challenges do exist because of this benefit, namely because of the cultural differences that exist between different countries. If you conduct a global survey, then you will want to review all of the questions to ensure that an offense is not unintentionally given.

6. Surveys give you the opportunity to compare results. After researchers quantify the information collected from surveys, the data can be used to compare and contrast the results from other research efforts. This benefit makes it possible to use the info to measure change. That means a questionnaire that goes out every month or each year becomes more valuable over time.

When you can gather a significant amount of data, then the picture you are trying to interpret will become much clearer. Surveys provide the capability of generating new strategies or identifying new trends to create more opportunities.

7. It offers a straightforward analysis and visualization of the data. Most surveys are quantitative by design. This process allows for the advantage of a straightforward analysis process so that the results can be quickly visualized. That means a data scientist doesn’t need to be available to start the work of interpreting the results. You can take advantage of third-party software tools that can turn this info into usable reports, charts, and tables to facilitate the presentation efforts.

8. Survey respondents can stay anonymous with this research approach. If you choose to use online or email surveys, then there is a fantastic opportunity to allow respondents to remain anonymous. Complete invisibility is also possible with postal questionnaires, allowing researchers to maximize the levels of comfort available to the individuals who offer answers. Even a phone conversation doesn’t require a face-to-face meeting, creating this unique benefit.

When people have confidence in the idea that their responses will not be directly associated with their reputation, then researchers have an opportunity to collect information with greater accuracy.

9. It is a research tool with fewer time constraints. Surveys have fewer time limits associated with them when compared to other research methods. There is no one on the other end of an email or postal questionnaire that wants an immediate answer. That means a respondent can take additional time to complete each answer in the most comfortable way possible. This benefit is another way to encourage more honesty within the results since having a researcher presence can often lead to socially desirable answers.

10. Surveys can cover every component of any topic. Another critical advantage that surveys provide is the ability to ask as many questions as you want. There is a benefit in keeping an individual questionnaire short because a respondent may find a lengthy process to be frustrating. The best results typically come when you can create an experience that involves 10 or fewer questions.

Since this is a low-cost solution for gathering data, there is no harm in creating multiple surveys that have an easy mode of delivery. This benefit gives you the option to cover as many sub-topics as possible so that you can build a complete profile of almost any subject matter.

List of the Disadvantages of Survey Research

1. There is always a risk that people will provide dishonest answers. The risk of receiving a dishonest answer is lower when you use anonymous surveys, but it does not disappear entirely. Some people want to help researchers come to whatever specific conclusion they think the process is pursuing. There is also a level of social desirability bias that creeps into the data based on the interactions that respondents have with questionnaires. You can avoid some of this disadvantage by assuring individuals that their privacy is a top priority and that the process you use prevents personal information leaks, but you can’t stop this problem 100% of the time.

2. You might discover that some questions don’t get answers. If you decide to use a survey to gather information, then there is a risk that some questions will be left unanswered or ignored. If some questions are not required, then respondents might choose not to answer them. An easy way to get around this disadvantage is to use an online solution that makes answering questions a required component of each step. Then make sure that your survey stays short and to the point to avoid having people abandon the process altogether.

3. There can be differences in how people understand the survey questions. There can be a lot of information that gets lost in translation when researchers opt to use a survey instead of other research methods. When there is not someone available to explain a questionnaire entirely, then the results can be somewhat subjective. You must give everyone an opportunity to have some understanding of the process so that you can encourage accurate answers.

It is not unusual to have respondents struggle to grasp the meaning of some questions, even though the text might seem clear to the people who created it. Whenever miscommunication is part of the survey process, the results will skew in unintended directions. The only way to avoid this problem is to make the questions as simple as possible.

4. Surveys struggle to convey emotions with the achievable results. A survey does not do a good job of capturing a person’s emotional response to the questions then counter. The only way to gather this information is to have an in-person interview with every respondent. Facial expressions and other forms of body language can add subtlety to a conversation that isn’t possible when someone is filling out an online questionnaire.

Some researchers get stuck trying to interpret feelings in the data they receive. A sliding-scale response that includes various levels of agreement or disagreement can try to replicate the concept of emotion, but it isn’t quite the same as being in the same room as someone. Assertion and strength will always be better information-gathering tools than multiple-choice questions.

5. Some answers can be challenging to classify. Surveys produce a lot of data because of their nature. You can tabulate multiple-choice questions, graph agreement or disagreement in specific areas, or create open-ended questions that can be challenging to analyze. Individualized answers can create a lot of useful information, but they can also provide you with data that cannot be quantified. If you incorporate several questions of this nature into a questionnaire, then it will take a long time to analyze what you received.

Only 10% of the questions on the survey should have an open-ended structure. If the questions are confusing or bothersome, then you might find that the information you must manually review is mostly meaningless.

6. You must remove someone with a hidden agenda as soon as possible. Respondent bias can be a problem in any research type. Participants in your survey could have an interest in your idea, service, or product. Others might find themselves being influenced to participate because of the subject material found in your questionnaire. These issues can lead to inaccurate data gathering because it generates an imbalance of respondents who either see the process as overly positive or negative.

This disadvantage of survey research can be avoided by using effective pre-screening tools that use indirect questions that identify this bias.

7. Surveys don’t provide the same level of personalization. Any marketing effort will feel impersonal unless you take the time to customize the process. Because the information you want to collect on a questionnaire is generic by nature, it can be challenging to generate any interest in this activity because there is no value promised to the respondent. Some people can be put off by the idea of filling out a generic form, leading them to abandon the process.

This issue is especially difficult when your survey is taken voluntarily online, regardless of an email subscription or recent purchase.

8. Some respondents will choose answers before reading the questions. Every researcher hopes that respondents will provide conscientious responses to the questions offered in a survey. The problem here is that there is no way to know if the person filling out the questionnaire really understood the content provided to them. You don’t even have a guarantee that the individual read the question thoroughly before offering a response.

There are times when answers are chosen before someone fully reads the question and all of the answers. Some respondents skip through questions or make instant choices without reading the content at all. Because you have no way to know when this issue occurs, there will always be a measure of error in the collected data.

9. Accessibility issues can impact some surveys. A lack of accessibility is always a threat that researchers face when using surveys. This option might be unsuitable for individuals who have a visual or hearing impairment. Literacy is often necessary to complete this process. These issues should come under consideration during the planning stages of the research project to avoid this potential disadvantage. Then make the effort to choose a platform that has the accessibility options you need already built into it.

10. Survey fatigue can be a real issue that some respondents face. There are two issues that manifest themselves because of this disadvantage. The first problem occurs before someone even encounters your questionnaire. Because they feel overwhelmed by the growing number of requests for information, a respondent is automatically less inclined to participate in a research project. That results in a lower overall response rate.

Then there is the problem of fatigue that happens while taking a survey. This issue occurs when someone feels like the questionnaire is too long or contains questions that seem irrelevant. You can tell when this problem happens because a low completion rate is the result. Try to make the process as easy as possible to avoid the issues with this disadvantage.

Surveys sometimes have a poor reputation. Researchers have seen response rates decline because this method of data gathering has become unpopular since the 1990s. Part of the reason for this perception is due to the fact that everyone tries to use it online since it is a low-cost way to collect information for decision-making purposes.

That’s why researchers are moving toward a rewards-based system to encourage higher participation and completion rates. The most obvious way to facilitate this behavior is to offer something tangible, such as a gift card or a contest entry. You can generate more responses by creating an anonymous process that encourages direct and honest answers.

These survey research advantages and disadvantages prove that this process isn’t as easy as it might see from the outside. Until you sit down to start writing the questions, you may not entirely know where you want to take this data collection effort. By incorporating the critical points above, you can begin to craft questions in a way that encourages the completion of the activity.

Skip navigation

World Leaders in Research-Based User Experience

10 survey challenges and how to avoid them.

February 26, 2023 2023-02-26

• Email article
• Share on LinkedIn
• Share on Twitter

How recently have you been invited to take some kind of survey? It likely hasn’t been more than a couple of days.

Our research shows that surveys are among the most commonly used research methods by UX practitioners. Surveys have their place among the various quantitative UX-research methods , but it is more challenging to create a good survey than many professionals think. It is very easy to write a bad survey that gathers flawed data.

In This Article:

Limitations of surveys, 1. recall bias, 2. recency bias, 3. social-desirability bias, 4. prestige bias, 5. acquiescence bias, 6. order effect, 7. current-mood or emotional-state bias, 8. central-tendency bias, 9. demand characteristics, 10. random-response bias, survey participants can filter responses.

Unlike observational methods , which reveal real-time behaviors that cannot be easily faked or filtered (such as the ease or difficulty of navigating a new mobile app), surveys are self-reported – which means that respondents get to filter everything they share with researchers before they share it. This is a major limitation of surveys, no matter how well they are written — because researchers are generally interested in phenomena as they really occur , not in (intentionally or unintentionally) censored information. Other self-report research methods where users can decide what they share include interviews , focus groups , and diary studies .

Researchers Cannot Probe in Surveys

Another limitation of surveys is that researchers cannot probe to better understand responses. This is because respondents usually complete them without the researcher being present. Even if researchers could afford to watch every single respondent take a survey, doing so would bias the responses and contaminate the data. Moreover, survey responses generally come as ratings on a simple scale or as selections on multiple-choice questions. It is impossible to know why someone responded the way they did without more information (a limitation shared by analytics data). Even if there is space provided for a respondent to type out additional comments, these open-ended responses tend to be brief, disconnected, or skipped entirely.

Surveys Only Collect Attitudinal Data

Surveys collect attitudinal data , representing how users think and feel — not how they behave. They are no substitute for observational methods such as usability testing or analytics , that can accurately reveal user behaviors. Surveys remove respondents from workflows and require them to reflect on experiences and behaviors rather than demonstrate them. For example, even if a survey asks users how often they tend to access a particular feature on a website, it is really asking them how often they think they access that feature. Questions about behaviors do not accurately capture how people actually perform those behaviors; instead, they reveal users’ perceptions and recollections about those behaviors. Use surveys to learn what users think and feel, not what they do.

Consider the following excerpt from a survey emailed to university students regarding a financial dashboard for monitoring and paying school-related expenses. The likely purpose of this survey is to gain insights about the usability of the dashboard. Unfortunately, respondents taking this survey are obviously not using the dashboard; moreover, depending on when they used the dashboard, their recollections about the UI will be incomplete, vague, or wrong. As a result, the survey responses cannot accurately reflect how “user friendly” the dashboard is. Research goals investigating usability, findability, discoverability, or other behavioral metrics must be studied with observational methods, such as user testing. Survey data can be used to complement performance-based data, but it cannot, by itself, offer a comprehensive assessment of the usability of a system.

Although developing a good survey can take time and multiple iterations , it is quick and cheap compared to many other research methods — which is why surveys have become so popular. Surveys can be useful for gathering large amounts of both quantitative and qualitative data when researchers are investigating what users think or feel. However, to gather good data, a survey must be written well to be both internally and externally valid . If a survey is not written well, it will produce bad data.

The following 10 points describe common ways in which research participants tend to misrepresent their true thoughts and feelings on surveys. While it is impossible to fully mitigate the effects of these tendencies, there is much that can be done to lessen their effects.

Although people tend to feel that they can accurately remember how things were in the past, they actually forget most of the details as time goes on , and their feelings about things change without them noticing. Imperfect memory presents a great challenge for any survey that focuses on a specific event or experience but is completed after the fact. Observational methods that capture insights as experiences occur do not suffer from recall bias in the same way that surveys do. People are most easily able to remember details about things that were recent, that they think about frequently, and that are associated with strong emotions.

Preventive tip:

Distribute a survey to participants as soon as possible following the event related to your research. For example, if your survey asks why a user subscribed to a particular service, have the subscription act as a trigger for the survey, while their rationale for subscribing is as fresh as possible. Don’t wait to send the survey to large groups of subscribers at once.

(Still, even if you follow this tip, there is no guarantee that people will complete the survey right away.)

Even better, consider using surveys in connection to an observational user-research method such as usability testing. For example, you could show a survey such as the SUS to users who have completed a test of a website. In this way, the experience would be fresh in their mind as they respond to the questions, and you will be able to use the survey-provided attitudinal data to complement other performance-based metrics such as success rates and task times.

Individuals tend to give more weight to recent events than to events from longer ago. When asked about an overall experience or opinion on a survey, it is likely that people will respond based on how they have been feeling lately rather than taking an accurate mental average of their feelings throughout time.

Preventive tips:

Ask about both a recent experience and a previous one. If the research question focuses on opinions and attitudes at multiple time points, then intentionally capture someone’s most recent feelings first, before prompting them to consider how those feelings have changed with time.

Use a longitudinal method (such as a diary study) rather than a survey. If your research question investigates user attitudes or feelings at various time points, have participants respond at those points rather than relying on one survey at the end that will likely be skewed toward more recent events.

People are highly motivated to conform to societal norms and portray themselves in socially desirable ways. For this reason, participants might (both consciously or subconsciously) distort their answers to be closer to what they think is socially acceptable. For example, they might overemphasize how important environmental sustainability is in their lives because it is increasingly valued by society, or they might underemphasize how much they love a particular indulgence if it is looked down upon by society.

Emphasize confidentiality (the researcher knows their respondents’ identity but will not share it with others) or anonymity (not even the researcher knows the identity of respondents) of their answers. Reassuring participants that their responses will not be connected to their identity helps them feel comfortable with being candid and truthful.

Use indirect questioning. This method can allow respondents to be more honest without feeling as strong of a need to conform to societal norms For example, a survey investigating feelings about a particular political candidate might also ask about candidates with similar political views to the candidate in question. Respondents could also be asked to describe how their close friends or colleagues feel about an issue. Open-ended responses are often necessary to capture relevant data in these cases but run the risk of not fully answering the research question.

People do not like to be negatively portrayed. When given the opportunity, survey respondents tend to distort their responses to make themselves seem more impressive, smart, or successful. Common examples include the tendency to round up one’s income or downplay (or fully deny) negative actions such as violence or abuse. Because surveys are a self-report method, respondents will not always accurately represent their true opinions and actions.

When exact numbers are not necessary, provide ranges in response choices. Allowing someone to select a range that includes their true answer is more likely to be accurate than asking respondents to report it directly. For example, providing age or income ranges will avoid requiring respondents to directly input their age or salary and will be less likely to result in a distortion. In many cases, larger ranges increase the chances that someone will provide an honest response (e.g., 65–80 years old vs. 66–70, 71–75, 76–80 years old).

Use other data sources for sensitive data that is critical to your research. Often, researchers find it easier to ask participants to report information about themselves as part of a survey than to collect this data from other available sources. However, sensitive data such as income, weight, or achievements will be more accurate when gathered from another source, if available.

People tend to agree with statements more than disagree. For example, if asked whether they agree or disagree with the statement [Our company] provides high-quality products , a larger proportion of respondents is likely to select “agree” than “disagree,” no matter the company. The tendency to agree is often due to the natural desire to be nice to others; it can also serve as a shortcut and conserve mental energy. It can also be the case that a respondent has no strong reason to disagree and gives the benefit of the doubt, which will generally result in agreement . The acquiescence bias can lead researchers to reach falsely positive conclusions.

Ask a direct, open-ended question rather than asking for agreement or disagreement responses to a statement. People will be most likely to report their true feelings when there is no obvious way to simply “agree.”

Use a semantic-differential scale rather than a Likert scale. A semantic-differential scale still asks participants to provide a rating but provides a continuum of response options specific to the nature of the question. There is generally no obvious “agreement” option available.

Include reverse-keyed items. When respondents indicate whether they agree or disagree with multiple similar types of statements in a row, they are more likely to quickly select positive responses to all. Alternating the focus of statements or questions participants are responding to can serve as an indicator of whether the participant was intentionally reading and responding to each individual item. For example, instead of having participants indicate their agreement or disagreement with a series of positively framed statements (e.g., [This company] has my best interest in mind ) all in succession, intermix them with some negatively framed statements (e.g., [This company] intentionally tries to deceive its customers ). However, reverse-keying items should be done sparingly, only when there are many of the same response types in a row, as it increases the overall cognitive load required to respond and the probability that an error will be made when responding or coding the responses.

The order in which response options are presented in a closed-ended survey question (such as multiple-choice or multi-select) affects which options are most likely to be chosen. The options near the beginning and end of a list tend to be the most likely to be chosen because of the primacy and recency effects. (The primacy effect describes the tendency to select the first seemingly satisfactory option a respondent sees. The recency effect describes how the last, or “most recently seen,” option in a list is freshest on the mind and thus, the most easily available for selection.)

The order in which the questions in a survey are presented can also bias responses. Question order can unintentionally reveal the purpose of a survey, might create pressure for respondents to be consistent with themselves, or might cause end questions to be neglected due to survey fatigue.

Organize options meaningfully. When options can be ordered meaningfully (e.g., alphabetically, chronologically, locationally, temporally, or categorically), respondents are more likely to quickly identify the options that apply to them rather than simply picking those at the beginning and at the end. This is particularly true with long lists of options that are conventionally listed in a particular order.

Randomize the order of response options. When there is no obvious meaningful order for response options, randomizing their order gives each option an equal chance of falling near the beginning or end of a list. For example, it would be appropriate to randomize the order of different color options, but not ranges of ages.

Randomize questions if possible. If the order of your survey questions is not important, consider presenting them in various orders to different participants to avoid the same initial questions consistently influencing responses on later ones. You can also show questions on separate pages to avoid possible confounds.

A respondent’s current emotional state will affect how (and whether) they respond to a survey. If someone is feeling rushed, tired, or apathetic, they are unlikely to even begin a survey in the first place, let alone provide meaningful responses. Your first goal when deciding when to distribute a survey is to maximize the response rate . It is unwise to blast out a survey when the majority of potential respondents are likely to be busy (e.g., at the start of a workday), or at a point in the customer journey when someone is dealing with complex problems (conduct a diary study instead to study these complex parts of an experience).

Beyond this initial challenge, as with all research studies, those who take a survey will provide answers based on their current mood. In some cases, the content of a survey itself might induce a particular mood (such as asking about a recent negative experience), while in others the content is neutral. Unlike observational methods, surveys cannot capture the mood of a respondent (even if you have a question about it!) — which prevents this information from being factored into the analysis and conclusions. This is one reason why survey data cannot be interpreted as a perfect reflection of reality. Calculating the statistical significance and confidence intervals  for quantitative findings can help account for this variability.

In cases where a survey is meant to capture overall impressions, it is not critical exactly when a participant takes a survey so long as they are able to relax, focus, and think clearly. In other cases (such as with the System Usability Scale, SUS, and the Single Ease Questionnaire, SEQ), the current mood of the participant is a valuable part of the data to be collected. These questionnaires are intended to capture a user’s reactions in the moments following the completion of a whole test or a single task, and thus must be completed while the relevant mood lingers.

Conduct a think-aloud test to understand how your survey makes respondents feel. Have a handful of participants take the survey and share how the questions make them feel. This approach is particularly useful when the content of a survey is likely to arouse strong emotional reactions (for example, if the survey is asking about a difficult experience or a controversial topic).

Encourage participants to take the survey in a particular mood. While this is unrealistic for a simple Net Promoter Score (NPS) that pops up in an email or notification, for longer, more significant surveys it can be valuable to encourage participants to set aside a time when they can focus and relax — particularly when you will be paying them an incentive.

Utilize customer-journey maps to identify opportunities for survey distribution. Just like fishermen studying the fish they intend to catch to know the best time and position for casting nets, researchers must also be intentional about the time and manner in which they distribute surveys. Customer-journey data can help a researcher to understand the current moods of users as they interact with the organization. However, there is obviously no way to predict the personal life circumstances of each participant.

People are often hesitant to provide extreme responses on rating scales. When survey respondents are asked to select a response on a scale, they often lean toward the middle, no matter how many points there are on the scale.

Because of respondents’ hesitancy to select the extremes (even if their circumstances likely merit such responses), large sample sizes generally lead to somewhat normally distributed data on rating scales. Response scales with an odd number of options inevitably have a middle point which is often labeled as “neutral” or “neither agree nor disagree.” While this middle option is a legitimate response, it also becomes an easy “out” for respondents who don’t want to spend the effort necessary to formulate a true response to the presented statement.

Use an even number of response options to encourage participants to lean one way or the other. Receiving many neutral responses will not end up being very helpful for researchers. If a neutral option is needed during analysis, the middle two points can be combined.

Use as few points on the response scale as are meaningful. You only want to have as many points on a response scale as are legitimately distinct and meaningful for the question you are asking. Simply having more points on a scale (such as 7 or more) does not guarantee you will be able to capture more nuanced and informative data. The differences between the responses to options such as “Very strongly agree” and “Strongly agree” on a 9-point scale will mean very little because participants are unlikely to have such specific feelings about most topics. We recommend using a 4-point or 6-point scale in most cases because they are straightforward to answer and analyze, and will capture most meaningful differences.

When participants in any type of research study become aware of the researcher’s aims and objectives, they are more likely to change their behaviors or responses accordingly. Participants might respond in specific ways in order to influence the outcomes of the study. This is particularly true when it can be personally beneficial to respond in specific ways. For example, professional participants might try to respond to recruiting screeners (which are a type of survey) in ways that might increase their chances of being selected for a study — regardless of whether they are a good fit.

Some participants intentionally provide extreme or untrue responses if they have some reason to be frustrated with the distributor of the survey, the organization associated with it, or the aims of the study.

Other participants (especially those with strong brand loyalty and positive feelings towards the survey distributor) might try to provide the responses they think the researchers are looking for out of goodwill, attempting to be “helpful.” This also tends to happen when respondents have a personal tie to the researcher — which is why it is important to recruit real users rather than colleagues or friends for any good research study.

Hide the true purpose of the survey. Avoid putting too much detail about the purpose of the survey in introductory materials such as emails, notifications, or survey titles and descriptions. It is also wise to mask any response options or survey questions that are critical to the purpose of the study by placing them among other plausible, but less critical options and questions.

Distribute the survey to various types of users. Distributing a survey to users who have had both positive and negative experiences, as well as those familiar and unfamiliar with your brand can potentially offset the demand characteristics described above.

When survey respondents do not know the answer to a question, they generally just guess. This is problematic because guessed responses are not accurate data but are impossible to identify. For example, if a survey asked how much time was spent researching a product before purchasing, respondents are likely to just guess since they have no way of accurately knowing — especially those who spent multiple days or weeks researching.

Respondents may also engage in guessing or selecting arbitrary options when they become fatigued or simply want to finish a survey for the offered incentive.

Provide an alternative response for those who do not know the answer. Closed-ended survey questions do not allow respondents to indicate that they do not have a meaningful response unless an option such as “none,” “other,” or “not applicable” is provided. However, it is not necessary to provide such options on all questions if it is likely that all respondents will have a meaningful answer. These types of alternatives are an easy way for people to opt out of responding, without fully considering the question (another reason not to have “neutral points in the middle of your scales).

Include reverse-keyed items. Respondents who answer randomly often select the same option for all similar types of questions. Including reverse-keyed items can help you recognize when a participant answered quickly or randomly, without reading the questions. Once again, however, this should be done sparingly, and only when there are many of the same response types in a row.

Keep surveys short . If respondents start guessing because they are getting bored or tired, your survey is just too long.

A well-designed survey can quickly and inexpensively gather many valuable insights, but even the best surveys are subject to response biases. Much of what we know about large groups of people has been gathered through surveys. However, surveys are not always the right method for gaining valuable UX insights and making improvements to digital designs. All survey data must be analyzed with a critical eye and not simply accepted as the truth.

Davies, R. S. (2020). Designing Surveys for Evaluations and Research . EdTech Books. https://edtechbooks.org/designing_surveys

Related Courses

Survey design and execution.

Learn how to use surveys to drive and evaluate UX design

How to Interpret UX Numbers: Statistics for UX

When research data should be trusted; what statistics to use when

Remote User Research

Collect insights without leaving your desk

Related Topics

• Research Methods Research Methods
• Behavior Patterns

Learn More:

Please accept marketing cookies to view the embedded video. https://www.youtube.com/watch?v=zaK4xfAqpY0

Survey Response Biases in User Research

How to Present UX Research Results Responsibly

Caleb Sponheim · 3 min

Why the UX Team Doesn't Get the Credit

Tanner Kohler · 4 min

Card Sorting: Why & When

Samhita Tankala · 3 min

Related Articles:

Should You Run a Survey?

Maddie Brown · 6 min

User-Feedback Requests: 5 Guidelines

Anna Kaley · 10 min

Rating Scales in UX Research: Likert or Semantic Differential?

Maria Rosala · 7 min

Confidence Intervals, Margins of Error, and Confidence Levels in UX

Raluca Budiu · 7 min

Hiring Interviews Are Terrible: Smart UX Teams Structure Them

Evan Sunwall · 11 min

Why You Cannot Trust Numbers from Qualitative Usability Studies

Raluca Budiu · 9 min

• Survey Software The world’s leading omnichannel survey software
• Online Survey Tools Create sophisticated surveys with ease.
• Mobile Offline Conduct efficient field surveys.
• Text Analysis
• Close The Loop
• Automated Translations
• NPS Dashboard
• CATI Manage high volume phone surveys efficiently
• Cloud/On-premise Dialer TCPA compliant Cloud & on-premise dialer
• IVR Survey Software Boost productivity with automated call workflows.
• Analytics Analyze survey data with visual dashboards
• Panel Manager Nurture a loyal community of respondents.
• Survey Portal Best-in-class user friendly survey portal.
• Voxco Audience Conduct targeted sample research in hours.

Find the best survey software for you! (Along with a checklist to compare platforms)

Get Buyer’s Guide

• 100+ question types
• Drag-and-drop interface
• Skip logic and branching
• Multi-lingual survey
• Text piping
• Question library
• CSS customization
• White-label surveys
• Customizable ‘Thank You’ page
• Customizable survey theme
• Reminder send-outs
• Survey rewards
• Social media
• Website surveys
• Correlation analysis
• Cross-tabulation analysis
• Trend analysis
• Real-time dashboard
• Customizable report
• Email address validation
• Recaptcha validation
• SSL security

Take a peek at our powerful survey features to design surveys that scale discoveries.

Download feature sheet.

• Hospitality
• Financial Services
• Academic Research
• Customer Experience
• Employee Experience
• Product Experience
• Market Research
• Social Research
• Data Analysis

Explore Voxco 

Need to map Voxco’s features & offerings? We can help!

Watch a Demo 

Download Brochures 

Get a Quote

• NPS Calculator
• CES Calculator
• A/B Testing Calculator
• Margin of Error Calculator
• Sample Size Calculator
• CX Strategy & Management Hub
• Market Research Hub
• Patient Experience Hub
• Employee Experience Hub
• Market Research Guide
• Customer Experience Guide
• The Voxco Guide to Customer Experience
• NPS Knowledge Hub
• Survey Research Guides
• Survey Template Library
• Webinars and Events
• Feature Sheets
• Try a sample survey
• Professional services

Find the best customer experience platform

Uncover customer pain points, analyze feedback and run successful CX programs with the best CX platform for your team.

Get the Guide Now

We’ve been avid users of the Voxco platform now for over 20 years. It gives us the flexibility to routinely enhance our survey toolkit and provides our clients with a more robust dataset and story to tell their clients.

VP Innovation & Strategic Partnerships, The Logit Group

• Client Stories
• Voxco Reviews
• Why Voxco Research?
• Careers at Voxco
• Vulnerabilities and Ethical Hacking

Explore Regional Offices

• Cloud/On-premise Dialer TCPA compliant Cloud on-premise dialer
• Predictive Analytics
• Customer 360
• Customer Loyalty
• Fraud & Risk Management
• AI/ML Enablement Services
• Credit Underwriting

Get Buyer’s Guide

• SMS surveys
• Banking & Financial Services
• Retail Solution
• Risk Management
• Customer Lifecycle Solutions
• Net Promoter Score
• Customer Behaviour Analytics
• Customer Segmentation
• Data Unification

Explore Voxco 

Watch a Demo 

Download Brochures 

• CX Strategy & Management Hub
• Blogs & White papers
• Case Studies

VP Innovation & Strategic Partnerships, The Logit Group

• Why Voxco Intelligence?
• Our clients
• Client stories
• Featuresheets

The Methodological Limitations of Survey Research

SHARE THE ARTICLE ON

As a tool to evaluate the viability of any and all interventions, efforts, and strategies, surveys offer several advantages, such as being accessible, easy, flexible, and cost-effective. You can administer a survey to reach a large and diverse audience worldwide, increasing the validity, accuracy, and generalizability of your findings. 

In this blog, we’ll focus on the limitations of survey research and provide ways you can overcome those to ensure an efficient and effective survey. 

What is Survey Research?

Survey research is a method of collecting data from a predefined group of respondents, or sample, via surveys. Researchers and organizations widely use survey research to better understand people, consumers, and societies.  

The process involves systematically gathering opinions, insights, attitudes, behaviors, or characteristics through online surveys or interviews. The insights help us understand human behavior and preferences across various industries like market research, healthcare, social sciences, and public opinion polls. 

While surveys offer various advantages, they are not without limitations. In the following sections, we will explore the methodological limitations of survey research. We have categorized the challenges into three groups to help you recognize which of these might be impacting your survey results. 

Read how Voxco helped Brain Research improve research productivity by 60%.

Types of limitations of survey research.

What are the types of limitations of survey research? The limitations of survey research can be categorized into three groups: survey constraints, survey errors, and survey-related effects.

The following three are the limitations of survey research that can impact the reliability, validity, and accuracy of your result. Recognizing these gaps can help you mitigate their effects. 

1. Survey errors: 

These refer to the discrepancies in your survey findings caused due to factors like sampling bias, response bias, or measurement errors during the data collection process. Survey errors may undermine the validity of the result. 

2. Survey constraints: 

These limitations arise from practical concerns like budget, time, and resources. These constraints may restrict the scope and quality of your survey research. 

3. Errors during survey process: 

These limitations include issues introduced by survey design or administration process. It can affect the representativeness of the survey result. 

Exploring the Limitations of Survey Research

Let’s explore these limitations of survey research in detail to understand what might be impacting the survey result. 

1. Survey errors

Survey errors can be further categorized into the following three groups:

A. Sampling bias:

This error can occur if the sample selected for your survey is not an accurate representation of the target population, resulting in a biased estimate of the population. 

For example, a retail survey only targets urban customers and ignores rural or lower-income demographics. The survey result may not accurately reflect the opinions and characteristics of the entire customer base. 

B. Response bias: 

This limitation of survey research arises when respondents share misleading information due to factors like social desirability or acquiescence. 

For example, respondents may empathize with their likelihood of purchasing a product that is socially accepted/celebrated. This can skew the real market predictions. 

C. Measurement error:

Gathering inaccurate data due to factors like ambiguous questions or response options can lead to measurement errors. 

For example, if a market research survey asks leading questions about a customer’s purchase behavior, the respondent may provide an inaccurate response. This can compromise the reliability of the data. 

D. Survey administration issues: 

This includes errors in the data collection process. Factors like incomplete surveys or data entry mistakes are a few examples of survey administration issues. 

For example, if a research firm fails to update the contact list, it can lead to misrepresentation of the target population. 

Also read: Accelerate Survey Research with Voxco Audience

Application of the Quasi-Experimental Design

Quasi-experimental studies have lower internal validity than true experiments and also cannot establish a causal relationship between variables as effectively. So why do researchers use it? 

There are certain situations where the use of a quasi-experimental design is more suited to the study. This is especially true for studies where it would be unethical to withhold treatment from a subject on a random basis. In such situations, researchers can utilize quasi-experimental design to circumvent any ethical issues.

Additionally, another context in which a quasi-experimental design is more appropriate is when the true experiment design is not feasible. This could be due to the high expenses associated with true experiments. It could also be due to the fact that true experiments generally involve a lot of work to effectively design an experimental intervention for the threshold of subjects required to justify the research as a true experiment.

Uncover valuable insights from the pre-screened audience with Voxco Audience.

Accelerate research time with high-quality samples.

10Mn+ respondents | 90+ profiling points | 20-30% ad hoc saving

2. Survey constraints 

Some other limitations of survey research can be related to resource, budget, and time constraints. 

A. Resource constraints: 

Limited availability of technology, human resources, and expertise may impact the survey process, from survey design to data analysis.

B. Time constraints: 

Unattainable deadlines can compromise the thoroughness of survey design, data collection, and analysis. A tight time frame can result in utilizing convenience sampling methods, sacrificing the representativeness of the data. 

C. Budget constraints: 

 Insufficient funding can restrict the use of the right technology. It can limit the scope and quality of the research project. 

Survey-related effects limit the precision of the conclusions that can be drawn from collected data. These are a few different survey-related effects:

Question-related: 

Although there is no single ‘correct’ way to word a question, different questions can yield different responses. Factors like wording, orderings, or formatting of your survey question can impact a respondent’s interpretation of the question, resulting in distorted data. 

Survey mode effects: 

This limitation of survey research occurs when the mode of survey administration influences your respondent’s behavior, responses, or attitude. For example, relying on online surveys exclusively may exclude the elderly or remote demographic, skewing the demographic representation. 

Overcoming Limitations of Survey Research

Recognizing the limitations of survey research can help you enhance the credibility and validity of the data, ensuring that decisions are made based on well-informed data. 

1. Sampling bias: 

Implementing various sampling methods like stratified sampling can help you ensure the respondent group is representative of the target population. Additionally, employing a market research panel lik Voxco Audience gives you access to verified user profiles so you can create your survey sample that represents the target audience. 

2. Response bias: 

Using validated scale questions and providing assurance of anonymity can encourage honest responses and candor among respondents and reduce social desirability or acquiescence bias. Moreover, leveraging diverse data collection methods can help reduce response bias. 

3. Measurement error: 

Utilizing skip-logic to tailor questions based on respondents’ answers, pretesting survey questionnaires, and refining survey introductions can help reduce measurement errors. Offering clear instructions and response options and asking questions relevant to individual respondents can enhance data quality. 

A robust tool like Voxco’s enables you to create a clear and concise survey by utilizing skip logic, question randomization, and customization to minimize measurement errors. 

Voxco helps the top 50 MR firms & 450+ global brands gather omnichannel feedback, measure sentiment, uncover insights, and act on them.

See how Voxco can enhance your research efficiency.

4. Survey administration issues: 

Implement survey testing and quality checks, monitor survey performance in real-time, and test survey responsiveness on all devices to mitigate survey administration issues. 

5. Survey question errors: 

Utilize clear and unambiguous language in your survey questions and ensure that your content is comprehensive and relevant to the general population. Additionally, various question formats should be used to avoid leading or biased questions and enhance data quality. 

6. Survey mode effects: 

Use modern survey software that empowers you with the flexibility of using various survey modes to conduct research. Voxco, for instance, offers three survey modes – online survey tool, phone surveys, and mobile-offline surveys. Leverage a platform that allows you to utilize different modes and gather data centrally to ensure seamless survey research. 

Survey research serves as a powerful tool to make informed decisions based on insights gathered directly from the target audience. However, the limitations of survey research can compromise the accuracy, reliability, and validity of the result. By understanding and proactively addressing these limitations, you can unlock the real potential of your survey data. 

Explore Voxco Survey Software

+ Omnichannel Survey Software 

+ Online Survey Software 

+ CATI Survey Software 

+ IVR Survey Software 

+ Market Research Tool

+ Customer Experience Tool 

+ Product Experience Software 

+ Enterprise Survey Software 

Experimental Research

Definition, Examples and Types of Experimental Research Designs SHARE THE ARTICLE ON Table of Contents What is Experimental Research ? Experimental research is a scientific

Column Chart

Column Chart SHARE THE ARTICLE ON Share on facebook Share on twitter Share on linkedin Table of Contents A column chart is a type of

How to Improve Customer Experience (CX)

How to Improve Customer Experience (CX) SHARE THE ARTICLE ON Table of Contents What is Customer Experience (CX)? Customer experience, commonly referred to as CX,

Outbound predictive dialers

Harness The Power of Outbound Predictive Dialers SHARE THE ARTICLE ON Table of Contents If your research relies on outbound dialing, predictive dialers are your

The Role of Panel Surveys in Ensuring High Data Quality and Consistency

The Role of Panel Surveys in Ensuring High Data Quality and Consistency SHARE THE ARTICLE ON Table of Contents Introduction When it comes to research

What is a customer journey?

What is a customer journey? SHARE THE ARTICLE ON Table of Contents A customer journey is just what it sounds like. It is taking your

We use cookies in our website to give you the best browsing experience and to tailor advertising. By continuing to use our website, you give us consent to the use of cookies. Read More

• Search Menu
• Advance articles
• Editor's Choice
• Supplements
• French Abstracts
• Portuguese Abstracts
• Spanish Abstracts
• Author Guidelines
• Submission Site
• Open Access
• About International Journal for Quality in Health Care
• About the International Society for Quality in Health Care
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Contact ISQua
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

What is survey research, advantages and disadvantages of survey research, essential steps in survey research, research methods, designing the research tool, sample and sampling, data collection, data analysis.

• < Previous

Good practice in the conduct and reporting of survey research

• Article contents
• Figures & tables
• Supplementary Data

KATE KELLEY, BELINDA CLARK, VIVIENNE BROWN, JOHN SITZIA, Good practice in the conduct and reporting of survey research, International Journal for Quality in Health Care , Volume 15, Issue 3, May 2003, Pages 261–266, https://doi.org/10.1093/intqhc/mzg031

• Permissions Icon Permissions

Survey research is sometimes regarded as an easy research approach. However, as with any other research approach and method, it is easy to conduct a survey of poor quality rather than one of high quality and real value. This paper provides a checklist of good practice in the conduct and reporting of survey research. Its purpose is to assist the novice researcher to produce survey work to a high standard, meaning a standard at which the results will be regarded as credible. The paper first provides an overview of the approach and then guides the reader step-by-step through the processes of data collection, data analysis, and reporting. It is not intended to provide a manual of how to conduct a survey, but rather to identify common pitfalls and oversights to be avoided by researchers if their work is to be valid and credible.

Survey research is common in studies of health and health services, although its roots lie in the social surveys conducted in Victorian Britain by social reformers to collect information on poverty and working class life (e.g. Charles Booth [ 1 ] and Joseph Rowntree [ 2 ]), and indeed survey research remains most used in applied social research. The term ‘survey’ is used in a variety of ways, but generally refers to the selection of a relatively large sample of people from a pre-determined population (the ‘population of interest’; this is the wider group of people in whom the researcher is interested in a particular study), followed by the collection of a relatively small amount of data from those individuals. The researcher therefore uses information from a sample of individuals to make some inference about the wider population.

Data are collected in a standardized form. This is usually, but not necessarily, done by means of a questionnaire or interview. Surveys are designed to provide a ‘snapshot of how things are at a specific time’ [ 3 ]. There is no attempt to control conditions or manipulate variables; surveys do not allocate participants into groups or vary the treatment they receive. Surveys are well suited to descriptive studies, but can also be used to explore aspects of a situation, or to seek explanation and provide data for testing hypotheses. It is important to recognize that ‘the survey approach is a research strategy, not a research method’ [ 3 ]. As with any research approach, a choice of methods is available and the one most appropriate to the individual project should be used. This paper will discuss the most popular methods employed in survey research, with an emphasis upon difficulties commonly encountered when using these methods.

Descriptive research

Descriptive research is a most basic type of enquiry that aims to observe (gather information on) certain phenomena, typically at a single point in time: the ‘cross-sectional’ survey. The aim is to examine a situation by describing important factors associated with that situation, such as demographic, socio-economic, and health characteristics, events, behaviours, attitudes, experiences, and knowledge. Descriptive studies are used to estimate specific parameters in a population (e.g. the prevalence of infant breast feeding) and to describe associations (e.g. the association between infant breast feeding and maternal age).

Analytical studies

Analytical studies go beyond simple description; their intention is to illuminate a specific problem through focused data analysis, typically by looking at the effect of one set of variables upon another set. These are longitudinal studies, in which data are collected at more than one point in time with the aim of illuminating the direction of observed associations. Data may be collected from the same sample on each occasion (cohort or panel studies) or from a different sample at each point in time (trend studies).

Evaluation research

This form of research collects data to ascertain the effects of a planned change.

Advantages:

The research produces data based on real-world observations (empirical data).

The breadth of coverage of many people or events means that it is more likely than some other approaches to obtain data based on a representative sample, and can therefore be generalizable to a population.

Surveys can produce a large amount of data in a short time for a fairly low cost. Researchers can therefore set a finite time-span for a project, which can assist in planning and delivering end results.

Disadvantages:

The significance of the data can become neglected if the researcher focuses too much on the range of coverage to the exclusion of an adequate account of the implications of those data for relevant issues, problems, or theories.

The data that are produced are likely to lack details or depth on the topic being investigated.

Securing a high response rate to a survey can be hard to control, particularly when it is carried out by post, but is also difficult when the survey is carried out face-to-face or over the telephone.

Research question

Good research has the characteristic that its purpose is to address a single clear and explicit research question; conversely, the end product of a study that aims to answer a number of diverse questions is often weak. Weakest of all, however, are those studies that have no research question at all and whose design simply is to collect a wide range of data and then to ‘trawl’ the data looking for ‘interesting’ or ‘significant’ associations. This is a trap novice researchers in particular fall into. Therefore, in developing a research question, the following aspects should be considered [ 4 ]:

Be knowledgeable about the area you wish to research.

Widen the base of your experience, explore related areas, and talk to other researchers and practitioners in the field you are surveying.

Consider using techniques for enhancing creativity, for example brainstorming ideas.

Avoid the pitfalls of: allowing a decision regarding methods to decide the questions to be asked; posing research questions that cannot be answered; asking questions that have already been answered satisfactorily.

The survey approach can employ a range of methods to answer the research question. Common survey methods include postal questionnaires, face-to-face interviews, and telephone interviews.

Postal questionnaires

This method involves sending questionnaires to a large sample of people covering a wide geographical area. Postal questionnaires are usually received ‘cold’, without any previous contact between researcher and respondent. The response rate for this type of method is usually low, ∼20%, depending on the content and length of the questionnaire. As response rates are low, a large sample is required when using postal questionnaires, for two main reasons: first, to ensure that the demographic profile of survey respondents reflects that of the survey population; and secondly, to provide a sufficiently large data set for analysis.

Face-to-face interviews

Face-to-face interviews involve the researcher approaching respondents personally, either in the street or by calling at people’s homes. The researcher then asks the respondent a series of questions and notes their responses. The response rate is often higher than that of postal questionnaires as the researcher has the opportunity to sell the research to a potential respondent. Face-to-face interviewing is a more costly and time-consuming method than the postal survey, however the researcher can select the sample of respondents in order to balance the demographic profile of the sample.

Telephone interviews

Telephone surveys, like face-to-face interviews, allow a two-way interaction between researcher and respondent. Telephone surveys are quicker and cheaper than face-to-face interviewing. Whilst resulting in a higher response rate than postal surveys, telephone surveys often attract a higher level of refusals than face-to-face interviews as people feel less inhibited about refusing to take part when approached over the telephone.

Whether using a postal questionnaire or interview method, the questions asked have to be carefully planned and piloted. The design, wording, form, and order of questions can affect the type of responses obtained, and careful design is needed to minimize bias in results. When designing a questionnaire or question route for interviewing, the following issues should be considered: (1) planning the content of a research tool; (2) questionnaire layout; (3) interview questions; (4) piloting; and (5) covering letter.

Planning the content of a research tool

The topics of interest should be carefully planned and relate clearly to the research question. It is often useful to involve experts in the field, colleagues, and members of the target population in question design in order to ensure the validity of the coverage of questions included in the tool (content validity).

Researchers should conduct a literature search to identify existing, psychometrically tested questionnaires. A well designed research tool is simple, appropriate for the intended use, acceptable to respondents, and should include a clear and interpretable scoring system. A research tool must also demonstrate the psychometric properties of reliability (consistency from one measurement to the next), validity (accurate measurement of the concept), and, if a longitudinal study, responsiveness to change [ 5 ]. The development of research tools, such as attitude scales, is a lengthy and costly process. It is important that researchers recognize that the development of the research tool is equal in importance—and deserves equal attention—to data collection. If a research instrument has not undergone a robust process of development and testing, the credibility of the research findings themselves may legitimately be called into question and may even be completely disregarded. Surveys of patient satisfaction and similar are commonly weak in this respect; one review found that only 6% of patient satisfaction studies used an instrument that had undergone even rudimentary testing [ 6 ]. Researchers who are unable or unwilling to undertake this process are strongly advised to consider adopting an existing, robust research tool.

Questionnaire layout

Questionnaires used in survey research should be clear and well presented. The use of capital (upper case) letters only should be avoided, as this format is hard to read. Questions should be numbered and clearly grouped by subject. Clear instructions should be given and headings included to make the questionnaire easier to follow.

The researcher must think about the form of the questions, avoiding ‘double-barrelled’ questions (two or more questions in one, e.g. ‘How satisfied were you with your personal nurse and the nurses in general?’), questions containing double negatives, and leading or ambiguous questions. Questions may be open (where the respondent composes the reply) or closed (where pre-coded response options are available, e.g. multiple-choice questions). Closed questions with pre-coded response options are most suitable for topics where the possible responses are known. Closed questions are quick to administer and can be easily coded and analysed. Open questions should be used where possible replies are unknown or too numerous to pre-code. Open questions are more demanding for respondents but if well answered can provide useful insight into a topic. Open questions, however, can be time consuming to administer and difficult to analyse. Whether using open or closed questions, researchers should plan clearly how answers will be analysed.

Interview questions

Open questions are used more frequently in unstructured interviews, whereas closed questions typically appear in structured interview schedules. A structured interview is like a questionnaire that is administered face to face with the respondent. When designing the questions for a structured interview, the researcher should consider the points highlighted above regarding questionnaires. The interviewer should have a standardized list of questions, each respondent being asked the same questions in the same order. If closed questions are used the interviewer should also have a range of pre-coded responses available.

If carrying out a semi-structured interview, the researcher should have a clear, well thought out set of questions; however, the questions may take an open form and the researcher may vary the order in which topics are considered.

A research tool should be tested on a pilot sample of members of the target population. This process will allow the researcher to identify whether respondents understand the questions and instructions, and whether the meaning of questions is the same for all respondents. Where closed questions are used, piloting will highlight whether sufficient response categories are available, and whether any questions are systematically missed by respondents.

When conducting a pilot, the same procedure as as that to be used in the main survey should be followed; this will highlight potential problems such as poor response.

Covering letter

All participants should be given a covering letter including information such as the organization behind the study, including the contact name and address of the researcher, details of how and why the respondent was selected, the aims of the study, any potential benefits or harm resulting from the study, and what will happen to the information provided. The covering letter should both encourage the respondent to participate in the study and also meet the requirements of informed consent (see below).

The concept of sample is intrinsic to survey research. Usually, it is impractical and uneconomical to collect data from every single person in a given population; a sample of the population has to be selected [ 7 ]. This is illustrated in the following hypothetical example. A hospital wants to conduct a satisfaction survey of the 1000 patients discharged in the previous month; however, as it is too costly to survey each patient, a sample has to be selected. In this example, the researcher will have a list of the population members to be surveyed (sampling frame). It is important to ensure that this list is both up-to date and has been obtained from a reliable source.

The method by which the sample is selected from a sampling frame is integral to the external validity of a survey: the sample has to be representative of the larger population to obtain a composite profile of that population [ 8 ].

There are methodological factors to consider when deciding who will be in a sample: How will the sample be selected? What is the optimal sample size to minimize sampling error? How can response rates be maximized?

The survey methods discussed below influence how a sample is selected and the size of the sample. There are two categories of sampling: random and non-random sampling, with a number of sampling selection techniques contained within the two categories. The principal techniques are described here [ 9 ].

Random sampling

Generally, random sampling is employed when quantitative methods are used to collect data (e.g. questionnaires). Random sampling allows the results to be generalized to the larger population and statistical analysis performed if appropriate. The most stringent technique is simple random sampling. Using this technique, each individual within the chosen population is selected by chance and is equally as likely to be picked as anyone else. Referring back to the hypothetical example, each patient is given a serial identifier and then an appropriate number of the 1000 population members are randomly selected. This is best done using a random number table, which can be generated using computer software (a free on-line randomizer can be found at http://www.randomizer.org/index.htm ).

Alternative random sampling techniques are briefly described. In systematic sampling, individuals to be included in the sample are chosen at equal intervals from the population; using the earlier example, every fifth patient discharged from hospital would be included in the survey. Stratified sampling selects a specific group and then a random sample is selected. Using our example, the hospital may decide only to survey older surgical patients. Bigger surveys may employ cluster sampling, which randomly assigns groups from a large population and then surveys everyone within the groups, a technique often used in national-scale studies.

Non-random sampling

Non-random sampling is commonly applied when qualitative methods (e.g. focus groups and interviews) are used to collect data, and is typically used for exploratory work. Non-random sampling deliberately targets individuals within a population. There are three main techniques. (1) purposive sampling: a specific population is identified and only its members are included in the survey; using our example above, the hospital may decide to survey only patients who had an appendectomy. (2) Convenience sampling: the sample is made up of the individuals who are the easiest to recruit. Finally, (3) snowballing: the sample is identified as the survey progresses; as one individual is surveyed he or she is invited to recommend others to be surveyed.

It is important to use the right method of sampling and to be aware of the limitations and statistical implications of each. The need to ensure that the sample is representative of the larger population was highlighted earlier and, alongside the sampling method, the degree of sampling error should be considered. Sampling error is the probability that any one sample is not completely representative of the population from which it has been drawn [ 9 ]. Although sampling error cannot be eliminated entirely, the sampling technique chosen will influence the extent of the error. Simple random sampling will give a closer estimate of the population than a convenience sample of individuals who just happened to be in the right place at the right time.

Sample size

What sample size is required for a survey? There is no definitive answer to this question: large samples with rigorous selection are more powerful as they will yield more accurate results, but data collection and analysis will be proportionately more time consuming and expensive. Essentially, the target sample size for a survey depends on three main factors: the resources available, the aim of the study, and the statistical quality needed for the survey. For ‘qualitative’ surveys using focus groups or interviews, the sample size needed will be smaller than if quantitative data is collected by questionnaire. If statistical analysis is to be performed on the data then sample size calculations should be conducted. This can be done using computer packages such as G * Power [ 10 ]; however, those with little statistical knowledge should consult a statistician. For practical recommendations on sample size, the set of survey guidelines developed by the UK Department of Health [ 11 ] should be consulted.

Larger samples give a better estimate of the population but it can be difficult to obtain an adequate number of responses. It is rare that everyone asked to participate in the survey will reply. To ensure a sufficient number of responses, include an estimated non-response rate in the sample size calculations.

Response rates are a potential source of bias. The results from a survey with a large non-response rate could be misleading and only representative of those who replied. French [ 12 ] reported that non-responders to patient satisfaction surveys are less likely to be satisfied than people who reply. It is unwise to define a level above which a response rate is acceptable, as this depends on many local factors; however, an achievable and acceptable rate is ∼75% for interviews and 65% for self-completion postal questionnaires [ 9 , 13 ]. In any study, the final response rate should be reported with the results; potential differences between the respondents and non-respondents should be explicitly explored and their implications discussed.

There are techniques to increase response rates. A questionnaire must be concise and easy to understand, reminders should be sent out, and method of recruitment should be carefully considered. Sitzia and Wood [ 13 ] found that participants recruited by mail or who had to respond by mail had a lower mean response rate (67%) than participants who were recruited personally (mean response 76.7%). A most useful review of methods to maximize response rates in postal surveys has recently been published [ 14 ].

Researchers should approach data collection in a rigorous and ethical manner. The following information must be clearly recorded:

How, where, how many times, and by whom potential respondents were contacted.

How many people were approached and how many of those agreed to participate.

How did those who agreed to participate differ from those who refused with regard to characteristics of interest in the study, for example how were they identified, where were they approached, and what was their gender, age, and features of their illness or health care.

How was the survey administered (e.g. telephone interview).

What was the response rate (i.e. the number of usable data sets as a proportion of the number of people approached).

The purpose of all analyses is to summarize data so that it is easily understood and provides the answers to our original questions: ‘In order to do this researchers must carefully examine their data; they should become friends with their data’ [ 15 ]. Researchers must prepare to spend substantial time on the data analysis phase of a survey (and this should be built into the project plan). When analysis is rushed, often important aspects of the data are missed and sometimes the wrong analyses are conducted, leading to both inaccurate results and misleading conclusions [ 16 ]. However, and this point cannot be stressed strongly enough, researchers must not engage in data dredging, a practice that can arise especially in studies in which large numbers of dependent variables can be related to large numbers of independent variables (outcomes). When large numbers of possible associations in a dataset are reviewed at P < 0.05, one in 20 of the associations by chance will appear ‘statistically significant’; in datasets where only a few real associations exist, testing at this significance level will result in the large majority of findings still being false positives [ 17 ].

The method of data analysis will depend on the design of the survey and should have been carefully considered in the planning stages of the survey. Data collected by qualitative methods should be analysed using established methods such as content analysis [ 18 ], and where quantitative methods have been used appropriate statistical tests can be applied. Describing methods of analysis here would be unproductive as a multitude of introductory textbooks and on-line resources are available to help with simple analyses of data (e.g. [ 19 , 20 ]). For advanced analysis a statistician should be consulted.

When reporting survey research, it is essential that a number of key points are covered (though the length and depth of reporting will be dependent upon journal style). These key points are presented as a ‘checklist’ below:

Explain the purpose or aim of the research, with the explicit identification of the research question.

Explain why the research was necessary and place the study in context, drawing upon previous work in relevant fields (the literature review).

State the chosen research method or methods, and justify why this method was chosen.

Describe the research tool. If an existing tool is used, briefly state its psychometric properties and provide references to the original development work. If a new tool is used, you should include an entire section describing the steps undertaken to develop and test the tool, including results of psychometric testing.

Describe how the sample was selected and how data were collected, including:

How were potential subjects identified?

How many and what type of attempts were made to contact subjects?

Who approached potential subjects?

Where were potential subjects approached?

How was informed consent obtained?

How many agreed to participate?

How did those who agreed differ from those who did not agree?

What was the response rate?

Describe and justify the methods and tests used for data analysis.

Present the results of the research. The results section should be clear, factual, and concise.

Interpret and discuss the findings. This ‘discussion’ section should not simply reiterate results; it should provide the author’s critical reflection upon both the results and the processes of data collection. The discussion should assess how well the study met the research question, should describe the problems encountered in the research, and should honestly judge the limitations of the work.

Present conclusions and recommendations.

The researcher needs to tailor the research report to meet:

The expectations of the specific audience for whom the work is being written.

The conventions that operate at a general level with respect to the production of reports on research in the social sciences.

Anyone involved in collecting data from patients has an ethical duty to respect each individual participant’s autonomy. Any survey should be conducted in an ethical manner and one that accords with best research practice. Two important ethical issues to adhere to when conducting a survey are confidentiality and informed consent.

The respondent’s right to confidentiality should always be respected and any legal requirements on data protection adhered to. In the majority of surveys, the patient should be fully informed about the aims of the survey, and the patient’s consent to participate in the survey must be obtained and recorded.

The professional bodies listed below, among many others, provide guidance on the ethical conduct of research and surveys.

American Psychological Association: http://www.apa.org

British Psychological Society: http://www.bps.org.uk

British Medical Association: http://www.bma.org.uk .

UK General Medical Council: http://www.gmc-uk.org

American Medical Association: http://www.ama-assn.org

UK Royal College of Nursing: http://www.rcn.org.uk

UK Department of Health: http://www.doh.gov

Survey research demands the same standards in research practice as any other research approach, and journal editors and the broader research community will judge a report of survey research with the same level of rigour as any other research report. This is not to say that survey research need be particularly difficult or complex; the point to emphasize is that researchers should be aware of the steps required in survey research, and should be systematic and thoughtful in the planning, execution, and reporting of the project. Above all, survey research should not be seen as an easy, ‘quick and dirty’ option; such work may adequately fulfil local needs (e.g. a quick survey of hospital staff satisfaction), but will not stand up to academic scrutiny and will not be regarded as having much value as a contribution to knowledge.

Address reprint requests to John Sitzia, Research Department, Worthing Hospital, Lyndhurst Road, Worthing BN11 2DH, West Sussex, UK. E-mail: [email protected]

London School of Economics, UK. Http://booth.lse.ac.uk/ (accessed 15 January 2003 ).

Vernon A. A Quaker Businessman: Biography of Joseph Rowntree (1836–1925) . London: Allen & Unwin, 1958 .

Denscombe M. The Good Research Guide: For Small-scale Social Research Projects . Buckingham: Open University Press, 1998 .

Robson C. Real World Research: A Resource for Social Scientists and Practitioner-researchers . Oxford: Blackwell Publishers, 1993 .

Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to their Development and Use . Oxford: Oxford University Press, 1995 .

Sitzia J. How valid and reliable are patient satisfaction data? An analysis of 195 studies. Int J Qual Health Care 1999 ; 11: 319 –328.

Bowling A. Research Methods in Health. Investigating Health and Health Services . Buckingham: Open University Press, 2002 .

American Statistical Association, USA. Http://www.amstat.org (accessed 9 December 2002 ).

Arber S. Designing samples. In: Gilbert N, ed. Researching Social Life . London: SAGE Publications, 2001 .

Heinrich Heine University, Dusseldorf, Germany. Http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/index.html (accessed 12 December 2002 ).

Department of Health, England. Http://www.doh.gov.uk/acutesurvey/index.htm (accessed 12 December 2002 ).

French K. Methodological considerations in hospital patient opinion surveys. Int J Nurs Stud 1981 ; 18: 7 –32.

Sitzia J, Wood N. Response rate in patient satisfaction research: an analysis of 210 published studies. Int J Qual Health Care 1998 ; 10: 311 –317.

Edwards P, Roberts I, Clarke M et al. Increasing response rates to postal questionnaires: systematic review. Br Med J 2002 ; 324: 1183 .

Wright DB. Making friends with our data: improving how statistical results are reported. Br J Educ Psychol 2003 ; in press.

Wright DB, Kelley K. Analysing and reporting data. In: Michie S, Abraham C, eds. Health Psychology in Practice . London: SAGE Publications, 2003 ; in press.

Davey Smith G, Ebrahim S. Data dredging, bias, or confounding. Br Med J 2002 ; 325: 1437 –1438.

Morse JM, Field PA. Nursing Research: The Application of Qualitative Approaches . London: Chapman and Hall, 1996 .

Wright DB. Understanding Statistics: An Introduction for the Social Sciences . London: SAGE Publications, 1997 .

Sportscience, New Zealand. Http://www.sportsci.org/resource/stats/index.html (accessed 12 December 2002 ).

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1464-3677
• Print ISSN 1353-4505
• Copyright © 2024 International Society for Quality in Health Care and Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

8.2: Pros and Cons of Survey Research

• Last updated
• Save as PDF
• Page ID 12588

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

\( \newcommand{\Span}{\mathrm{span}}\)

\( \newcommand{\id}{\mathrm{id}}\)

\( \newcommand{\kernel}{\mathrm{null}\,}\)

\( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\)

\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\)

\( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

\( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vectorC}[1]{\textbf{#1}} \)

\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

Learning Objectives

• Identify and explain the strengths of survey research.
• Identify and explain the weaknesses of survey research.

Survey research, as with all methods of data collection, comes with both strengths and weaknesses. We’ll examine both in this section.

Strengths of Survey Method

Researchers employing survey methods to collect data enjoy a number of benefits. First, surveys are an excellent way to gather lots of information from many people. In my own study of older people’s experiences in the workplace, I was able to mail a written questionnaire to around 500 people who lived throughout the state of Maine at a cost of just over $1,000. This cost included printing copies of my seven-page survey, printing a cover letter, addressing and stuffing envelopes, mailing the survey, and buying return postage for the survey. I realize that $1,000 is nothing to sneeze at. But just imagine what it might have cost to visit each of those people individually to interview them in person. Consider the cost of gas to drive around the state, other travel costs, such as meals and lodging while on the road, and the cost of time to drive to and talk with each person individually. We could double, triple, or even quadruple our costs pretty quickly by opting for an in-person method of data collection over a mailed survey. Thus surveys are relatively cost effective .

Related to the benefit of cost effectiveness is a survey’s potential for generalizability . Because surveys allow researchers to collect data from very large samples for a relatively low cost, survey methods lend themselves to probability sampling techniques, which we discussed in Chapter 7. Of all the data-collection methods described in this text, survey research is probably the best method to use when one hopes to gain a representative picture of the attitudes and characteristics of a large group.

Survey research also tends to be a reliable method of inquiry. This is because surveys are standardized in that the same questions, phrased in exactly the same way, are posed to participants. Other methods, such as qualitative interviewing, which we’ll learn about in Chapter 9, do not offer the same consistency that a quantitative survey offers. This is not to say that all surveys are always reliable. A poorly phrased question can cause respondents to interpret its meaning differently, which can reduce that question’s reliability. Assuming well-constructed question and questionnaire design, one strength of survey methodology is its potential to produce reliable results.

The versatility of survey research is also an asset. Surveys are used by all kinds of people in all kinds of professions. I repeat, surveys are used by all kinds of people in all kinds of professions. Is there a light bulb switching on in your head? I hope so. The versatility offered by survey research means that understanding how to construct and administer surveys is a useful skill to have for all kinds of jobs. Lawyers might use surveys in their efforts to select juries, social service and other organizations (e.g., churches, clubs, fundraising groups, activist groups) use them to evaluate the effectiveness of their efforts, businesses use them to learn how to market their products, governments use them to understand community opinions and needs, and politicians and media outlets use surveys to understand their constituencies.

In sum, the following are benefits of survey research:

• Cost-effective
• Generalizable

Weaknesses of Survey Method

As with all methods of data collection, survey research also comes with a few drawbacks. First, while one might argue that surveys are flexible in the sense that we can ask any number of questions on any number of topics in them, the fact that the survey researcher is generally stuck with a single instrument for collecting data (the questionnaire), surveys are in many ways rather inflexible . Let’s say you mail a survey out to 1,000 people and then discover, as responses start coming in, that your phrasing on a particular question seems to be confusing a number of respondents. At this stage, it’s too late for a do-over or to change the question for the respondents who haven’t yet returned their surveys. When conducting in-depth interviews, on the other hand, a researcher can provide respondents further explanation if they’re confused by a question and can tweak their questions as they learn more about how respondents seem to understand them.

Validity can also be a problem with surveys. Survey questions are standardized; thus it can be difficult to ask anything other than very general questions that a broad range of people will understand. Because of this, survey results may not be as valid as results obtained using methods of data collection that allow a researcher to more comprehensively examine whatever topic is being studied. Let’s say, for example, that you want to learn something about voters’ willingness to elect an African American president, as in our opening example in this chapter. General Social Survey respondents were asked, “If your party nominated an African American for president, would you vote for him if he were qualified for the job?” Respondents were then asked to respond either yes or no to the question. But what if someone’s opinion was more complex than could be answered with a simple yes or no? What if, for example, a person was willing to vote for an African American woman but not an African American man?I am not at all suggesting that such a perspective makes any sense, but it is conceivable that an individual might hold such a perspective.

In sum, potential drawbacks to survey research include the following:

• Inflexibility

KEY TAKEAWAYS

• Strengths of survey research include its cost effectiveness, generalizability, reliability, and versatility.
• Weaknesses of survey research include inflexibility and issues with validity.
• What are some ways that survey researchers might overcome the weaknesses of this method?
• Find an article reporting results from survey research (remember how to use Sociological Abstracts?). How do the authors describe the strengths and weaknesses of their study? Are any of the strengths or weaknesses described here mentioned in the article?

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Survey research, as with all methods of data collection, comes with both strengths and weaknesses. The following sections will examine both.

Strengths of survey method

Researchers employing survey methods to collect data enjoy a number of benefits. First, surveys are an excellent way to gather lots of information from many people, and they are relatively cost effective.

Related to the benefit of cost effectiveness is a survey’s potential for generalizability. Because surveys allow researchers to collect data from very large samples for a relatively low cost, survey methods lend themselves to probability sampling techniques, which we discussed in Chapter 7 “Sampling“. Of all the data-collection methods described in this text, survey research is probably the best method to use when you hope to gain a representative picture of the attitudes and characteristics of a large group.

Survey research also tends to be a reliable method of inquiry. This is because surveys are standardized; the same questions, phrased in exactly the same way, are posed to participants. Other methods, such as qualitative interviewing, which you will learn about in Chapter 10 “Qualitative Data Collection Methods” , do not offer the same level of consistency that a quantitative survey offers. One strength of survey methodology is its potential to produce reliable results. This is not to say that all surveys are always reliable. A poorly-phrased question can cause respondents to interpret its meaning differently, which can reduce that question’s reliability.

The versatility of survey research is also an asset. Surveys are used by all kinds of people in all kinds of professions. The versatility offered by survey research means that understanding how to construct and administer surveys is a useful skill to have for all kinds of jobs. For example, lawyers often use surveys in their efforts to select juries. Social service and other organizations (e.g., churches, clubs, fundraising groups, and activist groups) use them to evaluate the effectiveness of their efforts. Businesses use them to learn how to market their products. Governments use them to understand community opinions and needs, and politicians and media outlets use surveys to understand their constituencies.

The following are benefits of survey research:

• Cost-effectiveness.
• Generalizability.
• Reliability.
• Versatility.

Weaknesses of survey method

As with all methods of data collection, survey research also comes with a few drawbacks. First, while one might argue that surveys are flexible in the sense that they can ask any number of questions on any number of topics, the fact that the survey researcher is generally stuck with a single instrument for collecting data (the questionnaire) means that surveys could also be described as inflexible. For example, suppose you mail a survey out to 1,000 people and then discover, as responses start coming in, that your phrasing on a particular question seems to be confusing a number of respondents. At this stage, it is too late to change the question for the respondents who have not yet returned their surveys (however, if you conduct a pilot study first, you should avoid such a situation). When conducting in-depth interviews, on the other hand, a researcher can provide respondents further explanation if they are confused by a question, and can tweak their questions as they learn more about how respondents seem to understand them.

Validity can also be a problem with surveys. Survey questions are standardized; thus, it can be difficult to ask anything other than very general questions that a broad range of people will understand. Because of this, survey results may not be as valid as results obtained using methods of data collection that allow a researcher to more comprehensively examine the topic being studied.

Potential drawbacks to survey research include:

• Inflexibility; and

Research Methods, Data Collection and Ethics Copyright © 2020 by Valerie Sheppard is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Social Science Works

Center of Deliberation

The Limits Of Survey Data: What Questionnaires Can’t Tell Us

• Latest Posts

• Alienation Online: An Analysis of Populist Facebook Pages In Brandenburg - October 24, 2017
• Shy Tories & Virtue Signalling: How Labour Surged Online - June 16, 2017
• The Limits Of Survey Data: What Questionnaires Can’t Tell Us - March 7, 2017

All research methodologies have their limitations, as many authors have pointed before (see for example Visser, Krosnick and Lavrakas, 2000). From the generalisabilty of data to the nitty-gritty of bias and question wording, every method has its flaws. In fact, the in-fighting between methodological approaches is one of social science’s worst kept secrets: the hostility between quantitative and qualitative data scholars knows almost no bounds (admittedly that’s ‘almost no bounds’ within the polite world of academic debate) and doesn’t look set to be resolved any time soon. That said, there are some methods that are better suited than others to certain types of studies. This article will examine the role of survey data in values studies and argue that it is a blunt tool for this kind of research and that qualitative study methods, particularly via deliberation, are more appropriate. This article will do so via an examination of a piece of 2016 research published by the German ministry for migrants and refugees (the BAMF) which explored both the demographics and the social values held by refugees that have arrived in Germany in the last three years. This article will argue that surveys are unfit to get at the issues that are most important to people.

The Good, The Bad & The Survey

Germany has been Europe’s leading figure as the refugee crisis has deepened worldwide following the collapse of government in Syria and the rise of ISIS. Today, there are 65.3 million displaced people from across the world and 21.3 million refugees (UNHCR, 2016) , a number that surpasses even the number of refugees following the Second World War. The exact number of refugees living in Germany (official statistics typically count all migrants seeking protection as refugees, although there is some difference between the various legal statuses) is not entirely clear and the figure is unstable. And while this figure still lags behind the efforts made by countries like Turkey and Jordan, this represents the highest total number of refugees in a European country and matches the pro capita efforts of Sweden. Meanwhile, there are signs that Germany’s residents do not always welcome their new neighbours. For example, in 2016, there were almost 2,000 reported attacks on refugees and refugee homes (Antonio-Amedeo Stiftung, 2017) a similar trend was established by Benček and Strasheim (2016), and the rise of the far-right and anti-migrant party, the AfD in local elections last year points to unresolved resentment towards the newcomers.

In this context then, it makes sense for the BAMF ( Bundesamt für Migration und Flüchtlinge ) , the ministry responsible for refugees and migrants in Germany, to respond to pressure in the media and from politicians to get a better overall picture about the kinds of people the refugees to Germany are. As such, their 2016 paper: “Survey of refugees – flight, arrival in Germany and first steps of integration” [1] details a host of information about newcomers in Germany. The study, which relied on questionnaires given by BAMF officials in a number of languages, and a face-to-face or online format (BAMF, 2016, 11), asked questions of 4,500 refugee respondents. For the most part, the study offers excellent insight into the demographic history of refugees to Germany and will be helpful for policymakers looking to ensure that efforts to help settles refugees are appropriately targeted. For example, the study detailed the relatively high level of education enjoyed by typical refugees to Germany (an average between 10 and 11 years of schooling) (ibid., 37) and some of the specific difficulties this group have in successfully navigating the job market (ibid., 46) and where this group turns to for help for this.

In addition to offering the most up-to-date information about refugees’ home countries and their path into Germany, the study is extremely helpful for politicians and scholars looking to enhance their understanding of logistical and practical issues facing migrants; for example, who has access to integration courses? How many unaccompanied children are in Germany? How many men and how many women fled to Germany? Here, the study is undoubtedly helpful.  However, the latter stages of the report purport to examine the social values held by refugees, and it is this part of the study that this article takes issue with.

Respondents were asked to answer questions about their values. The topics included: the right form a government should take, the role of democracy, voting rights of women, the role of religion in the state, men and women’s equality in a marriage, and perceived difference between the values of refugees and Germans among others. While this article doesn’t take issue with the veracity of the findings reported in the article, it does argue that the methods used here are inappropriate for the task at hand. Consider first the questions relating to refugees’ attitudes toward democracy and government. The report found that 96% of refugee respondents agreed with the statement: “One should have a democratic system” [2] compared with 95% of the German control group (ibid., 52). This finding was picked up in the liberal media and heralded as a sign that refugees share central German social values. It is entirely possible that this is true. However, it isn’t difficult to see the ways in which this number might have been accidentally manufactured and should hence be treated with considerable caution.

To do so, one must first consider the circumstances of the interview or questionnaire. As a refugee in Germany, you are confronted with the authority of the BAMF regularly, and you are also likely aware that it is representatives from this organization that ultimately decide  on you and your family’s status in Germany and whether you will have the right to stay or not. You are then asked for detailed information about your family history, your education and your participation in integration courses by a representative from this institution. Finally, the interviewer asks what your views are on democracy, women’s rights and religion. Is it too much of a jump to suggest that someone who has had to flee their home and take the extraordinarily dangerous trip to Europe is savvy enough to spot a potential trap here? In these circumstances, there is a tendency to give the answer the interviewer wants to hear. This interviewer bias effect is not a problem exclusive to surveys of refugees’ social values (Davis, 2013), however the power imbalance in these interactions exacerbates the effect. The argument advanced here is not that refugees do not hold a positive view of democracy, but that the trying to find out their views via a survey of this sort is flawed. In fact, the report doesn’t find any significant points of departure between Germans and refugees on any of the major values other than the difficulties presented by women earning more money than their husbands and its potential to cause marital difficulties (ibid., 54).

Asking Questions About Essential Contested Concepts

Beyond the serious power imbalance noted above, another key issue not addressed in the BAMF study is the question of contested concepts. Essential contested concepts, an idea first advanced by W.B. Gallie in 1956, are the big topics like art, beauty, fairness and trust. These big topics, which also include traditionally social scientific and political topics like democracy and equality, are defined as ‘essentially contested’ when the premise of the concept – for example ‘freedom’ – is widely accepted, but how best to realise freedom is disputed (Hart, 1961, 156). The BAMF survey uses these big topics without offering a definition to go with them. What do people mean when they say that ‘men and women should have equal rights [3] ’ (BAMF, 2016, 52)? What does equality mean in this context? There are of course many different ways that ‘equality’ between men and women can be interpreted. For example, many conservative Catholic churches argue that men and women are ‘equal’ but different, and have clear family roles for men and women. Likewise, participants could equally mean to say that they believe that men and women should have equal, shared family responsibilities, there is no way to know this from this study. Hence, it is difficult to know how best to interpret these kinds of statistics without considerable context.

As part of the work undertaken by Social Science Works, the team are regularly confronted by these kinds of questions via deliberative workshops with Germans and refugees. In these workshops the team ask questions like “What is democracy?”, “What is freedom?”, “What is equality?”. In doing so, the aim of the workshops is to build a consensus together by formulating and reformulating possible definitions [4] , finding common ground between conflicting perspectives and ultimately defining the concepts as a group. What is among the most striking things about these meetings is the initial reluctance of participants to volunteer answers – there is a real lack of certainty about what these kinds of words mean in practice, even among participants who, for example, have studied social and political sciences or work in politics. With the benefit of hindsight, workshop participants have acknowledged these problems in dealing with essentially contested concepts, participants have commented :

“Social Science Works has encouraged me to question my own views and views more critically and to develop a more precise concept for large and often hard to grasp terms such as “democracy”, “freedom” or “equality”. This experience has shown me how complicated it is for me – as someone who I really felt proficient in these questions – to formulate such ideas concretely.”   (German participant from the 2016 series of workshops) “The central starting point for the training was, for me, the common notion of understanding of democracy and freedom. In the intensive discussion, I realized that these terms, which seem self-evident, are anything but.” (German participant from the 2016 series of workshops).

In attempting to talk about these big issues, it become clear just how little consensus there is on these kinds of topics. The participants quoted here work and volunteer in the German social sector and hence confront these kinds of ideas implicitly on a daily basis. The level of uncertainty pointed at here, and from Social Science Works’ wider experience working with volunteers, social workers and refugees suggests that the lack of fluency in essentially contested concepts is a wider problem. In the context of the BAMF research then, it is clear that readers ought to take the chapter detailing the ‘values’ of refugees and Germans with a generous pinch of salt.

Building Consensus & Moving Forward

This article does not seek to suggest that there is no role for survey data in helping to answer questions relating to refugees in Germany. For the most part, the BAMF research offers excellent data on key questions relating to demographics and current social conditions. Hence, the study ought to make an excellent tool of policy makers seeking to better target their support of refugees. However, it is equally clear that to discuss essentially contested concepts like democracy and equality, a survey is a very blunt tool, and here the BAMF study fails to convince. The study seeks to make clear that the social and political values between Germans and refugees are similar and the differences are minimal. The experience in the deliberative workshops hosted by Social Science Works suggests that this is probably true, insofar as both groups find these concepts difficult to define and have to wrestle to make sense of them. This is not something articulated in the BAMF research, however.

Our collective lack of fluency in these topics, even among social and political scholars, has long roots best described another time. However, if we are to improve our abilities to discuss these kinds of topics and build collective ideas for social change and cohesion, there are much better places to begin than a questionnaire. If we are to build a collective understanding of our political structures and our social values, we need to address this lack of fluency by engaging in discussions with diverse groups and together building a coherent idea about social and political ideas.

[1]  Original German: “Befragung von Geflüchteten – Flucht, Ankunft in Deutschland und erste Schritte der Integration“

[2] Original German: „Man sollte ein demokratisches System haben.“

[3] Original German: „Frauen haben die gleichen Rechte wie Männer“

[4] For a more detailed overview of the deliberative method in these workshops, see Blokland, 2016.

Amadeu Antonio Foundation (2016), Hate Speech Against Refugees , Amadeu Antonio Foundation, Berlin.

Benček, D. and Strasheim, J. (2016), Refugees Welcome? Introducing a New Dataset on Anti-Refugee Violence in Germany, 2014–2015 , Working Paper No. 2032, University of Kiel.

Davis, R. E.; et al. (Feb 2010). Interviewer effects in public health surveys , Health Education Research, Oxford University Press, Oxford.

Hart, H.L.A., (1961),  The Concept of Law , Oxford University Press, Oxford.

IAB-BAMF-SOEP (2016), B efragung von Geflüchteten – Flucht, Ankunft in Deutschland und erste Schritte der Integration , BAMF-Forschungsbericht 29, Nürnberg: Bundesamt für Migration und Flüchtlinge.

UNHCR (2016), Global Trends: Forced Displacement in 2015 , UNHCR, New York.

Visser, P. S., Krosnick, J. A., & Lavrakas, P. (2000), Survey research , in H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social psychology , New York: Cambridge University Press.

Leave a Reply Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed .

[email protected]

214-257-8909

Advantages and Disadvantages of Survey Research

Survey research is quickly becoming the number one tool that market researchers use to gather data. The advent of online survey tools has led to widespread use of quantitative surveys in order to collect, analyze, and use data that can contribute to a more effective business model, better marketing strategies, improved customer service and more.

Survey research methods have been shown time and time again to benefit market researchers and improve ROI. Yet depending on the type of research you are doing, survey research may not be as appropriate. Here are the advantages and disadvantages of surveys:

Advantages of Survey Research

Surveys – particularly online surveys – have a very small cost per participant. Even with incentives, the cost per response is often far less than the cost of running a focus group or a phone interview, while the responses you receive can number into the thousands.

• Data Collection

Data collection is much easier with surveys, which tend to use easy to read data sources that can be compiled and analyzed as needed for your market research needs. The qualitative research techniques of other market research methods may yield some interesting answers, but the ability to analyze themes becomes a much more difficult (and possibly inaccurate) process.

• Sample Size

Sample size is one of the most underrated aspects of all types of research, including market research. Far too many companies make the mistake of assuming their small sample of only a few dozen participants is enough to make any firm conclusions. Surveys allow you to reach thousands of possible participants if necessary, which ensures a more accurate sample in which to draw conclusions.

• Candid Responses

The anonymity of surveys allows people to feel more candid with their responses. To get accurate data, you need your participants to be as honest as possible with their answers. Surveys provide more honest responses than other types of research methodology, especially if it is clear that the answers will remain confidential.

Disadvantages of Survey Research

• Sample Choice

It’s up to the user to ensure they have a representative sample from which to collect data. Survey research is prone to researcher error, where assumptions are made about the sample that may not be accurate. For example, if you are researching low income communities, many may not have email access, so their voice won’t be heard in your data.

Another weakness of survey research is rigidity. Before you develop your survey, you need to remember to account for all possible answers and program the survey accordingly. If you fail to account for all possible answers you may be missing out on data. For example, if the participant wants to tell you something about your product, but doesn’t have space to do so, you will lose out on that data.

Survey Research for Marketing

Of all of the ways to perform market research, survey research is one of the most reliable and far reaching. It is, however, prone to user error. If the researcher is not prepared for running survey research, the data collected may not be as valuable. However, if the marketer understands how to use survey research effectively and accurately, it has been shown to be the best available way to collect and analyze data.

Leave a Reply Cancel Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

By using this form you agree with the storage and handling of your data by this website. *

• Privacy Policy
• Strictly Necessary Cookies
• Cookie Policy

This Privacy Policy sets out how we,  Methods Group LLC ("SurveyMethods") , collect, store and use information about you when you use or interact with our website,  surveymethods.com (our website) and where we otherwise obtain or collect information about you. This Privacy Policy is effective from 2 nd April 2020 .

Our details

When you visit our website, when you use our website, marketing communications, information obtained from third parties, disclosure and additional uses of your information, how long we retain your information, how we secure your information, transfers of your information outside the european economic area, your rights in relation to your information, changes to our privacy policy, children’s privacy.

This section summarises how we obtain, store and use information about you.  It is intended to provide a very general overview only. It is not complete in and of itself and it must be read in conjunction with the corresponding full sections of this Privacy Policy.

• Data controller: SurveyMethods
• How we collect or obtain information about you: When you provide it to us, e.g. by contacting us, placing an order on our website, completing registration forms, signing up for free trials, submitting and completing surveys, posting blogs, or signing up for content such as newsletters. From your use of our website, using cookies and occasionally, from third parties such as mailing list providers.
• Information we collect: Email address, password, first name, last name, job function, demographic information such as age, date of birth, gender, education, company name, phone, billing address, country, state/province/region, city, zip/postal code, and very limited credit card details (the cardholder’s name, only the last 4 digits of the credit card number, and the expiration date) for authentication (we do not store the CVV number), IP address, information from cookies, information about your computer or device (e.g. device and browser type), information about how you use our website (e.g. which pages you have viewed, the time when you view them and what you clicked on, the geographical location from which you accessed our website (based on your IP address), VAT number (if applicable), engagement history and transaction history).
• How we use your information: For administrative and business purposes (particularly to contact you and process orders you place on our website, to post and respond to surveys, to improve our business and website, to fulfill our contractual obligations, to advertise our and other’s goods and services, to analyse your use of our website, and in connection with our legal rights and obligations).
• Disclosure of your information to third parties: User information can be shared with partners for specific types of content and events where a user has registered their information. Other disclosures are only to the extent necessary to run our business, to our service providers, to fulfill any contracts we enter into with you and where required by law or to enforce our legal rights.
• Do we sell your information to third parties (other than in the course of a business sale or purchase or similar event): No,  SurveyMethods does not sell data. However, when you register or sign up for certain types of content, your registration data can be shared with sponsors and partners. Examples of where we do this include event registrations, webinar signups or whitepaper downloads. We will always make it clear where any information provided will be shared with other parties.
• How long we retain your information: For no longer than necessary, taking into account any legal obligations we have (e.g. to maintain records for tax purposes), any other legal basis we have for using your information (e.g. your consent, performance of a contract with you or our legitimate interests as a business) and certain additional factors described in the main section below entitled 'How long we retain your information'. For specific retention periods in relation to certain information which we collect from you, please see the main section below entitled 'How long we retain your information'.
• How we secure your information: Using appropriate technical and organisational measures such as storing your information on secure servers, encrypting transfers of data to or from our servers using Secure Sockets Layer (SSL) technology, encrypting payments you make on or via our website using Secure Sockets Layer (SSL) technology and only granting access to your information where necessary.
• Use of cookies and similar technologies: We use cookies and similar information-gathering technologies such as marketing automation tracking on our website including essential, functional, analytical and targeting cookies. For more information, please view our cookie policy .
• Transfers of your information outside the European Economic Area: By using our website, your information may be transferred outside of the European Economic Area. We take personal data seriously and as such we ensure appropriate safeguards are in place, including, for example, that the third parties we use who transfer your information outside the European Economic Area have self-certified themselves as compliant with the EU-U.S. Privacy Shield.
• Use of profiling: We use profiling to understand our users better through web and marketing analytics, provide targeted advertising and deliver a personalised user experience.
• to access your information and to receive information about its use
• to have your information corrected and/or completed
• to have your information deleted
• to restrict the use of your information
• to receive your information in a portable format
• to object to the use of your information
• to withdraw your consent to the use of your information
• to complain to a supervisory authority
• Sensitive personal information:  We do collect what is commonly referred to as ‘sensitive personal information’, however we will only capture essential minimum data where it is strictly necessary.  Any sensitive data will be held on our secure servers and will be transferred securely using SSL 256-bit encryption.

If you have any questions about this Privacy Policy, please contact the data controller.

The data controller in respect of our website is SurveyMethods and can be contacted at 800-601-2462 or 214-257-8909 .

You can also contact the data controller by emailing our data protection officer at [email protected] .

We collect and use information from website visitors in accordance with this section and the section entitled 'Disclosure and additional uses of your information'.

Web server log information

We use a third party server to host our website called  Google Cloud the privacy policy of which is available here: https://policies.google.com/

Our website server automatically logs the IP address you use to access our website as well as other information about your visit such as the pages accessed, information requested, the date and time of the request, the source of your access to our website (e.g. the website or URL (link) which referred you to our website), and your browser version and operating system.

Use of website server log information for IT security purposes

We collect and store server logs to ensure network and IT security and so that the server and website remain uncompromised. This includes analysing log files to help identify and prevent unauthorised access to our network, the distribution of malicious code, denial of services attacks and other cyber-attacks, by detecting unusual or suspicious activity.

Unless we are investigating suspicious or potential criminal activity, we do not make, nor do we allow our hosting provider to make, any attempt to identify you from the information collected via server logs.

Legal basis for processing:  Compliance with a legal obligation to which we are subject (Article 6(1)(c) of the General Data Protection Regulation).

Legal obligation:  We have a legal obligation to implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk of our processing of information about individuals. Recording access to our website using server log files is such a measure.

Legal basis for processing:  Our legitimate interests (Article 6(1)(f) of the General Data Protection Regulation).

Legitimate interests:  We have a legitimate interest in using your information for the purposes of ensuring network and information security.

Use of website server log information to analyse website use and improve our website

We use the information collected by our website server logs to analyse how our website users interact with our website and its features. For example, we analyse the number of visits and unique visitors we receive, the time and date of the visit, the location of the visit and the operating system and browser use.

We use the information gathered from the analysis of this information to improve our website. For example, we use the information gathered to change the information, content and structure of our website and individual pages based according to what users are engaging most with and the duration of time spent on particular pages on our website.

Legal basis for processing:  our legitimate interests (Article 6(1)(f) of the General Data Protection Regulation).

Legitimate interest:  improving our website for our website users and getting to know our website users’ preferences so our website can better meet their needs and desires.

Cookies are data files which are sent from a website to a browser to record information about users for various purposes.

We use cookies on our website, including essential, functional, analytical and targeting cookies. For further information on how we use cookies, please see our cookie policy .

You can reject some or all of the cookies we use on or via our website by changing your browser settings or non-essential cookies by using a cookie control tool, but doing so can impair your ability to use our website or some or all of its features. For further information about cookies, including how to change your browser settings, please visit  www.allaboutcookies.org  or see our cookie policy .

When you contact us

We collect and use information from individuals who contact us in accordance with this section and the section entitled 'Disclosure and additional uses of your information'.

When you send an email to the email address displayed on our website we collect your email address and any other information you provide in that email (such as your name, telephone number and the information contained in any signature block in your email).

Legitimate interest(s):  Responding to enquiries and messages we receive and keeping records of correspondence.

Legal basis for processing:  Necessary to perform a contract or to take steps at your request to enter into a contract (Article 6(1)(b) of the General Data Protection Regulation).

Reason why necessary to perform a contract:  Where your message relates to us providing you with goods or services or taking steps at your request prior to providing you with our goods and services (for example, providing you with information about such goods and services), we will process your information in order to do so).

Enquiry forms

When you contact us using an enquiry form, we collect your personal details and match this to any information we hold about you on record. Typical personal information collected will include your name and contact details. We will also record the time, date and the specific form you completed.

If you do not provide the mandatory information required by our contact form, you will not be able to submit the contact form and we will not receive your enquiry.

We will also use this information to tailor any follow up sales and marketing communications with you. For further information, see the section of this privacy policy titled 'Marketing communications'.

Messages you send to us via our contact form may be stored outside the European Economic Area on our contact form provider’s servers.

When you contact us by phone, we collect your phone number and any information provide to us during your conversation with us.

We may record phone calls with customers for training and customer service purposes.

Legal basis for processing:  Our legitimate interests (Article 6(1)(f) of the General Data Protection Regulation)

Reason why necessary to perform a contract:  Where your message relates to us providing you with goods or services or taking steps at your request prior to providing you with our goods and services (for example, providing you with information about such goods and services), we will process your information in order to do so.

If you contact us by post, we will collect any information you provide to us in any postal communications you send us.

Legal basis for processing:  Necessary to perform a contract or to take steps at your request to enter into a contract (Article 6(1)(b) of the General Data Protection Regulation).

We collect and use information from individuals who interact with particular features of our website in accordance with this section and the section entitled 'Disclosure and additional uses of your information'.

Social Media Tools

We have a wide range of social media tools to be able to use on our website.  These tools include (but are not limited to); Sharing, Likes, comments and submitting content both on and off our website. By using these tools, you are providing your consent to store and use the submitted data, whether personal information or general information, both on and off our website.

Legal basis for processing:  Your consent (Article 6(1)(a) of the General Data Protection Regulation). Consent: You give your consent to us storing and using submitted content using the steps described above.

We may also use this information to tailor any follow up sales and marketing communications with you. For further information, see the section of this privacy policy titled 'Marketing Communications'.

Information you submit may be stored both inside and outside the European Economic Area on our servers as well as third-party servers such as Facebook.

Registered Users

When you register as a user on our website:

• We collect and store one or more of the following: Your email address, password, first name, last name, job function, company name, phone, billing address, country, state/province/region, city, zip/postal code, and very limited credit card details (the cardholder’s name, only the last 4 digits of the credit card number, and the expiration date) for authentication. We do not store the CVV number.
• We use this data to provide you with customer support and other services, bill you for our services, collect feedback, send you account-related notifications, and keep you informed about our key features, important feature updates, and latest releases.
• We store data related to your surveys, polls, and newsletters in your account that you access using your login-id and password. This includes questions, responses, images, email lists, data you enter while configuring or customizing any settings, etc. This data is processed by SurveyMethods to enable you to perform functions like design and distribution of surveys, polls, newsletters, and analysis & reporting.
• We do not share any personally identifiable and account-related data with a third party without your explicit consent. However, if you use the SurveyMethods API or 3rd Party Integrations, you will need to share your SurveyMethods login-id and the “API Key” with the 3rd party for authentication. For more on our API Terms of Use, click here .
• We may display your organization’s name and/or logo on our customer listing (unless agreed upon otherwise by both parties herein).
• Your data will be visible to those with whom you share your published reports or extracted data/reports.
• If you collaborate your surveys with other Registered Users, all collaborated data and your login-id will be visible to them.
• If you are a Child User on an Enterprise account, the Enterprise Master User (Administrator) will be able to see the SurveyMethods login-id, first name, last name, phone number, account type, and expiration date of the Enterprise Child Accounts (Member Accounts). The Enterprise Child Accounts can view the SurveyMethods login-id, first name, last name, phone number, job title, job function, country, state/province/region, and city of the Enterprise Master User.
• Troubleshoot problems and fix bugs (issues).
• Ensure website compatibility across different devices and browsers.
• Identify trends and patterns in the usage of our Services.
• Gain insights for adding or improving the functionality and usability of our website.
• Monitor and prevent abuse.
• To prevent any undesirable, abusive, or illegal activities, we have automated processes in place that check your data for malicious activities, spam, and fraud.
• We may use your data if required by law, court orders, subpoenas, or to enforce our agreements.
• We collect information using cookies. Cookies are digital files that allow websites to recognize returning users. While most browsers allow users to refuse cookies or request permission on a case-by-case basis, our site will not function properly without them. SurveyMethods uses cookies primarily to enable the smooth functioning of its Services.
• While accessing SurveyMethods, you may be able to access links that take you to websites external to SurveyMethods. SurveyMethods is not responsible for the content, policies, or terms of these websites.

GDPR Legal Classification for registered users

Legitimate interest:  Registering and administering accounts on our website to provide access to content, allows you to buy goods and services and facilitates the running and operation of our business.

Transfer and storage of your information 

Information you submit to us via the registration form on our website may be stored outside the European Economic Area on our third-party hosting provider’s servers.

When you register as an end user;

• SurveyMethods’ Surveys and Polls sent by Registered Users
• Newsletters from SurveyMethods Newsletter module and sent by Registered Users
• When responding to a survey or a poll, End Users may provide personal data such as first name, last name, phone number, email address, demographic data like age, date of birth, gender, education, income, marital status, and any other sensitive data that directly or indirectly identifies them. SurveyMethods does not use or share any data of End Users in any way. The Registered User is solely responsible for ensuring that collection and sharing of any End User data, personal or otherwise, is done with the End User’s consent and in accordance with applicable data protection laws.
• Since the Registered User controls and manages all data of their surveys, polls, and newsletters, End Users may contact the Registered User for any concerns regarding consent, privacy and protection of their data, or if they wish to access, modify, or delete their data.

GDPR Legal Classification for End Users

Visitors to our website

When you visit our website:

• SurveyMethods may record your personal data (such as your name, email address, phone, company, and the reason you are contacting us) when you visit the SurveyMethods website and contact us using our online form. Any consent for the collection and use of your data in this case is entirely voluntary.
• We may use your contact information to respond to you. We do not share any personally identifiable information with a third party without your explicit consent.

GDPR Legal Classification for Visitors

When you place an order

We collect and use information from individuals who place an order on our website in accordance with this section and the section entitled 'Disclosure and additional uses of your information'.

Information collected when you place an order

Mandatory information

When you place an order for goods or services on our website, we collect your name, email address, billing address.

If you do not provide this information, you will not be able to purchase goods or services from us on our website or enter into a contract with us.

Legal basis for processing:  Compliance with a legal obligation (Article 6(1)(c) of the General Data Protection Regulation).

Legal obligation:  We have a legal obligation to issue you with an invoice for the goods and services you purchase from us where you are VAT registered and we require the mandatory information collected by our checkout form for this purpose. We also have a legal obligation to keep accounting records, including records of transactions.

Additional information 

We can also collect additional information from you, such as your phone number, full name, address etc.

We use this information to manage and improve your customer experience with us.

If you do not supply the additional information requested at checkout, you will not be able to complete your order as we will not have the correct level of information to adequately manage your account.

Legitimate interests: The ability to provide adequate customer service and management of your customer account.

Our content, goods and services

When signing up for content, registering on our website or making a payment, we will use the information you provide in order to contact you regarding related content, products and services.

We will continue to send you marketing communications in relation to similar goods and services if you do not opt out from receiving them.

You can opt-out from receiving marketing communications at any time by emailing [email protected] .

Legitimate interests:  Sharing relevant, timely and industry-specific information on related business services, in order to help your organisation achieve its goals.

Third party goods and services

In addition to receiving information about our products and services, you can opt in to receiving marketing communications from us in relation third party goods and services by email by ticking a box indicating that you would like to receive such communications.

Legal basis for processing:  Consent (Article 6(1)(a) of the General Data Protection Regulation).

Consent:  You give your consent to us sending you information about third party goods and services by signing up to receive such information in accordance with the steps described above.

Information for marketing campaigns will be stored outside the European Economic Area on our third-party mailing list provider’s servers in the United States.

For further information about the safeguards used when your information is transferred outside the European Economic Area, see the section of this privacy policy below entitled 'Transfers of your information outside the European Economic Area'.

Use of tracking in emails

We use technologies such as tracking pixels (small graphic files) and tracked links in the emails we send to allow us to assess the level of engagement our emails receive by measuring information such as the delivery rates, open rates, click through rates and content engagement that our emails achieve.

This section sets out how we obtain or collect information about you from third parties.

Information received from third parties

We can often receive information about you from third parties. The third parties from which we receive information about you can include partner events within the marketing industry and other organisations that we have a professional affiliation with.

It is also possible that third parties with whom we have had no prior contact may provide us with information about you.

Information we obtain from third parties will generally be your name and contact details but will include any additional information about you which they provide to us.

Reason why necessary to perform a contract:  Where a third party has passed on information about you to us (such as your name and email address) in order for us to provide services to you, we will process your information in order to take steps at your request to enter into a contract with you and perform a contract with you (as the case may be).

Consent:  Where you have asked a third party to share information about you with us and the purpose of sharing that information is not related to the performance of a contract or services by us to you, we will process your information on the basis of your consent, which you give by asking the third party in question to pass on your information to us.

Legitimate interests:  Where a third party has shared information about you with us and you have not consented to the sharing of that information, we will have a legitimate interest in processing that information in certain circumstances.

For example, we would have a legitimate interest in processing your information to perform our obligations under a sub-contract with the third party, where the third party has the main contract with you. Our legitimate interest is the performance of our obligations under our sub-contract.

Similarly, third parties may pass on information about you to us if you have infringed or potentially infringed any of our legal rights. In this case, we will have a legitimate interest in processing that information to investigate and pursue any such potential infringement.

Information obtained by us from third parties

In certain circumstances (for example, to verify the information we hold about you or obtain missing information we require to provide you with a service) we will obtain information about you from certain publicly accessible sources, both EU and non-EU, such as Companies House, online customer databases, business directories, media publications, social media, and websites (including your own website if you have one).

In certain circumstances will also obtain information about you from private sources, both EU and non-EU, such as marketing data services.

Legitimate interests:  Sharing relevant, timely and industry-specific information on related business services.

Where we receive information about you in error

If we receive information about you from a third party in error and/or we do not have a legal basis for processing that information, we will delete your information.

This section sets out the circumstances in which will disclose information about you to third parties and any additional purposes for which we use your information.

Disclosure of your information to service providers

We use a number of third parties to provide us with services which are necessary to run our business or to assist us with running our business.

These include the following: Internet services, IT service providers and web developers.

Our third-party service providers are located both inside and outside of the European Economic Area.

Your information will be shared with these service providers where necessary to provide you with the service you have requested, whether that is accessing our website or ordering goods and services from us.

We do not display the identities of our service providers publicly by name for security and competitive reasons. If you would like further information about the identities of our service providers, however, please contact us directly by email and we will provide you with such information where you have a legitimate reason for requesting it (where we have shared your information with such service providers, for example).

Legal basis for processing:  Legitimate interests (Article 6(1)(f) of the General Data Protection Regulation).

Legitimate interest relied on:  Where we share your information with these third parties in a context other than where is necessary to perform a contract (or take steps at your request to do so), we will share your information with such third parties in order to allow us to run and manage our business efficiently.

Legal basis for processing:  Necessary to perform a contract and/or to take steps at your request prior to entering into a contract (Article 6(1)(b) of the General Data Protection Regulation).

Reason why necessary to perform a contract:  We may need to share information with our service providers to enable us to perform our obligations under that contract or to take the steps you have requested before we enter into a contract with you.

Disclosure and use of your information for legal reasons

Indicating possible criminal acts or threats to public security to a competent authority.

If we suspect that criminal or potential criminal conduct has occurred, we will in certain circumstances need to contact an appropriate authority, such as the police. This could be the case, for instance, if we suspect that fraud or a cyber-crime has been committed or if we receive threats or malicious communications towards us or third parties.

We will generally only need to process your information for this purpose if you were involved or affected by such an incident in some way.

Legitimate interests:  Preventing crime or suspected criminal activity (such as fraud).

In connection with the enforcement or potential enforcement our legal rights

We will use your information in connection with the enforcement or potential enforcement of our legal rights including, for example, sharing information with debt collection agencies if you do not pay amounts owed to us when you are contractually obliged to do so. Our legal rights may be contractual (where we have entered into a contract with you) or non-contractual (such as legal rights that we have under copyright law or tort law).

Legitimate interest:  Enforcing our legal rights and taking steps to enforce our legal rights.

In connection with a legal or potential legal dispute or proceedings

We may need to use your information if we are involved in a dispute with you or a third party for example, either to resolve the dispute or as part of any mediation, arbitration or court resolution or similar process.

Legitimate interest(s):  Resolving disputes and potential disputes.

This section sets out how long we retain your information. We have set out specific retention periods where possible. Where that has not been possible, we have set out the criteria we use to determine the retention period.

Retention periods

Server log information: We retain information on our server logs for 3 months.

Order information: When you place an order for goods and services, we retain that information for seven years following the end of the financial year in which you placed your order, in accordance with our legal obligation to keep records for tax purposes.

Correspondence and enquiries: When you make an enquiry or correspond with us for any reason, whether by email or via our contact form or by phone, we will retain your information for as long as it takes to respond to and resolve your enquiry, and for 36 further months, after which we will archive your information.

Newsletter: We retain the information you used to sign up for our newsletter for as long as you remain subscribed (i.e. you do not unsubscribe).

Registration: We retain the information you used to register for as long as you remain subscribed (i.e. you do not unsubscribe).

Criteria for determining retention periods

In any other circumstances, we will retain your information for no longer than necessary, taking into account the following:

• the purpose(s) and use of your information both now and in the future (such as whether it is necessary to continue to store that information in order to continue to perform our obligations under a contract with you or to contact you in the future);
• whether we have any legal obligation to continue to process your information (such as any record-keeping obligations imposed by relevant law or regulation);
• whether we have any legal basis to continue to process your information (such as your consent);
• how valuable your information is (both now and in the future);
• any relevant agreed industry practices on how long information should be retained;
• the levels of risk, cost and liability involved with us continuing to hold the information;
• how hard it is to ensure that the information can be kept up to date and accurate; and
• any relevant surrounding circumstances (such as the nature and status of our relationship with you).

We take appropriate technical and organisational measures to secure your information and to protect it against unauthorised or unlawful use and accidental loss or destruction, including:

• only sharing and providing access to your information to the minimum extent necessary, subject to confidentiality restrictions where appropriate, and on an anonymised basis wherever possible;
• using secure servers to store your information;
• verifying the identity of any individual who requests access to information prior to granting them access to information;
• using Secure Sockets Layer (SSL) software to encrypt any payment transactions you make on or via our website;
• only transferring your information via closed system or encrypted data transfers;

Transmission of information to us by email

Transmission of information over the internet is not entirely secure, and if you submit any information to us over the internet (whether by email, via our website or any other means), you do so entirely at your own risk.

We cannot be responsible for any costs, expenses, loss of profits, harm to reputation, damages, liabilities or any other form of loss or damage suffered by you as a result of your decision to transmit information to us by such means.

Your information may be transferred and stored outside the European Economic Area (EEA) in the circumstances set out earlier in this policy.

We will also transfer your information outside the EEA or to an international organisation in order to comply with legal obligations to which we are subject (compliance with a court order, for example). Where we are required to do so, we will ensure appropriate safeguards and protections are in place.

Subject to certain limitations on certain rights, you have the following rights in relation to your information, which you can exercise by writing to the data controller using the details provided at the top of this policy.

• to request access to your information and information related to our use and processing of your information;
• to request the correction or deletion of your information;
• to request that we restrict our use of your information;
• to receive information which you have provided to us in a structured, commonly used and machine-readable format (e.g. a CSV file) and the right to have that information transferred to another data controller (including a third-party data controller);
• to object to the processing of your information for certain purposes (for further information, see the section below entitled 'Your right to object to the processing of your information for certain purposes'); and
• to withdraw your consent to our use of your information at any time where we rely on your consent to use or process that information. Please note that if you withdraw your consent, this will not affect the lawfulness of our use and processing of your information on the basis of your consent before the point in time when you withdraw your consent.

In accordance with Article 77 of the General Data Protection Regulation, you also have the right to lodge a complaint with a supervisory authority, in particular in the Member State of your habitual residence, place of work or of an alleged infringement of the General Data Protection Regulation.

Further information on your rights in relation to your personal data as an individual

You can find out further information about your rights, as well as information on any limitations which apply to those rights, by reading the underlying legislation contained in Articles 12 to 22 and 34 of the General Data Protection Regulation, which is available here: http://ec.europa.eu/justice/data-protection/reform/files/regulation_oj_en.pdf

Verifying your identity where you request access to your information

Where you request access to your information, we are required by law to use all reasonable measures to verify your identity before doing so.

These measures are designed to protect your information and to reduce the risk of identity fraud, identity theft or general unauthorised access to your information.

How we verify your identity

Where we possess appropriate information about you on file, we will attempt to verify your identity using that information.

If it is not possible to identity you from such information, or if we have insufficient information about you, we may require original or certified copies of certain documentation in order to be able to verify your identity before we are able to provide you with access to your information.

We will be able to confirm the precise information we require to verify your identity in your specific circumstances if and when you make such a request.

Your right to object

You have the following rights in relation to your information, which you may exercise in the same way as you may exercise by writing to the data controller using the details provided at the top of this policy.

• to object to us using or processing your information where we use or process it in order to  carry out a task in the public interest or for our legitimate interests , including ‘profiling’ (i.e. analysing or predicting your behaviour based on your information) based on any of these purposes; and
• to object to us using or processing your information for  direct marketing purposes (including any profiling we engage in that is related to such direct marketing).

You may also exercise your right to object to us using or processing your information for direct marketing purposes by:

• clicking the unsubscribe link contained at the bottom of any marketing email we send to you and following the instructions which appear in your browser following your clicking on that link;
• sending an email to [email protected] , asking that we stop sending you marketing communications or by including the words “OPT OUT”.

Sensitive Personal Information

‘Sensitive personal information’ is information about an individual that reveals their racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, genetic information, biometric information for the purpose of uniquely identifying an individual, information concerning health or information concerning a natural person’s sex life or sexual orientation.

Our website may allow you to register ‘Sensitive Information’, however when we ask for this, you will be considered to have explicitly consented to us processing that sensitive personal information under Article 9(2)(a) of the General Data Protection Regulation.

We update and amend our Privacy Policy from time to time.

Minor changes to our Privacy Policy 

Where we make minor changes to our Privacy Policy, we will update our Privacy Policy with a new effective date stated at the beginning of it. Our processing of your information will be governed by the practices set out in that new version of the Privacy Policy from its effective date onwards.

Major changes to our Privacy Policy or the purposes for which we process your information 

Where we make major changes to our Privacy Policy or intend to use your information for a new purpose or a different purpose than the purposes for which we originally collected it, we will notify you by email (where possible) or by posting a notice on our website.

We will provide you with the information about the change in question and the purpose and any other relevant information before we use your information for that new purpose.

Wherever required, we will obtain your prior consent before using your information for a purpose that is different from the purposes for which we originally collected it.

Because we care about the safety and privacy of children online, we comply with the Children’s Online Privacy Protection Act of 1998 (COPPA). COPPA and its accompanying regulations protect the privacy of children using the internet. We do not knowingly contact or collect information from persons under the age of 18. The website is not intended to solicit information of any kind from persons under the age of 18.

It is possible that we could receive information pertaining to persons under the age of 18 by the fraud or deception of a third party. If we are notified of this, as soon as we verify the information, we will, where required by law to do so, immediately obtain the appropriate parental consent to use that information or, if we are unable to obtain such parental consent, we will delete the information from our servers. If you would like to notify us of our receipt of information about persons under the age of 18, please do so by contacting us by using the details at the top of this policy.

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

OUR COOKIES AND YOU

Hello! If you are reading this, then you care about privacy – and your privacy is very important to us. Cookies are an important part of almost all online companies these days, and this page describes what they are, how we use them, what data they collect, and most importantly, how you can change your browser settings to turn them off.

WHAT IS A COOKIE?

A cookie is a file containing an identifier (a string of letters and numbers) that is sent by a web server to a web browser and is stored by the browser. The identifier is then sent back to the server each time the browser requests a page from the server.

Cookies may be either “persistent” cookies or “session” cookies: a persistent cookie will be stored by a web browser and will remain valid until its set expiry date, unless deleted by the user before the expiry date; a session cookie, on the other hand, will expire at the end of the user session, when the web browser is closed.

Cookies do not typically contain any information that personally identifies a user, but personal information that we store about you may be linked to the information stored in and obtained from cookies.

HOW WE USE COOKIES?

We use cookies for a number of different purposes. Some cookies are necessary for technical reasons; some enable a personalized experience for both visitors and registered users; and some allow the display of advertising from selected third party networks. Some of these cookies may be set when a page is loaded, or when a visitor takes a particular action (clicking the “like” or “follow” button on a post, for example).

WHAT COOKIES DO SURVEYMETHODS.COM USE?

We use cookies for the following purposes:

WHAT COOKIES ARE USED BY OUR SERVICE PROVIDERS?

Our service providers use cookies and those cookies may be stored on your computer when you visit our website.

Google Analytics

We use Google Analytics to analyse the use of our website. Google Analytics gathers information about website use by means of cookies. The information gathered relating to our website is used to create reports about the use of our website. Google’s privacy policy is available at https://www.google.com/policies/privacy/

DoubleClick/Google Adwords

We use Google Adwords which also owns DoubleClick for marketing and remarketing purposes.  Cookies are placed on your PC to help us track our adverts performance, as well as to help tailor our marketing to your needs.  You can view Googles Privacy policy here https://policies.google.com/privacy

Facebook and Facebook Pixel

We use Facebook and Facebook Pixel to track our campaigns and to provide social media abilities on our website such as visiting our Facebook page, liking content and more. To view Facebooks Privacy Policy click here https://www.facebook.com/policy.php .

We use hubspot to manage our relationship with our customers and to track conversions on our website.  You can view HubSpots Privacy Policy here https://legal.hubspot.com/privacy-policy

MANAGING COOKIES

Most browsers allow you to refuse to accept cookies and to delete cookies. The methods for doing so vary from browser to browser, and from version to version. You can, however, obtain up-to-date information about blocking and deleting cookies via these links:

https://support.google.com/chrome/answer/95647?hl=en

https://support.mozilla.org/en-US/kb/enable-and-disable-cookies-website-preferences

https://www.opera.com/help/tutorials/security/cookies/

https://support.microsoft.com/en-gb/help/17442/windows-internet-explorer-delete-manage-cookies

https://support.apple.com/kb/PH21411

https://privacy.microsoft.com/en-us/windows-10-microsoft-edge

Blocking all cookies will have a negative impact upon the usability of many websites. If you block cookies, you will not be able to use all the features on our website.

• Privacy Policy

Home » Limitations in Research – Types, Examples and Writing Guide

Limitations in Research – Types, Examples and Writing Guide

Table of Contents

Limitations in Research

Limitations in research refer to the factors that may affect the results, conclusions , and generalizability of a study. These limitations can arise from various sources, such as the design of the study, the sampling methods used, the measurement tools employed, and the limitations of the data analysis techniques.

Types of Limitations in Research

Types of Limitations in Research are as follows:

Sample Size Limitations

This refers to the size of the group of people or subjects that are being studied. If the sample size is too small, then the results may not be representative of the population being studied. This can lead to a lack of generalizability of the results.

Time Limitations

Time limitations can be a constraint on the research process . This could mean that the study is unable to be conducted for a long enough period of time to observe the long-term effects of an intervention, or to collect enough data to draw accurate conclusions.

Selection Bias

This refers to a type of bias that can occur when the selection of participants in a study is not random. This can lead to a biased sample that is not representative of the population being studied.

Confounding Variables

Confounding variables are factors that can influence the outcome of a study, but are not being measured or controlled for. These can lead to inaccurate conclusions or a lack of clarity in the results.

Measurement Error

This refers to inaccuracies in the measurement of variables, such as using a faulty instrument or scale. This can lead to inaccurate results or a lack of validity in the study.

Ethical Limitations

Ethical limitations refer to the ethical constraints placed on research studies. For example, certain studies may not be allowed to be conducted due to ethical concerns, such as studies that involve harm to participants.

Examples of Limitations in Research

Some Examples of Limitations in Research are as follows:

Research Title: “The Effectiveness of Machine Learning Algorithms in Predicting Customer Behavior”

Limitations:

• The study only considered a limited number of machine learning algorithms and did not explore the effectiveness of other algorithms.
• The study used a specific dataset, which may not be representative of all customer behaviors or demographics.
• The study did not consider the potential ethical implications of using machine learning algorithms in predicting customer behavior.

Research Title: “The Impact of Online Learning on Student Performance in Computer Science Courses”

• The study was conducted during the COVID-19 pandemic, which may have affected the results due to the unique circumstances of remote learning.
• The study only included students from a single university, which may limit the generalizability of the findings to other institutions.
• The study did not consider the impact of individual differences, such as prior knowledge or motivation, on student performance in online learning environments.

Research Title: “The Effect of Gamification on User Engagement in Mobile Health Applications”

• The study only tested a specific gamification strategy and did not explore the effectiveness of other gamification techniques.
• The study relied on self-reported measures of user engagement, which may be subject to social desirability bias or measurement errors.
• The study only included a specific demographic group (e.g., young adults) and may not be generalizable to other populations with different preferences or needs.

How to Write Limitations in Research

When writing about the limitations of a research study, it is important to be honest and clear about the potential weaknesses of your work. Here are some tips for writing about limitations in research:

• Identify the limitations: Start by identifying the potential limitations of your research. These may include sample size, selection bias, measurement error, or other issues that could affect the validity and reliability of your findings.
• Be honest and objective: When describing the limitations of your research, be honest and objective. Do not try to minimize or downplay the limitations, but also do not exaggerate them. Be clear and concise in your description of the limitations.
• Provide context: It is important to provide context for the limitations of your research. For example, if your sample size was small, explain why this was the case and how it may have affected your results. Providing context can help readers understand the limitations in a broader context.
• Discuss implications : Discuss the implications of the limitations for your research findings. For example, if there was a selection bias in your sample, explain how this may have affected the generalizability of your findings. This can help readers understand the limitations in terms of their impact on the overall validity of your research.
• Provide suggestions for future research : Finally, provide suggestions for future research that can address the limitations of your study. This can help readers understand how your research fits into the broader field and can provide a roadmap for future studies.

Purpose of Limitations in Research

There are several purposes of limitations in research. Here are some of the most important ones:

• To acknowledge the boundaries of the study : Limitations help to define the scope of the research project and set realistic expectations for the findings. They can help to clarify what the study is not intended to address.
• To identify potential sources of bias: Limitations can help researchers identify potential sources of bias in their research design, data collection, or analysis. This can help to improve the validity and reliability of the findings.
• To provide opportunities for future research: Limitations can highlight areas for future research and suggest avenues for further exploration. This can help to advance knowledge in a particular field.
• To demonstrate transparency and accountability: By acknowledging the limitations of their research, researchers can demonstrate transparency and accountability to their readers, peers, and funders. This can help to build trust and credibility in the research community.
• To encourage critical thinking: Limitations can encourage readers to critically evaluate the study’s findings and consider alternative explanations or interpretations. This can help to promote a more nuanced and sophisticated understanding of the topic under investigation.

When to Write Limitations in Research

Limitations should be included in research when they help to provide a more complete understanding of the study’s results and implications. A limitation is any factor that could potentially impact the accuracy, reliability, or generalizability of the study’s findings.

It is important to identify and discuss limitations in research because doing so helps to ensure that the results are interpreted appropriately and that any conclusions drawn are supported by the available evidence. Limitations can also suggest areas for future research, highlight potential biases or confounding factors that may have affected the results, and provide context for the study’s findings.

Generally, limitations should be discussed in the conclusion section of a research paper or thesis, although they may also be mentioned in other sections, such as the introduction or methods. The specific limitations that are discussed will depend on the nature of the study, the research question being investigated, and the data that was collected.

Examples of limitations that might be discussed in research include sample size limitations, data collection methods, the validity and reliability of measures used, and potential biases or confounding factors that could have affected the results. It is important to note that limitations should not be used as a justification for poor research design or methodology, but rather as a way to enhance the understanding and interpretation of the study’s findings.

Importance of Limitations in Research

Here are some reasons why limitations are important in research:

• Enhances the credibility of research: Limitations highlight the potential weaknesses and threats to validity, which helps readers to understand the scope and boundaries of the study. This improves the credibility of research by acknowledging its limitations and providing a clear picture of what can and cannot be concluded from the study.
• Facilitates replication: By highlighting the limitations, researchers can provide detailed information about the study’s methodology, data collection, and analysis. This information helps other researchers to replicate the study and test the validity of the findings, which enhances the reliability of research.
• Guides future research : Limitations provide insights into areas for future research by identifying gaps or areas that require further investigation. This can help researchers to design more comprehensive and effective studies that build on existing knowledge.
• Provides a balanced view: Limitations help to provide a balanced view of the research by highlighting both strengths and weaknesses. This ensures that readers have a clear understanding of the study’s limitations and can make informed decisions about the generalizability and applicability of the findings.

Advantages of Limitations in Research

Here are some potential advantages of limitations in research:

• Focus : Limitations can help researchers focus their study on a specific area or population, which can make the research more relevant and useful.
• Realism : Limitations can make a study more realistic by reflecting the practical constraints and challenges of conducting research in the real world.
• Innovation : Limitations can spur researchers to be more innovative and creative in their research design and methodology, as they search for ways to work around the limitations.
• Rigor : Limitations can actually increase the rigor and credibility of a study, as researchers are forced to carefully consider the potential sources of bias and error, and address them to the best of their abilities.
• Generalizability : Limitations can actually improve the generalizability of a study by ensuring that it is not overly focused on a specific sample or situation, and that the results can be applied more broadly.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Majority of Americans over 50 worry they won't have enough money for retirement: Study

Among aarp survey findings: 61% of americans 50 and up are worried they won’t have enough money for retirement. and only 21 percent of people have a retirement plan..

An increasing number of people are worried that they won’t have enough money to live comfortably in retirement, and men aren’t as financially secure as they once were, according to an annual survey from AARP.

The AARP Financial Security Trends Survey , conducted in January and released in April, included interviews with more than 8,300 Americans over 30 across every state in the country. Conducted by NORC at the University of Chicago , the survey aims to analyze the financial experiences and attitudes among Americans.

One of the survey’s biggest findings is that 61% of those 50 and up are worried they won’t have enough money for retirement, Indira Venkat, senior vice president of research at AARP, told USA TODAY on Wednesday.

And if you break those numbers down even more, one in five of people who have not retired have no savings at all, Venkat said.

“Among retirement savers who are 50 and over, there's a growing number compared to last year that don't expect that they will save sufficiently if they continue at the current rate that they save,” she said. 

Those numbers “should be a wake-up call for all of us,” she said.

Adults are worried about having enough for retirement. What does this mean?

Venkat said the reason some people may be worried about saving for retirement is because they don’t have access to retirement plans through their jobs.

“Today, almost half of those who are in the private sector workforce, and that's 57 million workers, don't have the option to save for retirement through their work,” she said. 

People who have access to retirement plans are more likely to save, she said.

Another reason for retirement woes is inflation. Calling it a “bread and butter” and “kitchen table” issue, she said many people are struggling to afford basic necessities and gas. 

In fact, over 70% of all respondents expressed concern about prices rising at a faster rate than their income. 

Many people also have month-to-month balances on credit cards, she said. According to the researchers, the average amount of credit card debt carried from month to month increased from $7,538 in January 2023 to $8,169 in January 2024.

Gender and financial planning

AARP noted findings among men specifically.

Among men who are regularly saving for retirement, 28% are saving 10% or more of their income. In January 2022, more men – 35% – were saving 10% or more.

Other findings include:

• 40% of men are worried about basic expenses (up from 33% in January 2022)
• 42% of all men age 30 and up describe their financial situation as “only fair” or “poor” (up from 34% in January 2022)
• 43% of men carry a credit card balance (up from 38% in January 2022)
• 62% of men have emergency savings today (down from 69% in January 2022)

“We are seeing that men today are a little bit more worried than they were in prior years about their financial security and situation,” Venkat said. “Forty-two percent of men, if they're asked to describe their financial situation, say it's terrible.”

The team also compared responses among men and women, finding that:

• 62% of men are more likely to have an emergency savings vs. 58% of women
• 42% of men are less likely to be worried about managing debt vs. 47% of women
• 61% of men with debt are more likely to view their debt as manageable vs. 52% of women

Limitations of the survey

Venkat said the survey was “very robust” and was done using probability samples, but one limitation is that it was conducted in English and Spanish only. 

“To the extent there are other dominant spoken languages in the home, those individuals are not represented in this study,” she said. “And then, of course, the standard margin of error with any survey applies to this.”

Financial planning among survey respondents

There were some surprising findings surrounding financial planning, Venkat said.

Among older adults who are not yet retired, 94% said it is important for them to have a plan to manage their money during retirement. Among them, just 21% of them have a plan for how to manage their money in retirement, she said.

She said AARP has resources to help people plan, including tools like a Social Security calculator, an AARP retirement calculator, a required minimum distribution calculator and a 401(k) calculator .

The concerns respondents expressed while completing the survey further prove the importance of having a plan for retirement, AARP says.

“Plans need to address not only how to build savings during working years," the company writes, "but also how to spend money in retirement in a way that will allow it to last."

Saleen Martin is a reporter on USA TODAY's NOW team. She is from Norfolk, Virginia – the 757. Follow her on Twitter at @SaleenMartin or email her at [email protected] .