The importance of a strong Delphi survey
A Delphi consensus study is a methodical approach towards gaining structured, anonymised expert opinion on a topic that may suffer currently from conflicting, ambiguous or absent evidence. It provides citable, credible evidence that can be used to campaign for better decisions to be actioned. The survey stage of a Delphi study is an opportunity to validate the statements developed by the steering group on a scale that is representative of the study population, using a wider expert panel. It provides the steering group with the confidence to make recommendations that are backed up by and advocated for by expert peers who work in the field.
This blog will share some tips for designing a strong Delphi survey which provides the data necessary to create compelling and actionable recommendations that help drive change.
Prepare the Delphi survey introduction
The introductory text that accompanies the survey is important as it maps out the wider context of the project and sets the expectations of the responders so they can give informed consent to participate in the Delphi study. Below are some suggestions on what to include to create a strong survey introduction
- Purpose of the survey – why is the survey being conducted? What research question does it aim to answer?
- Briefly outline the methodology being employed?
- Survey population – Describing the qualifying criteria for the survey population will help the respondent ensure they fit the criteria to fill in the survey.
- Anonymity – It is important to establish that survey responses are anonymous
- Funding – For transparency, it is important to include whether the survey is being funded and who is funding it.
A strong survey introduction will also be beneficial when analysing the data. If the introduction is clear, there will be fewer responses which fail to meet the inclusion criteria that may then need to be excluded from the data set.
Define screening criteria
Screening questions are useful to include as they will make sure the respondent fits the characteristics of the study population. Especially in Delphi studies, it is important to ensure only people who have experience in the topic area that is being researched are completing the survey as this provides the steering group with confidence that the recommendations they ultimately make are supported by peers in the field.
It is advisable to include screening questions at the beginning of the survey to avoid wasting the time of the respondent if they don’t meet the inclusion criteria. Common screening questions for healthcare studies may include questions on role, experience, and geographic region. However, more can be added if the study topic is very specific. It is important to be aware that the more screening questions that are asked the fewer responses you are likely to get.
Consider survey fatigue
When constructing a Delphi survey, consideration should be given towards survey fatigue. Survey fatigue can occur when surveys are poorly designed. Responders can become bored and abandon their survey entry or contaminate the final data set by entering thoughtless responses.
The length of the survey can ease survey fatigue. It has been recommended that HCP surveys should not take longer than 30 minutes to complete and consumer/patient surveys no longer than 20 minutes1. Generally, surveys should be kept as short as possible in order to fit into HCPs’ busy schedules.
Consider the rating scale
Delphi surveys can be developed using a range of rating scales to measure the strength of agreement. Most commonly Likert scales between 4- and 7-points are used. It is recommended to use a 4 or 6-point scale to avoid fence-sitting but allow the strength of agreement to be understood.
Consider statement design
Strong statements are the key to developing actionable recommendations for campaigning. Triducive characterises strong statements if they fit the following criteria:
- Clear & specific
- Short & succinct
- Includes reason & response
- Includes only one concept
- Assertive
Consider the need for translations
Translating a survey into multiple languages can be useful when looking to engage responders from countries where English is not the first language. You may be able to reach a wider audience by reducing any friction for those responders who fear they could misinterpret statement intent or emphasis, despite their proficiency in the English language. Surveys which are translated into the preferred language of the study population will promote inclusivity and provide more accurate data and questions are less likely to be misinterpreted.
However, care should be taken when translating surveys as conceptual variations may occur. Words may mean something (subtly) different when translated which could lead to confusion around the purpose of the questions. It is advised that the draft translation is checked by a native speaker with an understanding of the topic area in hand. Cultural and organisational variations should also be considered. For example, healthcare systems vary worldwide, so some questions may not apply to certain respondent populations.
Key takeaways
- The survey introduction is important as it maps out the wider context of the project and sets the expectations of the responders so they can give informed consent to participate in the Delphi study.
- Statements should be carefully written as they are key to creating actionable recommendations for campaigning.
- Screening criteria help ensure the data collected is from credible experts who have experts in the research field. This will strengthen the validity of the findings.
- Choose a rating scale that avoids fence-sitting.
- Limit survey length to under 30 mins with HCPs to minimise survey fatigue.
- Translations can be very useful for reaching a larger audience however conceptual and cultural and organisational variations will need to be considered when constructing the survey.