Protocols using multiple data collection modes (in person, by telephone, online, etc.) are not new in Official Statistics surveys. However, the proliferation of Internet use, the evolution of household practices in terms of communication and participation and the standardisation of data collection tools are profound changes that are favourable to the expansion of mixed-mode data collection in household surveys. This evolution has been keenly promoted within official statistical services since the 2010s and has recently accelerated in the specific context of the Covid-19 health crisis. More than ever, it is emerging as a strong strategic focus for Official Statistics. However, mixed-mode data collection complicates both data collection processes and data processing following collection, creating a risk of breaks in series. Beyond the opportunities that this paradigm shift offers for household surveys, it is important to identify the challenges that this transition poses to statisticians. The choices made in France are based on a consideration of international dimensions.

The mixed-mode nature of a survey relates to both the contact protocol and the collection protocol. The initial contact very often consists of a letter, although the collection is rarely carried out via a paper questionnaire.

In reality, however, we refer to a mixed-mode survey when the collection protocol involves the use of multiple information collection modes: in person with an interviewer, by telephone, online, etc.

Multiple parameters come into play when determining the appropriateness of mixed-mode data collection: the target population, the information available to reach it (the sampling frame and contact data) and the subject. Several Official Statistics surveys have thus been conducted in a mixed-mode manner for many years, e.g.:

Over the past decade, INSEE has embarked on a series of experiments to assess the impact of the introduction of online collection in survey protocols. This increasing use of online collection leads to a fresh analysis of mixed-mode protocols. In fact, in Official Statistics surveys, online collection is rarely envisaged as the only collection mode and is viewed as more of a supplementary mode.

There are three reasons for switching to mixed-mode surveys.

The first reason is the downward trend in the response rate for household surveys, although the situation in France is less problematic than in other European countries (Ouvrir dans un nouvel ongletLuiten et al., 2020). However, diversification of the ways of contacting people increases the chances of reaching them and, in principle, the collection rate, even though the response rate does not in itself constitute a definitive indicator of quality (Ouvrir dans un nouvel ongletGroves and Peytcheva, 2008). Thus, recent work (Ouvrir dans un nouvel ongletCornesse and Bosnjak, 2018) has shown that mixed-mode surveys are of better quality in terms of representativeness and that surveys conducted exclusively online have low participation rates and significant selection effects.

Generally speaking, conducting surveys in person seems better than doing so by telephone and online in terms of the representativeness of certain segments of the population, particularly the least privileged or educated, immigrants, those with a poor command of the French language or those living in urban areas (Ouvrir dans un nouvel ongletBuelens and Van den Brakel, 2010).

The development of household appliances and practices is also a favourable context for mixed-mode surveys. Over the last decade, the rate of households obtaining Internet access (hardware and subscriptions) has skyrocketed in France: Internet is now in common use. Such a situation now brings France more in line with countries with more long-standing Internet take-up, such as the Netherlands, the Nordic countries, the United Kingdom and Germany. The conditions are therefore increasingly in favour of using this mode to survey households. In addition, the expectations of the households themselves seem high: some of the households surveyed increasingly fail to understand why such a service is not available, given that recent advances in web-based administrative processes are highly visible.

However, some people only access the Internet via smartphone: over 60% of the population of Europe are able to access the Internet via mobile phone. This underscores the challenge of offering an interface and questionnaire designed from the outset for this type of media (Mobile first), the page format of which adapts in an ergonomic way (responsive design) (Box 1).

Telephone surveys however, which have been used widely in the 1990s by polling institutes and part of Official Statistics, are now faced with lower participation rates, due to the weariness caused by incessant sales calls by phone. Landline telephones, which were the basis of such surveys until the 2010s, are now a tool that is very rarely answered (or connected) by new generations.

Budget constraints also generate an interest in the costs associated with the various collection modes: interviews with interviewers (in person or by telephone) are the most expensive and it is preferable to use them only for subjects or populations for which their experience is essential.

Given that each collection mode has its strengths and weaknesses in terms of coverage and selection bias, it is important to take advantage of the benefits that each of them offers when developing survey protocols.

The analysis of measurement bias – i.e. of the impact of the collection mode on the response to surveys – contrasts self-administered methods (paper and online) and the modes involving interviewers (in person and by telephone). The two main forms of bias caused by the presence or absence of an interviewer are social desirability and satisficing:

Furthermore, self-administered survey modes suffer from lower response rates and a greater risk of self-selection relating to the subject of the survey, given that responding requires a more pro-active approach by respondents.

These effects may vary depending on the characteristics of the respondents. Thus, for the most highly educated young people, people with digital skills and an interest in the digital world, responding online may seem perfectly natural: their responses via this mode are unlikely to be affected by satisficing. Conversely, these populations may be more difficult to interview by telephone or in person and their responses may be affected by a high level of social desirability in the presence of an interviewer. Furthermore, the contact details available vary: while the postal address is known for all households, this is not always the case for the mobile or landline phone numbers and the email address. The availability of these different contact details differs throughout the population, together with the ability to effectively contact people and their appetite to respond via the different collection modes.

A study conducted among a population of migrants thus showed that paper was the most appropriate mode for men and the telephone was most suitable for women – particularly to limit the impact of their partner’s presence (Ouvrir dans un nouvel ongletKappelhof, 2015).

Each mode therefore has its advantages and disadvantages, depending on the subject, the target populations and the contact modes available (Figure 1).

Figure 1. Each collection mode has its own characteristics

Protocols created on the basis of the different possibilities offered by the collection modes take very different forms (Figure 2).

The different collection modes can be proposed to respondents at the same time, with the modes being placed into some form of competition (this is referred to as a competitive mixed-mode protocol), or they can be proposed successively, with an alternative being proposed only to non-respondents in the previous phase (this is referred to as a sequential mixed-mode protocol). Generally speaking, a sequential mixed-mode protocol involves proposing the least costly collection modes first.

Most protocols result in a period in which several modes are competing: this is the case with sequential online/telephone protocols for which the ability to respond online remains available even when the telephone survey phase has begun. This is referred to as a combined protocol.

Figure 2. With mixed-mode protocols, data collection is becoming more flexible

The main limitation of the sequential protocol is that it makes some respondents respond using a mode that is not their preferred mode and that this could degrade the quality of their responses (a sensitive subject, an aversion to the mode, etc.). It also generates a risk that respondents who would have agreed to respond if they had been approached by an interviewer in the first place will find themselves in a worse disposition when such contact is made after they have first refused to respond via a self-administered mode of collection: persistence with their initial refusal, marked reluctance to make the necessary efforts to ensure the quality of the answers, etc. Furthermore, the sequential protocol imposes a minimum collection time at each stage, without which the chances of success of each mode are diminished. A short period of field time is therefore an incentive to propose a competitive or combined mixed-mode protocol rather than a sequential mixed-mode protocol.

However, generally speaking, the sequential mixed-mode protocol is considered most effective, particularly in financial terms (Dillman et al., 2014). In addition, leaving the choice of response mode to the respondent can be perceived as an additional cognitive burden (Schwartz, 2005), at a time when the respondent is often waiting for the interviewer, or the protocol if there is no interviewer, to guide them to allow them to complete the task as quickly as possible. Leaving the choice of response mode to the respondent is particularly interesting in panel surveys or with re-interviews, as the respondent can express their preference at the end of the first round of interviewing and provide their contact details, facilitating the use of a particular mode in subsequent rounds.

An integrated mixed-mode protocol refers to the use of an additional collection mode during the collection for all respondents. This is the case, for example, when an interviewer in the home allows the respondent to answer part of the questionnaire on a computer alone, as in the case of the CVS survey, where they can use a headset and answer alone via the keyboard.

This type of protocol seems particularly well suited to sensitive questions (Ouvrir dans un nouvel ongletTurner et al., 1998). It undoubtedly makes it possible to take advantage of the different modes requested, but it is generally the most expensive.

In surveys conducted via telephone, it is also possible to request a switch to fully completing the questionnaire online (Ouvrir dans un nouvel ongletLegleye and Charrance, 2021). For a series of sensitive questions, it is possible for the interviewer to be replaced by Interactive Voice Response (IVR), with which it is hoped that social desirability bias will be reduced (Ouvrir dans un nouvel ongletKreuter et al., 2008). This leads to quite a number of people abandoning the questionnaire during collection, as the experience is probably not considered very pleasant by respondents who lose the motivation caused by the audible presence of the interviewer and may result in the respondent not continuing with the questionnaire. It therefore seems preferable to use this collection mode only for short series of questions.

In longitudinal surveys, it is possible to adapt the collection modes to the different rounds of interviewing. The presence of an interviewer is often necessary to start the collection process. In contrast, subsequent rounds of interviewing (re-interviews) can often be carried out using less costly and less invasive collection modes, such as via telephone or online. In France, the Labour Force Survey and the Rents and Charges Survey (enquête Loyers et Charges) have been based on this type of protocol for many years now, combining an initial round of interviewing in person, followed by re-interviewing via telephone before the end of the process in-person. Since January 2021, a sequential online/telephone mixed-mode protocol is proposed for re‑interviewing related to the Labour Force Survey (Guillaumat-Tailliet and Tavan, 2021).

There are also two-phase protocols based on a double survey. The first phase entails a short survey conducted among a very large number of people, based on questions aimed at identifying a population of specific interest (people with disabilities, victims of violence, sexual minorities, etc.). Given the size of the sample, it may be advantageous to implement a sequential mixed-mode protocol using the less costly collection modes: online, then paper, then potentially telephone, for some of the non-respondents. The second phase, which can also use a mixed-mode protocol, is carried out among a selection of respondents from the first phase, in which certain characteristics of interest are over-represented thanks to the information from the filter survey. It can also use a mixed-mode protocol. Such a protocol was successfully adopted for the Everyday Life and Health/Autonomy (Vie quotidienne et santé (VQS)/Autonomie) survey of the Directorate of Research, Studies, Evaluation and Statistics (service statistique du ministère des Solidarités et de la Santé – DREES) and for the Ouvrir dans un nouvel ongletGender and Security (Genre et Sécurité – GENESE) survey by the Ministerial Statistical Department for Internal Security (SSMSI) of the Ministry of the Interior.

Finally, the increasing market penetration of smartphones now offers researchers the ability to collect data from smartphone users through passive data collection through the phone’s application use and history (geolocation, physical movements, online behaviour, etc.). Some of the collection can even be done on a smartphone (a diary or short questions at regular intervals, for example). This opportunity should not conceal the difficulties that some respondents face when they download an application to their smartphone or the challenges raised in relation to their consent to the collection and use of such data.

Ultimately, the constraints of each survey project may lead designers to focus their choices among the possibilities mentioned above (see the choices made for certain major European surveys in Box 2).

These different types of mixed-mode protocols may not apply in the same way to all sampled units in a single survey. Thus, in recent years, protocols allowing the adaptation of the data collection strategy to the socio-demographic profile of the respondents have been designed and are now beginning to be implemented (Ouvrir dans un nouvel ongletChun et al., 2018). Adaptive design thus uses the information available prior to data collection in order to select the collection mode that, in principle, is the most effective for each unit surveyed, whether in terms of response rate or limitation of measurement bias.

For example, in the case of surveys on alcohol, tobacco and illicit drug use, sex and age appear to be dividing characteristics in terms of choosing the collection mode (self-administered or via telephone) (Ouvrir dans un nouvel ongletBeck et al., 2014). One of the differences between adaptive design and the other forms of mixed-mode protocols is that the preference for a particular mode is fixed from the outset by the designer (as opposed to competitive mixed-mode protocols in which the respondent makes the choice) and that it is assumed (such a mode is better suited to such a sub-population).

Given that the length of the questionnaire is a factor that limits interest in the use of the Internet, it is also possible to target a population with a good ability to respond online without any degradation of quality (young people, from households of 1 or 2 people, who therefore do not have an overly complex household to describe), while the rest of the population is interviewed via telephone or in person.

There are also more complicated protocols involving the use of the information available during collection (data collected by questionnaire or obtained from paradata) to adjust the collection strategy for the rest of the sample: this is referred to as responsive design. It may thus be decided, during the collection, to focus efforts on a particular profile by targeting it directly or indirectly via a specific collection mode. Before being used in French Official Statistics surveys, this type of protocol would need to be tested to assess its appropriateness.

The implementation of a mixed-mode protocol requires that all stages of survey design and production be adapted to provide an ergonomic approach that minimises the effects of the mode. This effort is greatly facilitated by tools for the design and generation of collection media based on active metadata.

The first challenge is to adapt the questionnaire to the different modes envisaged: should we go all out to retain the same structure and the same questions for all modes or, instead, should we try to do the best we can in each mode? International recommendations strongly support the first solution (Ouvrir dans un nouvel ongletDe Leeuw, 2018), although the most recent work calls for a third way seeking to optimise each questionnaire while minimising differences between the different modes.

This omni-mode (or agnostic) approach, which is used by multiple national statistical institutes (e.g. the Netherlands and the United Kingdom), consists of designing the questionnaire independently of the collection mode and then making the necessary adaptations in each mode, so as to minimise the mode effects. The main recommendations in this area are:

The necessary adaptations include choices regarding graphic charts, the size and position of buttons/tick boxes and whether the “don’t know”/“do not want to answer” options should be shown always, only for a second attempt or never.

Generally speaking, the switch to mixed-mode protocols, particularly the introduction of self‑administered collection, is a good opportunity to re-examine what it is that the protocol is attempting to measure and to thus review the complications that may have been introduced over the various editions of surveys. If comprehension of the concepts and questions is reliant on the interviewer, it may not be passed on to the respondents themselves, which risks non‑response or incorrect answers.

The use of self-administration thus requires the use of simple question formulations and the provision of examples to provide clarification for respondents, particularly those among the general population, given that no rephrasing or guidance can be provided by the interviewer (Koumarianos and Schreiber, 2021). The way in which instructions, guidance and examples are presented must be considered on an ad hoc basis, to ensure that they are presented in an attractive and appropriate manner. Interviewers perform an important task of checking the information entered and compliance with the instructions during the switch: when the interviewers are absent, all this verification work is transferred to the statisticians in charge of data clearance, with a significant risk of error due to the inability to ask for clarifications regarding the information entered after the fact.

Responding to a survey is always a difficult exercise; however, participation in a survey and the quality of the information collected depend on the length and complexity of the questionnaire. While the presence of an interviewer, particularly in person, opens up possibilities for long and complex questions, self-administration, in contrast, very quickly comes up against the limits of what the least literate section of the population, or the section least interested in or affected by the subject, can or want to accomplish. Beyond a certain length of time that depends on the collection mode, the risk of satisficing increases, along with the risk of random answers, abandonment, increased use of “don’t know”, shorter responses to open questions and an overall decline in quality (Ouvrir dans un nouvel ongletGalesic and Bosnjak, 2009). In the case of mixed-mode surveys, the length of the questionnaire is determined by the collection mode that generates the most constraints for the respondent.

The maximum recommended duration for telephone surveys is around half an hour (Ouvrir dans un nouvel ongletRoberts et al., 2010), but the presence of an interviewer often leads designers to exceed this limit, though they usually do not exceed three quarters of an hour. For online surveys, it is highly recommended not to exceed 20 minutes, with the ideal being a questionnaire with a median response time of 10 minutes (Revilla and Ochoa, 2017).

There are several possible solutions to limit questioning time if reducing the questionnaire is insufficient. The first is to administer the questionnaire in several sessions, which is like “panelising” the survey. This is not without risk or cost: attrition, management of household moves or breakdowns, extension of the reference period, complication of the statistical operations after collection, etc. Adaptive design protocols seek to address these problems by adjusting the questionnaire so that each respondent has an optimal questioning time from the point of view of the quality of the information collected. Thus, it may be the case that not all respondents are asked every question, but instead a common core of questions is determined and then different blocks of questions are randomly allocated to several sub-samples (Ouvrir dans un nouvel ongletBeck and Richard, 2013). The matter of financial incentives, in the case of more difficult or longer questionnaires, is raised in Box 3.

In mixed-mode surveys, the aggregation of data from different collection modes requires certain precautions. Indeed, that which is observed via one collection mode may not be directly comparable to that which is observed via another mode. This difference is called a mode effect. It can be broken down into two parts:

The existence of selection bias stems from one of the objectives of mixed-mode surveys, i.e. from increasing the ways for respondents to respond. It is not in itself a problem, provided that there is no measurement effect and that the set of respondents, regardless of the mode of collection they use, is representative of the target population.

It is therefore important to neutralise all composition biases in order to properly assess that which is exclusively due to the collection mode, i.e. to assess the measurement bias.

To do this, modes are used that make it possible to render those responding via each collection mode perfectly comparable. In practice, however, it can be difficult to distinguish between the biases resulting from these two effects, selection and measurement (Ouvrir dans un nouvel ongletKlausch, 2014). Indeed, all of these modes are based on the strong assumption that there is no non-ignorable selection bias, i.e. an unobserved factor that influences both participation and the value of the variables of interest and which has little correlation with the observed factors for which controls are implemented (Rubin and Little, 2002). In such a case, particularly complex endogenous selection problems occur that are difficult to handle (Ouvrir dans un nouvel ongletLee, 2009; Castell and Sillard, 2021; Ouvrir dans un nouvel ongletHeckman, 1979).

Work carried out on different data sets from French surveys has made it possible to determine the differences between the values obtained by reference surveys conducted in-person and online asking the same questions (Razafindranovona, 2015). These differences are lessened by taking socio-demographic characteristics into consideration and can be lessened further if the control variables are enriched by survey variables that are directly related to the subject of the survey.

However, a residual difference may remain, reflecting both a measurement bias and a composition bias on non-observable variables, such as interest in the subject of the survey (endogenous selection bias).

Collection mode effects are undoubtedly the most powerful constraint on the development of mixed-mode surveys: they complicate analyses, make it difficult to estimate the “true” levels of the survey’s indicators of interest and can lead to breaks in series. Estimating and correcting these biases requires assumptions to be made. To ensure the plausibility of these assumptions, specific resources must be available, which must be considered prior to collection (Box 4).

The MIMOD project (Box 2) has made it possible to identify the modes specifically used in national statistical institutes to aggregate data from different collection modes and to update the knowledge gained on mixed-mode surveys. While there is beginning to be a wealth of experience in measuring the effects of collection modes shared among European countries, experience in correcting such effects remains rare, as does scientific literature on the subject.

There are several possible correction modes, but they remain bound by strong limitations, hence the most commonly taken option of not correcting for collection mode effects.

First, measurement bias is conceptually a problem at partial response level (compared to selection bias which is a problem at complete response level). In this respect, it can seem like a missing-value problem: it is thus effectively processed by imputation. However, this method constitutes a strong methodological, or even ethical, choice as it changes the responses of the respondents. Instead, “parsimonious” methods are given preference, selecting the observations with the most bias so as to minimise changes in the database (Ouvrir dans un nouvel ongletLegleye et al., 2019).

Another method is to use calibration. This makes it possible to neutralise not only selection bias, but also measurement bias at aggregated level, by introducing constraints regarding the levels of the variable of interest. Thus, instead of changing the data, the weighting is changed so as to obtain the correct level in all or part of the sample. In contrast, measurement bias is not corrected at individual level. Again, margins will be given preference on the sub-populations with the most bias and the use of a control sample collected via a single mode is strongly recommended.

It is also possible not to correct the measurement bias, but to instead try to contain it in time: in repeated surveys, it is possible to introduce a constraint in the adjustment that maintains the shares of the different collection modes at constant values (Ouvrir dans un nouvel ongletBuelens and Van den Brakel, 2010). This has the benefit of simplicity, but also raises questions of sustainability if a collection mode disappears or in the event of a change in the shares of the different collection modes over time.

Beyond measurement bias, it may be necessary to correct another bias: a non-observable selection bias affecting all respondents. This issue is not directly related to mixed-mode surveys which, on the contrary, can improve the representativeness of the survey, depending on the protocol. However, mixed-mode surveys are most often associated with the introduction of the Internet and the use of collection modes with a lower response rate and a lower representation of respondents than in person. In such cases, correction methods exist, based on Heckman’s approach (Ouvrir dans un nouvel ongletHeckman, 1979). However, their validity is based on the strong assumption that there is no measurement bias (Castell and Sillard, 2021) because, at present, there is no method that makes it possible to deal with both simultaneously if the choice to respond via a certain method is made by the respondent (Ouvrir dans un nouvel ongletLee, 2009).

Generally speaking, the methods of correcting measurement effects require specific work to be carried out. Thus, not all variables in a survey can be analysed, let alone corrected. It is therefore important to carefully select the main variables of interest. Furthermore, the methods require the assumption that one of the collection modes constitutes the reference, either because it is the historical mode used by a survey series or because it is deemed to produce the best quality measurement. A quality control sample, collected via a single mode (or with embedded mixed-mode collection) is therefore often required.

Thus, the detection of collection mode effects may justify adapting the survey protocol and its sampling method, in particular to ensure that disjointed sub-samples can be collected in accordance with the compositions of the different collection modes (Box 4).

The development of mixed-mode collection gives us an opportunity to reconsider the design of household surveys around fundamental and structural good practices, which guarantee a minimum quality of responses such as meeting a maximum response time for a questionnaire and the use of concepts and formulations that are understandable to the large majority of the population.

In its “mixed-mode collection” programme, INSEE has chosen to propose online as the sole option only in certain contexts (short or sequenced surveys, simple concepts and re-interviews following initial mediated contact). At a time when some polling institutes are abandoning in-person surveys and, in some instances, even telephone surveys, INSEE’s strength is that it has a network of interviewers who are highly experienced in terms of in-person interviews and are well spread throughout France.

Ultimately, the evolution towards mixed-mode data collection is indeed an opportunity to improve surveys in many respects; however, it also represents an organisational cost and a challenge in terms of managing collection mode effects: this should call for us to take greater care in the methodological choices considered when designing the survey. In view of the spread of mixed-mode data collection in Europe, it would be necessary, for example, to shift European regulations towards omni-mode questionnaires, which would have to be simplified compared with those currently in use which were designed in a world dominated by in-person data collection. In autumn 2021, an ad hoc working group (Ouvrir dans un nouvel ongletEurostat, 2021) presented recommendations along those lines: a position paper (coordinated by France) identifies the fields of inquiry for the coming years (in terms of methodological and data collection issues) to adapt European household surveys to the context of mixed-mode data collection.