| Annette Scherpenzeel (CentERdata, Tilburg University) |
| Effects of data collection technique on the quality of data: A MTMM study of CATI, CAPI and Online interviews |
| |
| In this study, a split-ballot mtmm design is used to compare the quality of the data collected by web interviews with the quality of the data collected by traditional data collection methods. A probability sample of the Dutch population was contacted and interviewed by one of two traditional computer assisted data collection modes: telephone interviewing or face-to-face interviewing. At the end of these interviews, the respondents were asked to become a member of an online household panel. Respondents who were willing to participate but had no internet access were equipped with a user-friendly computer with internet access. All interviewed respondents who joined the panel were asked to fill in the same interview in a web interview format, a few months after the first telephone of face-to-face measurement. The mtmm analysis gives estimates of both reliability as the complement of random error variance and validity as the complement of systematic method variance for every measure in the study. These estimates show that panel data collected using a self-administered web interview are at least as valid and reliable as data collected using more traditional modes of interviewing, when the sample is kept constant. |
|
|
| Nathalie Sonck (PhD fellow at the Centre for Sociological Research, University of Leuven) |
| Representativeness of web surveys to the general public |
| |
| Despite significant time- and cost-reductions, the online survey method has to cope with important methodological problems, which are mainly related to the lack of representativeness of the results. This is especially the case for online access panel surveys. Therefore, the main question that will be addressed is how representative online survey data are for the general public. Based on recent figures of the European Social Survey, it seems that important differences in demographic characteristics still exist between people who have internet access and who haven’t. As a possible solution to reduce possible biases of unrepresentative web surveys, it has been suggested to apply post-stratification weights (to adjust for demographic under- and over-representations in the sample) and, more recently, to use propensity-score adjustments (to correct for differences due to the varying inclination to participate in online panel surveys). The impact of these frequently applied weighting techniques on the representativeness of online survey results was evaluated by conducting a comparison study between an online panel survey and a random reference survey. It was found that these weighting techniques not necessarily make substantive answers to attitude questions (e.g. about politics, work satisfaction, ethnocentrism) similar to those of the general population. |
|
|
| Sonja Utz (Department of Communication Science, VU University) |
| The reparation of trust after negative feedback – the results of two online experiments among eBay users |
| |
| Research on reputation systems has mainly focused on the trust building function of reputation systems. The present research looks also at the trust rebuilding function of reputation systems, more specifically, the role of the short text comments given in reaction to a negative feedback. Online markets are noisy environments; rebuilding trust is therefore often necessary. This paper analyzes whether sellers´ reactions have the potential to re-build the trust between business partners. The central questions were: Do different types of trust violations have more or less detrimental effects on trust? Do reactions of sellers have any trust re-building effects? Which types of sellers' reactions are more effective in re-building trust? A distinction is made between morality-based and competence-based violation of trust and between two types of reactions that a seller can display, namely an apology or denial of the trust violation. Two online experiments among active eBay users in the Netherlands were conducted, using a random sample of members of a large Dutch commercial ‘opt in’ access panel. The advantages of this methodological approach will be discussed. The results show that the text comments accompanying negative feedback indeed influence trustworthiness judgments. Contrary to what has been reported in the literature, denial in case of morality-based violations of trust did not re-build trust. The implications for further research on re-building trust on the internet are discussed. |
|
|
| Katja Lozar Manfreda (Faculty of Social Sciences, University of Ljubljana) |
| Effectiveness of incentives in web surveys within mixed-mode systems: An evaluation of errors & costs |
| |
| Incentives are often used to address the problem of declining survey response rates. However, while they can substantially increase response rates, they can also significantly heighten research costs. Furthermore, their impact on survey responses often remains uncontrolled. This is especially unclear in mixed-mode surveys of the general population where web surveys are increasing included as a cheaper component. In this talk we discuss the problem of using incentives in web surveys within mixed-mode designs for surveying the general population. The experiment was conducted on a sample of the general Slovene population. Participants were randomly assigned into one of experimental groups, manipulating two different combinations of modes (web/mail and web/telephone) and three different types of incentives. We first present the impact of incentives on response rates and compare errors using the Mean Square Error approach. Then, a detailed analysis of bias in response patterns is performed. Finally, cost aspects are taken into account and evaluated by a simulation of different initial sample sizes and their effect on the costs-errors optimization. The findings suggest that the use of incentives in mixed-mode surveys should be methodologically well elaborated from the aspect of response rates, survey errors and survey costs. |
|
|
| Ulf-Dietrich Reips (Social and Economic Psychology, University of Zurich) |
| Investigating causal relationships with power: Online experiments |
| |
| An up-to-date overview of basics, techniques, methods, tricks, and tools for Internet-based experimentation. The talk summarizes principles of (online and offline) experimentation and ways of dealing with issues in experimental design, security, recruitment, sampling, self-selection, multiple submissions, reactance-free question design, response time measurement, dropout, error estimation, distributed experimentation, data handling, data quality, and log file analysis. Among other methods, the presenter will explain the warm-up technique, seriousness check, sub-sampling procedures, multiple site entry, ways to check for motivational confounding and when and why the high hurdle technique may work or not. Free Web services are introduced, like vasgenerator.net for creating visual analogue scales, the "::web experiment list::" for recruitment of participants, and Scientific LogAnalyzer for data analysis. Empirical results from investigations into the validity of several of the techniques for Internet-based experimenting will be discussed. |
|
|
| Patrick J.F. Groenen and Martijn Kagie (Erasmus School of Economics, Erasmus University Rotterdam) |
| A usability of a new graphical shopping interface through GlobalPark |
| |
| To facilitate the search process of users on an electronic commerce website we developed a new Graphical Shopping Interface (GSI). This interface presents groups of recommended products in a map based on product characteristics. In this map, products that are similar to each other are located close to each other, while dissimilar products are located far apart. The recommendation approach used in the GSI is works as follows. First, eight products are randomly selected and shown in a map to the user. When the user chooses the product she likes most, we recommend seven new products more similar to this product and show these in a new map together with the chosen product. This process repeats itself leading to evermore similar products. We evaluated the GSI in a usability study created in GlobalPark. In this study, the GSI Java applet was integrated into the survey environment, so that no switching between the GSI and the survey was necessary. The applet accessed a database outside Global Park to store information gathered in the GSI. In total, we gave the respondents four tasks, which they had to perform either with the new GSI or a traditional interface. To avoid order effects, both order and assignment of tasks to an interface was randomized. In this presentation, we discuss the setup of the present usability study and show how the implementation was done in the Global Park environment. |
