International Portal of Teacher Education

Source: Journal of Research on Technology in Education Vol. 43, Iss. 1. p. 29-52, (Fall 2010).
(Reviewed by the Portal Team)

The current article explores some common methodological issues facing educational technology research. This paper also highlight new data collection approaches using examples from the literature and the authors' own experience.
Given that surveys and questionnaires remain widespread and dominant tools across nearly all studies of educational technology, the authors first discuss the background and limitations of how researchers have traditionally used surveys to define and measure technology use (as well as other variables and outcomes).
The authors argue that the overall lack of methodological precision and validity is of particular concern. The authors claim that the current research tools used to study programs such 1:1 laptop initiatives often provide inadequate information about the extent to which technology is used across the curriculum and how these uses may affect student learning.

Although this article outlines a number of common methodological weaknesses in educational technology research, the current lack of high-quality research is undoubtedly a reflection of the general lack of support provided for researching and evaluating technology in schools.
Producing high-quality research is an expensive and time-consuming undertaking that is often beyond the resources of most schools and individual school districts. At the state and federal level, vast amounts of funds are expended annually on educational technology and related professional development, yet few funds are earmarked to research the effects of these massive investments.

The authors recognize that collecting research in educational settings will always involve compromises and limitations imparted by scarce resources. Hence, the authors suggest that extensive opportunities currently exist to improve data collection and analysis within the structure of existing research designs. Just as technology has transformed the efficiency of commerce and communication, we feel that technology can provide many opportunities to advance the art and science of educational research and measurement.

In their own work, the authors have devised new solutions to overcome the obstacles encountered while conducting research in schools by capitalizing on those technologies increasingly available in schools.
In this article, the authors have specifically shared some of the techniques and approaches that they have developed over the course of numerous studies in a wide variety of educational settings. For instance, the authors have found the visual analog scale to be an improvement over our past efforts to quantify the frequency of technology use via survey.
Similarly, the authors have shared other examples of our struggles and successes in measuring the impact of educational technology practices on student achievement. The examples from the literature and their own examples both serve to underscore how quickly things can change when examining technology in education.
Next, the authors highlight other challenges and opportunities inherent in the study of educational technology, including the potential for computer adaptive surveying. In addition, the authors discuss the critical importance of aligning outcome measures with the technological innovation, concerns with computer-based versus paper-based measures of achievement, and the need to consider the hierarchical structure of educational data in the analysis of data for evaluating the impact of technology interventions.

In conclusion, the authors hope that the issues this article raises and the specific examples it includes spur critical reflection on some of the details important to data collection and educational technology research. In addition, the authors hope that their own examples reported here also serve to encourage others to proactively develop and share what will be the next generation of research tools.