Computer-Managed Data Collection and Quality

A major concern in data collection is “quality.” The quality of data is partly a function of using the right instrument and using the right instrument is partly a function of fingertip availability of a large and diverse inventory of instruments from which to draw. ProtoGenie serves this important requirement of good research through a wide and expanding toolkit of data collection instruments. Also, the large numbers of shared protocols in the ProtoGenie public library enable users to see how concepts were measured in other studies and to borrow and modify them in much neglected replications of studies.

Quality also is a function of consistency. The manual recording of measurements and subsequent transcriptions is notorious for both random and systematic unreliability and error. ProtoGenie removes this source of error through the computer management of data collection. The quality of data is not completely determined by the suitability of measurement instruments and computer-management. In many studies, conformity is a serious concern, as in clinical trials that require the voluntary self-administration of treatments and measurements. To remedy this problem, a ProtoGenie computer-supported protocol can monitor the actions of subjects and prompt them when they fall.

An important dimension of data quality is measurement reliability, which means that repeated measurements using the same instruments (like a questionnaire) will produce the same results (data). This is sometimes called measurement consistency. There are three ways that data can be internally inconsistent. The first way is for an instrument itself to change from one presentation to another. For example, manually written instruction to interviewers may become corrupted during the research process. The electronic presentation of measurements by ProtoGenie remedies this problem. The second way that data can be of poor quality due to internal inconsistency is for research staff to present measurements differently from measurement to measurement and subject to subject. The automatic presentation of measurements by ProtoGenie remedies this problem. The third way that data can be internally inconsistent involves the transcription of measurement results into data files. Errors in transcription are both random and systematic. Random errors are as likely to be in one direction as another. Systematic errors tend to be "biased" in one direction. ProtoGenie's automatic transcription of results eliminates this problem.

Another major source of poor data quality is missing or incomplete data. One source of missing data is the failure of data collectors to get the information called for by the instrument. This can be due to poor training or the lack of supervision. ProtoGenie provides instant data storage as sessions are conducted and can also provide summary data on the collection process itself. Therefore, regular feedback and automatic analysis of responses can be built into a ProtoGenie protocol to tell each data collection person how he or she is doing on getting the targeted data. This kind of feedback is often enough to correct the problem of missing data of this kind. When it is not, the same collection process data can be used by supervisors to detect potential problems with individual data collectors, sometimes calling for corrective actions, such as additional training, greater supervision, and change of data collection personnel.

Another major cause of bad data can emerge when subjects self-administer their own treatments and measurements and do not comply with prescribed regimens. This can be a matter of great concern about the quality of data in clinical trials or studies that use similar data collection processes. For example, a subject may be expected to take a certain medication of a certain dosage every day at the same time and then fill in an entry to a questionnaire about other daily activities and symptoms and then make an entry into a daily or periodic log. In many cases, the subject forgets or simply neglects to follow the prescribed regimen. ProtoGenie can go a long way to solving this problem by enabling the principle investigator to write into a ProtoGenie protocol the detection of non-compliance and notification to that effect the next time that the subject opens his/her log. Often, this is enough to improve the quality of data in these studies. However, non-compliance can also be detected by caseworkers or study supervisors at any computer that has Internet access. The caseworkers or study supervisors can then speak with the subjects who are having difficulty following the prescribed regimen and resolve the problem or remove the subject from the study, giving the reasons why..

Dr. Kuang-Mon Tuan, School of Optometry, The University of California at Berkeley