CP 6691 - Week 8

Multiple Baseline Designs
Threats to Internal and External Validity of Multiple-Baseline Experiments
Evaluating Sample Study #17 (Analysis of Effects of a Reading ...)

Assignment for Week 9

Suppose a concerned parent wants you to help decrease her child's TV-watching behavior in addition to the child's bed wetting behavior. You could, of course, tackle the TV-watching behavior after the bed wetting behavior had been corrected. But, that might take 30 weeks or more. The A-B design you learned about in the Week 7 lesson can be easily adapted to measuring multiple behaviors simultaneously, as long as the behaviors are mutually exclusive, or independent of each other. This is important so that the treatment for each behavior has no effect on the other behavior(s), otherwise, we have confounding effects.

In our example, Since bed wetting occurs when the child is asleep, and TV watching is, obviously, done while the child is awake, these behaviors are independent of each other. You decide to track both behaviors simultaneously, and you come up with a treatment to reduce bed wetting (T_BW) and a different treatment to reduce TV watching (T_TV).

You would follow the same process as described in the Week 6 lesson for the A-B design structure, except that instead of going back to the A baseline after each treatment, you would alternate treatments. Since the treatments are independent, each treatment acts as a baseline for the other behavior. The diagram below illustrates this. Note that each behavior is color coded to help you differentiate between them. Also notice that when the bed wetting treatment is in effect, TV watching is at baseline and vice versa.

Internal Threats:
 

• History: A possible threat depending on whether you can identify an event external to the study that is beyond the researcher's control that can affect the dependent variable, and if the study is long enough for history to have an effect.
   

• Maturation: A possible threat depending on whether you can argue that the dependent variable can be affected by maturational effects, and depending on if the study is long enough for maturation to have an effect.
   

• Testing: If pencil-and-paper tests are being used as the measures, and if they are all the same test, and if the interval between tests is relatively short, then it is possible for subjects' performance on the tests to improve due to remembering (the testing effect).
   

• Instrumentation: Instrumentation takes on an additional character in multiple-baseline studies, because, often, repeated measures are taken. If the measures used are tests, be alert to the possibility that subjects could become test-wise (can guess the answers based on unintentional clues in the question stems or in other questions on the test). If the measures are observations, be sensitive to the fact that inter-observer agreement should be consistent across all observations, especially if different observers are used at different times.
   

• Statistical Regression: Multiple-baseline studies may deal with single subjects or with small groups of subjects. If the study involves a single subject, statistical regression is not, usually, a problem because single scores do not regress reliably. If multiple subjects are used in the study, then their scores could be averaged, and the resulting group mean score on the pretest could (if it is excessively low or high) be subject to regression effects on the posttest.
   

• Differential selection: Not an issue (since there is no control group, subjects are not assigned -- they all receive the treatments). Actually, each subject is his/her own control during the "B"(or baseline) portion of the A-B design. Nevertheless, since all subjects receive all the treatments, differential selection is not generally a threat in multiple-baseline designs.
   

• Selection-maturation interaction: Not an issue because differential selection is not generally an issue.
   

• Experimental mortality: Always a potential problem. It is expecially troublesome since the number of subjects in multiple-baseline designs is usually small to begin with.

Remember, that there is no control group or random assignment process to cancel out any of the internal threats.

 

External Threats:
 

• Population Validity: Threats to population validity are not an issue in multiple-baseline designs because subjects are not selected from a general population with the idea of generalizing results. We don't need to address the issue of random or nonrandom selection. Consequently, you must be very careful to whom you attempt to generalize results. Generalizations must be make by considering the subjects' demographic information provided by the researcher.
   

• Personological Variable Validity: A potential threat just as in other experimental studies.
   

• Ecological Validity: A potential threat just as in other experimental studies.

Now try your hand at evaluating a multiple baseline study. It's number 21 in your Supplementary Book (SB). After you evaluate the study, compare your answers with mine below.

1. What kind of research design is this?

This is an experimental design because the researcher is manipulating the independent, treatment variable (instruction in outlining). There are only 8 subjects in this study, and there is no control group formed. All subjects receive the treatment. This is a multiple baseline design with the three different types of outlining constituting the different behaviors being tracked. Fortunately, the research article confirms this is the design of the study (see page 284, last paragraph on the page, entitled Experimental design).
 

2. What is the research hypothesis, objective, or question(s), or if none, so state.

There is no research hypothesis, objective, or question. The purpose statement on page 283 (just above the Method Section) does not meet the criteria for being a good research objective.
 

3. To what population would you feel comfortable generalizing results of this study?

The first paragraph in the Methods Section on page 283 describes the subject in some detail. It would have to be a personal decision as to whether the information provided was enough to give you confidence to generalize results. It might be possible to generalize results to similar severely learning disabled subjects.
 

4. Identify the threats to internal and external validity in this study.
 

Internal Threats:
 

• History: Study only lasted 10 days. Not long enough for history to have a significant impact on the dependent variable.
   

• Maturation: Study only lasted 10 days. Not long enough for maturation to have a significant impact on the dependent variable.
   

• Testing: Not a threat since each test involved outlining a different section from history and science texts. The follow-up tests involved sections of history text #1 not previously used in the study.
   

• Instrumentation: You might at first think that there would be a difference in difficulty outlining a history text and a science text. But, this is not necessarily true. Since we don't have the texts to look at, we should not assume they were different levels of difficulty. However, it would have been better if the researchers told us more about the texts used to perform the outlining.
   

• Statistical Regression: Since a small group of subject was used in this study, we should consider this threat. The initial scores of all subjects was zero on all behaviors measured (see Figure 1 in the article, page 285). It is theoretically possible for subjects' scores to regress toward the mean (improve) on subsequent testing. However, looking at the steady improvement over time, we can probably rule out regression effects since they would be felt mostly on the second testing and to a lesser extent on the third, fourth, and subsequent tests.
   

• Differential Selection: Not a threat in this type of design.
   

• Selection-Maturation Interaction: Not a threat because differential selection is not a threat.
   

• Experimental Mortality: There is no evidence of mortality in this study.

External Threats:
 

• Population Validity: Subjects were not randomly selected, and sample size is very small; so population validity is low.
   

• Personological Variable Validity: Not much demographic data provided on subjects -- only IQ and gender data presented.
   

• Ecological Validity: Study was conducted in a special classroom setup (a Visual Response System classroom). Consequently, if the results of this study could be generalized, I would expect they would generalize best to situations involving the use of similarly outfitted classrooms.
   

5. Are there any ethical problems in this study?

Since it does not appear that the treatment is potentially harming the subjects of the study in any way, we could conclude that there are no potential ethical problems in this study.

If you have any questions concerning this evaluation (if you found things I didn't discuss here, or if you don't understand something I've discussed here), talk with other members of the class to see if you can resolve the issues with them. If not, discuss your questions with the instructor in class.

Due Next Week

Prepare for Evaluation Quiz 4 Submit Research Proposal