CP 6691 - Week 6 (Part 2)

Practice Evaluations

Interactive Table of Contents (Click on any Block)

Practice Evaluation Mini-Study 1
Practice Evaluation Mini-Study 2

Assignment for Week 7
Part 2 of 2

Now I want you to practice evaluating studies that are presented in the form of abstracts, rather than the full-length articles you have been practicing with over the past several weeks. You will read a short description of a research study and evaluate it. You'll be surprised how much you can evaluate from such a short description like an abstract. I'll start you off with the following example. After reading the study abstract, follow through my evaluation and see if you understand my analysis. Here we go:

Practice Evaluation Study #1

Two military doctors designed a research project to test the success of a new drug in the treatment of diarrhea. This study involved about 300 randomly selected patients out of about 3000 military members deployed overseas to an African nation. They asked patients (U.S. military members) who needed medication for their ailment to take one bottle of medicine from one of three different boxes. In one box, there was just bottles of sugar pills, which had no medicinal value. The bottles in the second box contained standard diarrhea medication normally issued by the military hospital for diarrhea. The bottles in the third box contained an experimental diarrhea medication.

Each patient was tracked over the next month to determine how long it took to stop the diarrhea. Of the 225 subjects remaining at the end of the month, the researchers found that, on average, those on the new drug recovered from diarrhea in one-third the time it took for patients on the standard medication. This difference turned out to be statistically significant.

1. What kind of research design is this?

This is a true experimental study because the researchers created three groups (a "placebo" group, a "standard medication" group and an "experimental medication" group. This eliminates descriptive and correlational designs. The researcher manipulated the independent variable (treatment) by determining who would get which bottle of medicine. This eliminates causal-comparative designs. Finally, they allowed subjects to randomly assign themselves to one of the treatments, depending on which box of bottles they choose from. This eliminates quasi-experimental designs.
 

2. To what population (if any) can the findings of this study be generalized?

Because the sample was relatively large and randomly selected, the findings could be generalized to all US military members because eating and drinking habits (which are main contributors to diarrhea) are relatively the same for the majority of Americans, particularly military members, making them especially susceptible to illness from the parasitic organisms present in this African country.
 

3. Discuss the threats to internal validity and justify whether they are or are not threats.

  • History: Even though this was a relatively short study, there are external "event" that could affect the dependent variable. Since diarrhea is affected by what patients eat and drink (and the parasites in them), it is possible that patients could continue to ingest these parasites by eating and drinking the wrong things while on the medication, thus extending the healing time. Having said this, however, because of random assignment, it is just as likely for this to happen with patients in each of the treatment groups. Thus, this historic threat would be canceled out thanks to random assignment. Bottom line: no threat from history.
     

  • Maturation: No threat in this study because the study was too short and because of random assignment.
     

  • Testing: This was essentially a post test only control group design. No premeasure was taken on the patients. Only a post test (the time it took to recover from the illness) was recorded. So, testing is not a threat in this study.
     

  • Instrumentation: Possible threat because we aren't told how the subjects were tracked regarding their health.  Was a valid and reliable tracking instrument used?  It's unclear from the scenario.  Therefore, it's a possible threat.
     

  • Statistical Regression: No threat because there is no pretest.
     

  • Differential Selection No threat because subjects are randomly assigned.
     

  • Selection-Maturation Interaction: Since neither maturation nor differential selection are threats, it is impossible for there to be an interaction effect.
     

  • Mortality: There is a slight mortality threat in the study: 75 subjects dropped out of the study. It would have been valuable to know how many subjects dropped out of each group (we can't just assume that they were equal).
     
  • Appropriate Use of Inferential Statistics:  Researcher tracked recovery time for all groups and then analyzed the difference in recovery time between the groups.  Since recovery time (in days, or weeks, or hours) is considered continuous data, the most appropriate inferential statistic to use would be a parametric test.
     

4. Are there any ethical problems in this study?

It is probably unethical to deny the standard treatment to one third of the sample (those who chose the placebo treatment). This would lead to undue suffering by the patients who are looking to the medical staff in good faith for relief. In reality, such a situation would never be permitted without the patient's permission. You may wonder how such permission would be secured. Here's one way to do it: Ask the patients if they want to be part of a study to test an experimental diarrhea drug and clearly inform them that there is a possibility that they may not obtain relief from the medication they will be provided. If they agree with those terms, then it is ethical to keep the placebo as a treatment option. Of course, informing the subjects of the study in this way may inject a possible psychological factor into the study that could confound the results. So, the researcher must weigh the consequences of each before deciding which way to go.

If you have any questions concerning this evaluation (if you found things I didn't discuss here, or if you don't understand something I've discussed here), talk with other members of the class to see if you can resolve the issues with them. If not, discuss your questions with the instructor in class.

Practice Evaluation Study #2

A program to teach sexual abstinence to adolescents was tested in a junior high school setting on seventh grade students. The test program was administered at a public junior high school. The comparison group for this study was a group of seventh grade students in a parochial junior high school in the same city. The parochial school used a different program to teach sexual abstinence. The program lasted for one school year. All students in both schools were given the same pretest to determine their initial attitudes on sexual abstinence. Analysis of pretest results revealed that there was no statistically significant difference between the attitudes of the two groups at the beginning of the study. At the end of the school year, the same attitude measure (post test) was administered to both groups. A difference in mean attitude scores of 7.6 percentage points was found between the two groups, with a "p" value of .08. Since the p value was not less than the .05 alpha level set for the study, it was determined by the researcher that the test program would not produce statistically significant changes in adolescents’ attitudes regarding sexual abstinence.

1. What kind of research design is this?

This is a quasi-experimental study because the researcher created two groups; manipulated the independent variable (treatment); but did not randomly assign subjects to groups -- the experimental group came from one school and the control group came from another (the parochial school).
 

2. To what population (if any) can the findings of this study be generalized?

Since only two seventh grade classes were used in two different schools (one public, one parochial) it's unlikely that the results would easily generalize. We would have to know a lot more about the subjects of the study (demographically) than was provided in this abstract.
 

3. Discuss the threats to internal validity and justify whether they are or are not threats.

  • History: This was a long study (one school year). Historically, it is reasonable to assume these children will watch television and see movies, both of which contain frequent public service advertisements on sexual abstinence, the effects of unsafe sex, and cautions about date-rape. These messages are intended to affect children's attitudes toward sex. Thus, this historic event could drastically impact the dependent variable being studied. Also, in parochial schools, there are additional pressures by teachers and in the curriculum about the dangers of sexual contact and the need to avoid it both for religious and health reasons. This additional pressure would not be matched in the public school. Thus, there is the real possibility that this historic event is unbalanced between the two schools due, in large part, to the lack of random assignment. Bottom line: history is a definite threat in this study.
     

  • Maturation: Although these children will mature physically, mentally, and emotionally during the year, these are the pubescent years, which will exacerbate the sexual abstinence problem among these adolescents. So, rather than alleviating the situation, maturation may actually worsen it. Even though there was no random assignment in this study, it is reasonable to assume that seventh graders in the public schools are maturing at roughly the same rate as those in parochial schools. So, one could argue that the maturational effects described above would be equal across both groups and, therefore, cancel each other out. By this reasoning, maturation would not be a threat in this study.
     

  • Testing: Testing is not threatened in this study because attitude measures don't suffer from the memorization effect. People generally respond with their actual attitudes, so if they had changed from the first to the second measure, that change would have registered on the measure.
     

  • Instrumentation: Possible threat because one could question whether the measure was valid and reliable. We don't know if it was a standardized instrument, so the question is a legitimate one. If you picked up on this and said that instrumentation might be threatened because of it, give yourself credit.
     

  • Statistical Regression: No threat because the study was not dealing with extreme groups.
     

  • Differential Selection Definitely a threat because subjects came from two different schools whose views on sexual abstinence might be taught in different levels (one would expect a parochial school to considerably more strict on issues of morality than would a secular, public school).
     

  • Selection-Maturation Interaction: Since we argued that maturation was not a threat in this study, then the interaction cannot be a threat, because there is nothing for differential selection to interact with..
     

  • Mortality: No threat in this study because there were no dropouts mentioned.
     
  • Appropriate Use of Inferential Statistics:  Since academic test scores are being analyzed, that makes the data continuous.  Therefore, the most appropriate inferential statistic to use would be parametric.
     

4. Are there any ethical problems in this study?

Since nothing was done to harm students who were given the experimental abstinence program, and since the control group was not denied anything (they still received their normal abstinence program), there do not appear to be any ethical problems in this study.
 

5. How could the threats you identified above have adversely affected the results of the study?

History and differential selection are the two outstanding threats in this study. Actually, since differential selection actually caused the history threat, we need only deal with the history threat. We can easily argue that since the parochial school probably taught a more consistent, stronger set of moral values to its students, they had the benefit of this additional training on top of their "normal" sexual abstinence education. The combination of these factors may have produced a larger than expected improvement in student attitudes about sexual abstinence. This would have made the difference between the experimental and control groups less and, perhaps, even nonsignificant. Thus, the fact that this study found no statistically significant difference between the two groups' attitudes does not necessarily mean that the experimental abstinence program would not produce statistically significant changes in adolescents’ attitudes regarding sexual abstinence. In other words, if the study were conducted in such a way that both experimental and control groups were in the same school (either school actually), then the researcher would get a fairer test of the effectiveness of the experimental program.

If you have any questions concerning this evaluation (if you found things I didn't discuss here, or if you don't understand something I've discussed here), talk with other members of the class to see if you can resolve the issues with them. If not, discuss your questions with the instructor in classl.

Got the hang of it? It's really not any harder than evaluating a full-length article except that you have less information to work with. So, your justifications for supporting or refuting threats needs to be based on what's logical and reasonable to assume.

End of Week 6 Lesson
Assignment For Next Week
Gall: Chapter 13 ("A-B" Designs)
SB: Study 17
Guide:  Chapter 4
Extra Evaluation Practice: Try your hand at evaluating these scenario-based research studies (I call them mini-studies.) For each problem, read the scenario (first page) and try to evaluate it using the 5 evaluation questions in the Typical Evaluation Quiz format we've been using above. Don't look at the answers on the second page until you have answered all the questions yourself. Then compare your answers with those provided in the problem. If you have questions or don't agree with or understand the answers provided, E-mail me and let's discuss it.
 

Scenario Pack 3 (MS Word docs): Scenario A, Scenario B, Scenario C,  Scenario D, Scenario E, Scenario F, Scenario G, Scenario H, Scenario I.
Due Next Week
Prepare for Evaluation Quiz 3