xml version 1.0 encoding UTF8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchemainstance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 22 Ka 4500
controlfield tag 007 crbnuuuuuu
008 s2010 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14SFE0003398
035
(OCoLC)
040
FHM
c FHM
049
FHMM
090
XX9999 (Online)
1 100
Guo, Jing.
0 245
Extending the principal stratification method to multilevel randomized trials
h [electronic resource] /
by Jing Guo.
260
[Tampa, Fla] :
b University of South Florida,
2010.
500
Title from PDF of title page.
Document formatted into pages; contains X pages.
Includes vita.
502
Dissertation (Ph.D.)University of South Florida, 2010.
504
Includes bibliographical references.
516
Text (Electronic dissertation) in PDF format.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
3 520
ABSTRACT: The Principal Stratification method estimates a causal intervention effect by taking account of subjects' differences in participation, adherence or compliance. The current Principal Stratification method has been mostly used in randomized intervention trials with randomization at a single (individual) level with subjects who were randomly assigned to either intervention or control condition. However, randomized intervention trials have been conducted at group level instead of individual level in many scientific fields. This is so called "twolevel randomization", where randomization is conducted at a group (second) level, above an individual level but outcome is often observed at individual level within each group. The incorrect inferences may result from the causal modeling if one only considers the compliance from individual level, but ignores it or be determine it from group level for a twolevel randomized trial. The Principal Stratification method thus needs to be further developed to address this issue. To extend application of the Principal Stratification method, this research developed a new methodology for causal inferences in twolevel intervention trials which principal stratification can be formed by both group level and individual level compliance. Built on the original Principal Stratification method, the new method incorporates a range of alternative methods to assess causal effects on a population when data on exposure at the group level are incomplete or limited, and are data at individual level. We use the Gatekeeper Training Trial, as a motivating example as well as for illustration. This study is focused on how to examine the intervention causal effect for schools that varied by level of adoption of the intervention program (Earlyadopter vs. Lateradopter). In our case, the traditional Exclusion Restriction Assumption for Principal Stratification method is no longer hold. The results show that the intervention had a stronger impact on LaterAdopter group than EarlyAdopter group for all participated schools. These impacts were larger for later trained schools than earlier trained schools. The study also shows that the intervention has a different impact on middle and high schools.
590
Advisor: Hendricks Brown, Ph.D.
653
Causal Effect
CACE
MultiLevel Randomized Trials
Noncompliance
Rubin Causal Model
690
Dissertations, Academic
z USF
x Epidemiology & Biostatistics
Doctoral.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.3398
PAGE 1
Extending the Principal Stratification Method To MultiLevel Randomized Trials by Jing Guo A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Epidemiology and Biostatistics College of Public Health University of South Florida CoMajor Professor: Hendricks Brown, Ph.D. CoMajor Professor: Yiliang Zhu, Ph.D. Mary E. Evans, Ph.D. Wei Wang, Ph.D. Date of Approval: April 12, 2010 Keywords: Causal Effect, CACE, MultiL evel Randomized Trials, Noncompliance, Rubin Causal Model Copyright 2010, Jing Guo
PAGE 2
Dedication This work is dedicated to my husband and my son, Zhaohui and Avery Wang, who have been giving me their selfless support for my study and research all these years; to my parents for their unfailing support.
PAGE 3
Acknowledgments First of all, I would like to express sin cere appreciation to my major advisor, Dr. C. Hendricks Brown, who has devoted hi mself on mentoring and nourishing me throughout my study and researc h. I could not have gone so far without his guidance and support. What I have learned from him and his research enables me to overcome many hurdles and accomplish what I have dreamed of. I believe his profound positive impact will continue to benefit my future professional work. I would also like to thank Dr. Yiliang Zhu, my comajor advisor, whose unselfish contribution and diligent work has leaded me to the finish line. Dr. Zhu has been a mentor and a friend to me, and most of all, a great scientist. Special thanks are given to Drs. Mary E. Evans, Wei Wang who have served as my committee members, and provided cri tical inputs to my study along the way. Recognition is given to Dr. Peter A Wyman, who is coPI of this NIMH supported project. He partially supported my study by allowing me to access the preventive trial data. Finally, I would like to thank all faculties and staff members at the Department of Epidemiology and Biostatistics, College of Publ ic Health, University of South Florida for all assistances they have provided. I am very proud of being part of this community.
PAGE 4
i Table of Contents List of Tables..................................................................................................................... iv List of Figures.................................................................................................................... vi Abstract....................................................................................................................... ...... vii Chapter One Introduction................................................................................................1 1.1 Causal Effects in Intervention Trials.................................................................. 1 1.2. Motivating Examples......................................................................................... 3 1.2.1 Example One: A Case of Two Principal Strata....................................... 3 1.2.2 Example Two: A Case of Four Principal Strata....................................... 7 1.3 Proposed New Methodology.............................................................................10 Chapter Two Methodology for Tw oLevel Principal S tratification.............................. 13 2.1 Principal Stratification Me thod in Single Level Trials ..................................... 14 2.2 Blooms Model for a SingleLevel Randomized Trial with Active Intervention versus Control...............................................................................20 2.3 Principal Stratification in TwoLevel Randomized Trials ................................ 24 2.4 PS Models for TwoLevel Randomized Trials with Active Intervention versus Control ...................................................................................................29 2.4.1. AIR Model............................................................................................33 2.4.2. GBlooms Model................................................................................. 38 2.4.3. GBPB Model/GAPA Model..............................................................43 2.5 Incorporating Individual Characteristics through Regression Models ............. 52
PAGE 5
ii 2.6 Mixtures and Marginal Maximum Likelihood Approach for TwoLevel Random ized Trials............................................................................................ 55 2.6.1 TwoLevel Randomized Trials with Active Intervention versus Control ..................................................................................................55 2.6.2 TwoLevel Randomized Trial w ith Random Ti me of Crossover from Control to Active Intervention..................................................... 59 Chapter Three Using the TwoLevel Prin ciple Stratification Me thod to Evaluate the QPR Gatekeeper T raining Program................................................62 3.1 Introduction....................................................................................................... 63 3.2 Method..............................................................................................................65 3.2.1 Study Design and Participant Population.............................................. 65 3.2.1.1 Introduction of the QPR Gatekeeper Training..........................65 3.2.1.2 Study Design.............................................................................67 3.2.1.3 Participant Population............................................................... 72 3.2.1.4 Measures...................................................................................75 3.2.2 Methodology..........................................................................................76 3.2.2.1 Method......................................................................................80 3.2.2.2 Hypotheses................................................................................85 3.2.2.3 Model Selection Strategy.......................................................... 86 3.3 Analysis.............................................................................................................89 3.3.1 Analyses Limited to the First Study Period........................................... 89 3.3.2 Entire Study Period................................................................................ 92 3.3.2.1 Summary of Analyses for All 32 Schools.................................93 3.3.2.2 Summary of Analyses for 20 Middle Schools.......................... 97 3.3.2.3 Summary of Analyses for 12 High Schools............................ 101 3.4 Conclusion......................................................................................................104 Chapter Four Conclusion.............................................................................................105
PAGE 6
iii 4.1 Methodological Contributions........................................................................105 4.2 Limitations...................................................................................................... 107 4.3 Application of Findings..................................................................................107 4.4 Further Discussion.......................................................................................... 108 References.......................................................................................................................113 Appendices......................................................................................................................117 Appendix A: Compute a Chisquare Difference Test ........................................... 118 Appendix B: Mplus Code ITT Model 01: Main effects within first training period .................................................................................119 Appendix C: Mplus Code ITT Model 02: Main effects with equal slops within first training period ..............................................................122 Appendix D: Mplus Code ITT Model 09: M ain effects with interactions within first training period..............................................................125 Appendix E: Mplus Code ITT Mode l 13: M ain effects within first training period with weaken condition...........................................128 Appendix F: Mplus Code AS Model 28: M ain effects within first training period with weaken conditi on under Astreat status....................... 131 Appendix G: Mplus Code ITT M odel 88: m edian time 4period...................... 134 Appendix H: Mplus Code ITT Mode l 132: A S main effects 4period ms........ 137 Appendix I: Mplus Code ITT Mode l 176: AS m ain effects 4period hs........... 140 About the Author...................................................................................................End Page
PAGE 7
iv List of Tables Table 1: Principal Strata of the Flu Shot study Presented in 2 by 2 Table ......................... 9 Table 2: Principal Stratification Defined on the B asis of Potential Mediator Outcome in Singlelevel Randomized Trials...................................................... 16 Table 3: Potential Values of Medi ator Outcom e for Blooms Model.............................. 21 Table 4: Principal Stratification based on Potential Mediator Outcom e in Twolevel Randomized Trials..................................................................................... 27 Table 5: Summary of Models for Twole vel Random ized Trials with Group and Individual Level Participation............................................................................. 31 Table 6: Principal Stratification of Flu Shot Study ...........................................................37 Table 7: GBlooms Mode : GA Gatekeeper Study...........................................................41 Table 8a: Baltimore Good Behavior Game : GBPB Model for Intervention of First Graders in Baltimore (with restriction on individual level)..................... 49 Table 8b: Baltimore Good Behavior Ga m e: GBPB Model for Intervention of First Graders in Baltimor e (without restriction)............................................... 51 Table 10: Study Design of the QPR Gatekeeper Training P rogram................................. 69 Table 11: Ethnic Distribution of Cobb County School Students ......................................72 Table 12: Demographic Distribution of Students from the 32 Study Schools................. 73 Table 13: Ethnicity Distributi on of Cobb County School Staffs ...................................... 74 Table 14: Demographic Distribution of School Staffs from the 32 Study Schools .......... 75
PAGE 8
v Table 15. Model Selection Strategy................................................................................. 88 Table 16: Results of the Poisson Re gression Model under W eak Exclusion Restriction Assumption during the First Period of Study.................................91 Table 17: Estimate Comparison for All 32 Schools over Four Periods ............................ 95 Table 18: Effect Estimates for all 32 Schools over Four Periods ..................................... 96 Table 19: Estimate Comparison for 20 Middle Schools over F our periods..................... 99 Table 20: Effect Estimates for 20 Middle Schools over Four Periods ........................... 100 Table 21: Estimate Comparison for 12 High Schools over Four Periods ....................... 102 Table 22: Effect Estimates for 12 High Schools over Four P eriods............................... 103 Table 23. Comparison of Alternative Methods............................................................... 112
PAGE 9
vi List of Figures Figure 1. Consort Diagram for JOB II Study......................................................................6 Figure 2: Consort Diagram for the Study Design of the QPR Ga tekeeper Training Program .............................................................................................................70 Figure 3. SameDay Assessment Rate s of 6 Block Schools across Tim e........................ 79 Figure 4. A Simplified Schematic Drawing of the Model Analysis for the Georgia Gatekeeper Project ............................................................................................ 82
PAGE 10
vii Extending the Principal Stratification Me thod for MultiLevel Randomized Trials Jing Guo Abstract The Principal Stratification m ethod estim ates a causal intervention effect by taking account of subjects differences in participation, adherence or compliance. The current Principal Stratification method has b een mostly used in randomized intervention trials with randomization at a single (individual) level with subjects who were randomly assigned to either interventi on or control condition. However, randomized intervention trials have been conducted at group level inst ead of individual leve l in many scientific fields. This is so called t wolevel randomization, where randomization is conducted at a group (second) level, above an individual level but outcome is often observed at individual level with in each group. The incorrect infere nces may result from the causal modeling if one only considers the compliance fro m individual level, but ignores it or be determine it from group level for a twole vel randomized tria l. The Principal Stratification method thus needs to be fu rther developed to a ddress this issue. To extend application of the Principa l Stratification met hod, this research developed a new methodology for ca usal inferences in twoleve l intervention trials which principal stratification can be formed by both group level and indivi dual level compliance. Built on the original Principal Stratification method, the new method incorporates a range of alternative methods to assess causal eff ects on a population when data on exposure at
PAGE 11
viii the group level are incomplete or limited, and are data at individual level. We use the Gatekeeper Training Trial, as a motivating ex ample as well as for illustration. This study is focused on how to examine the interventi on causal effect for sc hools that varied by level of adoption of the inte rvention program (Earlyadopter vs. Lateradopter). In our case, the traditional Exclusion Restriction Assumption for Principal Stratification method is no longer hold. The results show that th e intervention had a stronger impact on LaterAdopter group than EarlyAdopter group for a ll participated schools. These impacts were larger for later trained schools than earlier trained schools. The study also shows that the intervention has a different im pact on middle and high schools.
PAGE 12
1 Chapter One Introduction 1.1 Causal Effects in Intervention Trials In medical research, clinical trials are conducted to coll ect data on the safety and efficacy of new drugs or devices. Random assi gnm ent of patients is required in phase III clinical trials to produce in tervention and control conditions under which patients are similar in characteristics at baseline. In th eory, randomization is a powerful tool to yield statistically unbiased causal estim ates of the intervention effects when clinical outcomes are compared between the interv ention and control conditions. In practice, however, evalua tions could be subject to a number of problems even under randomization (Bloom, 1984). One of thes e problems is called participation bias (Brown, et al. 1999), a reference to the potenti al difference in outcomes due to a lack of full participation in intervention by subject s when assigned. In order to objectively evaluate the causal effect of intervention and exam how levels of participation influence the intervention effects, a key issue is to classify study subjects according to randomized assignment as well as level of participa tion, adherence, or compliance. Principal stratification (Frangakis, et al 2002) is such a method, which aims to address some of the limitations of the IntenttoTreat (ITT) me thod, a traditional appr oach in randomized clinical trials. IntenttoTreat method ev aluates the intervention effects based on intervention status originally assigned to subjects, and it i gnores subjects compliance to
PAGE 13
2 the study protocol. Therefore, intervention eff ects estimated under ITT could be biased in the presence of subjects noncompliance. In order to estimate intervention effect s while taking account for noncompliance, Bloom (1984) applied an instrumental variable (IV) approach in which the estimates of intervention effects are adjusted for subjects rate of compliance. More recently, the IV approach has been refined with a more ri gorous framework (Angrist, Imbens, & Rubin, 1996; Imbens & Angrist, 1994). The refined approach resulted in the estimation of complier average causal effect (CACE) whic h is a causal effect of intervention on the subjects who would comply with any assignment and is a likelihoodbased method. CACE has led to significant improvement over the IV approach. Imbens and Rubin (1997) demonstrated CACE estimation through the maximumlikelihood (ML) estimation method using the EM algorithm as well as a Bayesian data augmentation algorithm. However, a major difficulty involved in CA CE estimation is how to deal with missing data on compliance among study participants because a subjects participation or compliance is often unobservable. As th e issue of noncomplia nce receives more attention (Frangakis, el at 2002), more and more researchers have started applying the Principal Stratification method in randomized intervention trials to evaluate intervention effect for compliers individuals who w ould receive the intervention if offered. However, all previous analyses only focuse d on single level trials which subjects who were randomly assigned to eith er intervention or control condition. This study will extend Principal Stratification method on twolevel tria ls to discuss how to define the principal strata membership, how to extend assumptions from singlelevel on twolevel trials, and
PAGE 14
3 how to estimate appropriate cau sal effect. To illustrate the Pr incipal Stratification method, we give two examples in the next section. 1.2. Motivating Examples The exam ples given in this section util ized the Principal Stratification method on individual level to evaluate intervention effects. In the fi rst example the control group had a zero probability to receive the intervention and in the s econd example the control group had same probability to receive the interv ention as the interven tion group. Different numbers of strata were formed due to these different situations. 1.2.1 Example One: A Case of Two Principal Strata Conducted by the University of Michig an, the Job Search Intervention Study (JOBSII) is a random ized field experiment intended to prevent poor mental health and to promote highquality reemployment of unemployed workers (Vinokur et al 1995). Previous studies indicated a strong positive impact of job search intervention on highrisk workers. Those with high risk were the ta rgets of JOBSII. All study participants were randomly assigned into either an intervention condition or a control condition. The intervention condition consisted of five halfday training sessions. Participants who completed at least one tr aining session were categorized as compliers. Those who completed none were categorized as nevertakers. Based on the definition, the compliance rate was 55% in this study. The part icipants who were assi gned to the control condition did not have a chance to attend a ny session of the study. Therefore, their compliance status was not observable, the case of missing data As a result, the
PAGE 15
4 compliance rate in the control group was a ssumed to be the same as that in the intervention group on the basis that the two group s were similar under randomization. A total sample size of 486 was included for an alyses after removi ng individuals with missing records. One of the outcome variables was a baseline risk score, which could be computed based on risk variable s at baseline screening and al so at followups to predict depressive symptoms (depression, fina ncial strain, and assertiveness). To apply principal stratification to eval uate the intervention effects, we first identify two principal strata. Let iZ denote the intervention st atus assigned to subject i Control 0 on Interventi 1iZ and iS the participation status of subject i tsParticipanNon 0 tsParticipan 1iS Here Sis called a posttreatment variable, which is measured after treatment assignment and before the assessment of primar y outcome. It is used in conjunction with iZ to define two principal strata, compliers (or participants) a nd nevertaker (or nonparticipants). Figure 1 depicts the design where subjects were randomly assigned to either the intervention ( Z = 1) or control ( Z = 0) condition. The compliance rate represents the proportion of subjects in the intervention condition who are actu ally compliers (iS= 1 iZ = 1) and the rest in th e group are nevertakers (iS= 0  iZ = 1) In other words, compliers are those who participate if the intervention is offered, and nevertakers are those who would not participate when the interven tion is offered. The response variable ( Y ) can be
PAGE 16
5 measured for compliers (S= 1 Z =1) and nevertakers (S= 0 Z =1) in interventional condition as well as those rece iving the control condition ( Z = 0). In principle, our goal is to infer on the causal effects among Participants, and among NonParticipants. That is, to compare Y ( Z = 1; S = 1) with Y ( Z = 0; S = 1) and to compare Y ( Z = 1; S = 0) with Y ( Z = 0; S = 0). However, the difficulty is that the true compliance status S is unobservable in the control condition ( Z = 0) because subjects assigned to the control condition had no chance to participate. As a result, it is not possible to distinguish compliers ( Z = 0; S = 1) from nevertakers ( Z = 0; S = 0) within the control condition. The main interest of JOBSII was to estimat e the causal effects of intervention for compliers, i.e., the difference between Y ( Z = 1; S = 1) and Y ( Z = 0; S = 1). This is a case of single (individual) le vel intervention. Principal stratum membership is more complex with a twolevel study when individual participati on status is combined with group level participation. The next ex ample is a twolevel intervention study.
PAGE 17
6 Figure 1. Consort Diagram for JOB II Study Intervention Control Condition Condition Z =1 Z = 0 Participants NonParticipants Y ( Z = 0) S = 1 S = 0 Y ( Z = 1; S = 1) Y ( Z = 1; S = 0) Randomized
PAGE 18
71.2.2 Example Two: A Case of Four Principal Strata Many randomized experiments suffer from noncompliance (Imbens, el at., 2000). To address this problem some studies adopt an encouragement design that encourages subjects to change their behavior. Because the level of participa tion is determined by many selfselection factors, an encouragement design may mediate these factors, thereby encouraging a higher level of complia nce or participation. We then evaluate intervention effects while adju sting for the impact of noncompliance or participation level. The study of inoculation against influenza or flu shot described below is an example of encouragement design study (McDonald et al, 1992). In this study, physicians were randomly selected to receive a letter that encouraged them to inoculate their patients at risk for flu. The research in terest was to see if the encouragement letter (first level intervention) woul d lead to more patients recei ving the flu shot (second level intervention). The outcome is whether the intervention woul d reduce flurelated hospitalization. Let Z be an indicator of the physicians receiving the encouragement letter. If the physician of patient i received a letter, iZ = 1, otherwise iZ = 0. The posttreatment variable S is whether the patient received a flu shot. Let iS be the indicator for receiving a flu shot given assignmentiZ, then iS = 1 indicates the thipatient received a flu shot, otherwise iS = 0. Let ) (Z Si be an indicator for the receipt of flu shot given assignment Z If the physicia n of patient idid not receive an encouragement letter (iZ= 0) and the patient received a flu shot ) 0( Si=1, otherwise ) 0( Si= 0. Likewise, if the physician did receive an encouragement letter (iZ= 1) and the patient received a flu shot )1( Si=1, otherwise ) 1( Si= 0. The combination of these two assignments, receiving
PAGE 19
8 encouragement intervention ( Z ) and receiving flu shot (S), forms four compliance/behavior types or principal strata (C): As presented in Table 1, this is an exampl e of principal stratification with 4 strata. Membership in each stratum takes into consideration of all possi ble results of random assignment Z Note that the four strata cannot be distinguished through crosstabulation by Z and S, and is only partly observable. For example, among those patients who did not take a flu shot and whose physicians did not receive the encouragement letter ( Z = 0; S= 0), some would be never takers and othe rs would be compliers; among those patients who did not take a flu shot but th eir physicians received the letter ( Z =1; S= 0), some would be never takers and others would be defiers. Therefore, the observation ( Z = 0; S= 0) or ( Z =1, S= 0) alone cannot determine whether a patient was a nevertaker or a defier. Similarly, among those patients who t ook a flu shot but whose physician did not receive the letter ( Z = 0, S=1) some would be always takers and others defiers; and among those who took a flu shot but whos e physicians received the letter ( Z =1, S=1), some would be alwaystakers and others would be compliers. Thus again, observation ( Z = 0, S=1) or ( Z =1, S=1) is insufficient to determine always takers. For the same reason, observed data cannot be used to distingui sh compliers from always takers. In this study a set of population probabilities is assumed under which patients may fall into 1 0for1)(if taker)(always 1 0for1)( if (defier) 1 0for)(if (complier) 1 0for0)(if taker)(never )( Types /Behavior Compliance Z Z S a Z Z, Z S d Z Z, Z S c Z Z S n Ci i i i i
PAGE 20
9 one of these four principle strata. The probabilities can be dependent on patients level covariates such as demographics an d disease history (Little & Yau 1998). Table 1: Principal Strata of the Flu Shot study Presented in 2 by 2 Table Control ( Z = 0) No Flu Shot (S = 0) Flu Shot (S = 1) No Flu Shot (S = 0) Never Taker (n) Defier (d) Intervention ( Z =1) Flu Shot (S = 1) Complier (c) AlwaysTaker (a) The response variable ( Y ) is a binary outcome which indicated whether a patient subsequently experienced a flurelated hos pitalization during the study period. While the primary interest was to estimate intervention e ffects in each of the four principle strata defined by the compliance/behavior type, the fo ur principle strata cannot be identified solely based on observed patie nts flu shot status (S= 0 or 1) and their physician intervention status (receiving the encouragement letter) ( Z = 0 or 1). Thus the principal stratification method aimed to estimate the causal eff ect in the absence of compliance/behavior classi fication. It further involved making an unverifiable assumption, namely that there are no defiers, in order to identify all remaining parameters. The method allowed the researchers to go beyond a standard intenttotreat analysis and to adjust for the impact of compliance/beha vior in each stratum. The researchers found strong evidence that the encour agement letter had a benefici al effect on the compliers, which is similar to that of the alwaystaker (Imbens el at., 2000). Based on this finding,
PAGE 21
10 they concluded that the flu shot had little beneficial e ffects on reducing flurelated hospitalization. This second example is inherently a twolevel intervention but has only been analyzed to date in a one dimensional fram ework by assuming that all physicians were fully participants. In other words, the posttr eatment variable is at individual level, and principal stratification is thus also defined at this level th e stateoftheart of principal stratification. 1.3 Proposed New Methodology The preced ing examples illustrate the idea and rational of principal stratification (PS) method and how it overcomes the limitati ons of the method of intenttotreat in interpreting intervention effects. Previous applications of pr incipal stratification (PS) are focused on studies principal stratification is defined using a single level posttreatment variable, often at the individua l level. This dissertation extends the principal stratification method to the case where principal stratification may be defined by posttreatment variables at two levels, both group and individual level, and then develops causal inferences accordingly. Intervention trials with randomization at group level occur, yet compliance status can be determined at group level as well as at individual level. For example, we may assess an intervention that is delivered at the school level, A main challenge in such trials is that intervention exposure (participation le vel) can differ at the group level as well as at indivi dual level; when this happens it is important, difficult but not impossible to distinguish the role that individual level selfsel ection factors and the role that the group level factor s play in these groups in their impact on the effects of
PAGE 22
11 intervention. Our new method inco rporates a range of models to evaluate causal effects on a population when intervention exposure (p articipation or compliance level) is determined at both group and individual leve ls, and moreover may not be completely observable at either level. To illustrate our proposed method, we use as a motivating example the Gatekeeper Training Trial a multilevel randomized trial to improve services to middle and high school childre n with suicidal ideation in Cobb County, Georgia, USA. This Gatekeeper Training Trial is delivered to sc hool teachers, who in turn provide QPR (Question, Persuade, and Re fer) service to the st udents. Evaluation is focused on the schoollevel service of QPR. One technical aspect of the evaluatio n of QPR training is to examine the intervention effect in association with the timing of program adoption (earlyadopter vs. lateradopter). The varying tim e of program adoption implie s the intervention status changes over time, i.e. exposur e level potentially varies betw een earlyadopters and lateradopters. Moreover, participati on level varies across school staffs. In this context, PS is determined at multilevel, where randomization was applied at school level with intervention occurring at school staff level, a nd outcomes were collected at student level. As a result, some assumptions associated wi th Principal Stratification method of single level may no longer hold, and need to be re laxed. We also consider a randomeffects component in the models under PS method. Ou r approach thus differs from modeling associated with single level PS method. This dissertation is organized as follow s. Basic concepts related to Principal Stratification method are introduced and reviewed first in Chapter Two, followed by a discussion of an extension of the PS method to multilevel trials. As an illustration for our
PAGE 23
12 multilevel PS method, the randomized QPR Gatekeeper Training trial is discussed in detail in Chapter Three. The Gatekeeper study adopted a randomi zed crossover design where schools were randomly assigned to a time at which to change from control to intervention. The twostage design allowed us to look at where there is group level variation in participation, and examine wh ether intervention differences between early and late trained schools in ear lier phase would continue over time at individual level. The multilevel PS analysis of QPR for the Gate keeper program is conducted and compared with two traditional methods, IntenttoTrea t (ITT) and AsTreated (AT) analyses. Discussion and conclusion are presented in Chapter Four, including limitations of the PS method.
PAGE 24
13 Chapter Two Methodology for TwoLevel Principal Stratification Principal Stratification m ethod has been mostly used in randomized intervention trials which a single level posttreatment va riable determined principal stratification, often at an individual level. This chapter examines and discusses a new methodology that extends existing Principal Stratification met hod to randomized trials with multiple level posttreatment variables. The discussion is fo cused on how participation varies at either individual level or group le vel in a twolevel randomi zed trial and how principal stratification can be used to develop causal inferences in su ch studies. We begin with an introduction of Principal Stra tification method and its underlyi ng assumptions for a single level trial design. We revisit the two examples given in Chapter One. For historical reasons, we call the twostratum model in th e first example the Blooms model which we believe is the earliest application of the genera l principal stratification approach to correct for participation bias in randomized tria ls (Bloom, 1984). The model in the second example with four strata is referred to as the AngristImbensRubi n (AIR) model, which is the first to formalize part icipation/compliance involving co ncepts such as monotonicity and exclusion restriction assumptions. Both Blooms model and AIR model were developed for cases where the posttreatment va riable is at the indi vidual level (first level). We then discuss how to extend the Blooms model and AIR model to cases where participation may be determined at both gr oup (second) and individual level and analyze intervention causal effects using mixtures and marginal maximum likelihood as in Jo
PAGE 25
14 (2002). The method is capable of considering covariate effects at individual level in making inference of causal effects in all pr incipal strata. Finally, we demonstrate how timeeffect can be incorporated in the analys is in a randomization trial with longitudinal data. 2.1 Principal Stratification Me thod in Single Level Trials In a standard single level (individual le vel), twoarm randomized trial that tests one active intervention against a control c ondition, a set of individuals are randomly assigned to either in tervention condition (iZ = 1) or control condition (iZ = 0) (i= 1, n). However, among those assigned to th e intervention condition, some subjects participate in or comply with the interv ention, i.e., take prescribed medication or attend a designed program, while others do not participate. Intervention impact under the method of intenttotreat (ITT) is the difference in the m ean outcome between all those assigned to intervention and those assigned the control. The ITT approach ignores individuals participation completely. An alte rnative to ITT is to compare the mean of those participants in the intervention arm w ith that of the entire control condition. Because participants may differ from nonparticip ants, this estimate can still be biased and cannot be interpreted as pure causal effects. Ideally we would estim ate the causal effects based on those who participate in the inte rvention with those controls who would participate in intervention had th ey been offered the intervention. Assuming there are intervention benefits among those participating in the intervention condition, it is natural to assume that the nonparticipan ts would not receive the benefit because they have no (or minimal) exposure to the intervention. Because those
PAGE 26
15 who participate can differ from those who do not want to participate, any assessment of the causal effect of intervention on the participants must account for potential participation bias. The following are notations that lay out a genera l method for adjusting for participation bias. Let Z represent intervention status, with Z = 0 and Z = 1 being the control and intervention condition, respectively. This not ation carries over to experiments where there are two different active interventions instead of a control and an intervention condition. Let S be a posttreatment variable which is measured after the treatment assignment but before assessment of the final ou tcome of interest. In the present context, S is a binary indicator of partic ipation in the intervention, with S =1 representing participation and S = 0 nonparticipation. We use p iS to represent participation status at individual level for subject i. Here the superscript p refers to personlevel, in distinction from the g or group level that we introdu ce later for twolevel designs. In general, participation status of an in dividual depends on the random assignment iZ. Thus, p iS( Z ) is the indicator of pa rticipation of subject igiven the randomly assigned conditioniZ. For example, p iS(1) and p iS(0) correspond to the par ticipation status when subject iis assigned to intervention or control, respectively. We use the term potential mediator outcome p iS( Z ) to represent participation status across the two intervention conditions. Note that an individual can be assigned to either Z = 0 or Z = 1, and can be either a participant or nonparticipant. Thus p iS(0) and p iS(1) are the potential mediator outcomes. In the most general case, both can take value of 1 or 0. The potential mediator
PAGE 27
16 outcomes of p iS(0) and p iS(1) define all possible subsets of individuals, corresponding to all possible combinations of (p iS(0), p iS(1)): (0, 0), (0, 1), (1, 0), (1, 1) (Table 2). Such a classification is called principal stratification and the resultant classes are called principal strata. Table 2: Principal Stratification Define d on the Basis of Potential Mediator Outcome in Singlelevel Randomized Trials The preceding discussion illustrates that principal stratification is a crossclassification of study subject s under the potential outcome of mediator variable(s) (Frangakis, et al., 2002). Let p iC = (p iS(0), p iS(1)) = (0, 0), (0, 1), (1, 0), or (1, 1) be the indicator of possible principal stratum membership of subject i. Each and every subject is then randomly assigned intervention Z so that we observe partial information on the potential outcome of the mediator. For exampl e, if a subject is a ssigned to intervention Potential mediator outcome Potential outcome Principal stratum membership pC pS(0) pS(1) iY(pC 0)iY(pC 1) Individual causal effect given pC Never takers (pn) 0 0 iY(pn 0) iY(pn 1) iY(pn 1) iY(pn 0) Compliers (pc) 0 1 iY(pc 0) iY(pc 1) iY(pc 1) iY(pc 0) Always takers (pa) 1 1 iY(pa 0) iY(pa 1) iY(pa 1) iY(pa 0) Defiers (pd) 1 0 iY(pd 0)iY(pd 1)iY(pd 1) iY(pd 0)
PAGE 28
17 and participates, then the two possibilities are p iC = (0, 1) or (1, 1). That is, the participant could be either a complier or an alwaystaker. Let Y represents a response variable of in terest, which can be measured postintervention on each subject. Further let ) 1(iY denote the potential out come for individual i if the intervention cond ition is assigned and ) 0(iY the potential outc ome if the control condition is assigned. Depending on the participation status p iS, principal effects are defined as difference between potential ou tcomes under interventi on and under control within a principal stratum. Consider examin ing the individual level causal effects on each subject whose participat ion status is given by pC= pc, pc= (0, 0), (0, 1), ( 1, 0), or (1, 1). These individual level causal effects ar e defined as the difference between ) 1(iY )0(iY for each individual iconditional on pC given above: {) 1(iY )0(iY, i,p ic pc = (0, 0), (0, 1), (1, 0), or (1, 1)} (2.1) The average casual effect, T of intervention over all i ndividuals is the expected value of the difference ) 1(iY ) 0(iYconditional on pC= pc, pc= (0, 0), (0, 1), (1, 0), or (1, 1) : T = )1(( YE )) 0( Y (2.2) = )) 1(( YE )) 0(( YE Remark. For randomized trials that we focus here, we assume the assignment to intervention condition is independe nt of all baseline level ch aracteristics (Holland, 1986). There is a fundamental pr operty we impose on the poten tial mediator outcome p iS ( Z ). That is p iS (0) and p iS (1) are unrelated to the assi gned intervention condition, or
PAGE 29
18 equivalently, an individuals participation status in the active intervention condition is assumed not to change as a function of actual assignment to the intervention condition. Property 1 : The random indicator of principal strata p iC to which the thi subject belongs, is independent of the actual assignment of intervention cond ition. That is, p iC is independent of the treatment assignmentiZ (Frangakis & Rubin, 2002). Property 2 is a consequence of Property 1. Property 2: The expected principal effect within any principal stratum, as defined in Equation (2.2), is a causal eff ect (Frangakis & Rubin, 2002). There are two fundamental assumptions that underline Princi pal Stratification method. Assumption 2.1. Randomized Intervention Assignment. Here we assume that the intervention assignment iZ is exchangeable. That is, all individuals have the same probability of assignment to intervention. Additionally, Z is independent of all other random variables at the baseline. This assumption ensures comparability of subjects between the intervention condition and the control condition prior to the delivery of the interven tion (Holland, 1986; Rubin, 1974, 1978, 1980). Assumption 2.2: Stable unit treatment value assumption (SUTVA)
PAGE 30
19 Potential outcome for any individual is unrelated to the intervention assignment of other individuals in the sa mple (Cox 1956, Rubin 1978). Extending the notation of potential outcome Y (iZ) to iY(1Z = 1z, 2Z = 2z, iZ = iz nZ = nz ) as the response of subject i when all subjects in the sample are assigned to their respecti ve intervention condition 1z, 2z, nz SUTVA is expressed by iY(1Z = 1z, 2Z = 2z, iZ = iz nZ = nz ) = iY(iZ = iz ). This also implies that stratum membership of an individual is unc hanged regardless of the assignment to others. That is, iS(1Z = 1z, 2Z = 2z, iZ = iz nZ = nz ) = iS(iZ = iz ). Put it simply, SUTVA dictates that the potential outcomes are independent among individuals in the sample. It is conceivable, however, that SUTVA could be violated in some experiments. For example, if indivi duals assigned to the control condition have contact with individual s assigned to the intervention condition, the controls may become disappointed for not receiving intervention wh en the intervention condition is perceived to be more attractive. They could respond diffe rently if their friends were in the same control condition. We have laid out a general framework for single level experiments with selfselection into participation status. We perm it subjects in both intervention conditions to have the chance of participating or not par ticipating. However, for some experiments individuals assigned to one c ondition, i.e., control, may not have the possibility of participating. In the next section we revi ew how Bloom applied Pr incipal Stratification
PAGE 31
20 method to a standard single level twoarm randomized trial where individuals assigned to control have a zero probability to be exposed to active intervention. 2.2 Blooms Model for a SingleLevel Randomi zed Trial with Active Intervention versus Control As the earliest applicatio n of principal stratificat ion m ethod to correct for participation bias in randomized trials, th e Bloom model deals with the situation where individuals assigned to th e control condition have no chance to be exposed to intervention. In contrast, those assigned to the intervention condition can decide to participate or not to participate. That is, pS (0) cannot be equal to 1. Framed in statistical terms used in the preceding section to define principal strata, we have Pr(pS (0) = 1) = 0, Pr(pS (0) = 0) = 1, Pr(pS (1) = 0) > 0, and Pr(pS (1) = 1) > 0. When this situation arises in an experiment, data associated with pS(0) = 1 are always missing, i.e., part of the data in the strata of alwaystakers or defiers are not observable (Tables 3). However, intervention particip ation rate among individuals assigned to control condition could be borrowed from th at in the intervention group as assured by randomization. Table 3 shows that data are only available in two strata instead of four as the result of pS (0) = 1 not being possible. The two principal strata are compliers (pS (0) = 0, pS (1) =1) and nevertakers (pS (0) = 0, pS (1) = 0).
PAGE 32
21Table 3: Potential Values of Mediator Outcome for Blooms Model Note: these strata do not exist because subjects in the control condition can never get access to the intervention. Principal stratum membership Potential mediator outcome Potential outcome Individual causal effect given pC pC pS(0) pS(1) iY(pC  0)iY(pC  1) Never takers ( NonParticipation) (pn) 0 0 iY(pn  0) iY(pn  1) iY(pn  1) iY(pn  0) Compliers (Participation) (pc ) 0 1 iY(pc  0) iY(pc  1) iY(pc  1) iY(pc  0) Always takers (pa ) Not Possible* Not Possible* Defiers (pd) Not Possible* Not Possible*
PAGE 33
22 Bloom was interested in determining how effective an intervention was for those who participated in the intervention. He introduced a met hod of moment estimator that produces an unbiased estimate of the causal e ffect of the interven tion. In addition to assumptions 2.1 (Randomized Intervention A ssignment) and 2.2 (SUTVA), the Blooms model requires the following assumptions. Monotonicity Assumption (Angrist et al. 1996). Pr (p iS (0) = 1) = 0. That is, no one in the control group can receive the active intervention. Strongly Ignorable Treatment Assignment (Rubin 1978). Under the assumption Randomization Intervention Assignment the potential outcomes (iY (1), iY(0)) are independent of intervention assignmentiZ. It also implies that stratum membership is independent of intervention assignmentiZ. The assumptions of Randomized Assignment and Monotonicity together lead to Pr (p iC = complier  iZ = 1) = Pr (p iC = complier  iZ = 0), i = 1 , n Define ) 11Pr(ZSp to be the population leve l participation rate under intervention condition. It represents the pr oportion of subjects in the entire population who would participate if assigned to the intervention conditio n. We consider the case where the outcome variable Y is continuous. Recall that iY(0) and iY(1) are the potential outcomes for subject i if assigned to contro l or intervention, respectively. From the definition of causal effects given in Table 2 and equation 2.2, Bl ooms Average Causal Effect is in fact defined among the complie rs: ))0,0(())1,1(())0(())1(( p c p c p pSZYESZYEcZYEcZYEp p
PAGE 34
23 To estimate the causal effect above, we need to know ))0,0(( p cSZYEp which is not directly available. Bloom used an additional Exclusion restriction The Average Causal E ffect among nonparticipants is zero: 0))0,0(())0,1(( p n p nSZYESZYEp p With this exclusion restriction, we can see )1,0()1()0,1( )1,0()1()0,0( )()0()()0()0(p c p n p c p n p r c p r nSZ E Y SZ E Y SZ E Y SZ E Y cpZ E YnpZ E Y ZEYp p p p p p Therefore, /))1)(0,1()0(()1,0( p n p cSZ E YZEY SZ E Yp p and the Average Causal Effect among the participants is given by ) ))0,1(()1())0(( ())1,1(( SZ YE ZYE SZ YE p n p cp p (2.3) Assuming that the distribution of ZY (= 0), ZYpn(= 1, pS= 0), and ZYpc(= 1, pS =1) has the same variance, Note that sample means )0( ZY )0,1( p nSZYp, and )1,1( p cSZYpare unbiased estimate of ))0(( ZYE, ))0,1(( p nSZYEp, and ))1,1(( SZYEpc, respectively. Bloom proposed the following unbiased moment estimator of the ACE ) )0,1() 1()0( ()1,1( S Z Y ZY S Z YACEp pn c (2.4) Bloom further gave an estim ate of standard error (SE) for the ECA
PAGE 35
24 S Z Y Z Y S Z Y p pn c 2 2 ))0,1((ra v) 1())0((ra v ))1,1((ra v (2.5) 2.3 Principal Stratification in TwoLevel Randomized Trials In this section, we extend Principle Stratification to twolevel random ized trials. A twolevel randomized trial refers to one in which randomization is placed at a group (second) level, above the indivi dual level, but participati on can vary at both group and individual level, and moreover outcome is m easured at individual level within each group. For example, physicians are randomly assigne d to one of two interventions, and we assess the impact of these in tervention conditions on individu al patients, a nd individual patients can decide whether to participate. Schoollevel randomized trial is another example where impact is assessed on student s but intervention is randomized at school level, participation level can vary at school and individual levels. Similar to the notation in Section 2.2, let Z represent random assignment of intervention at the 2nd level, where Z = 0 or 1, representing c ontrol and intervention, respectively. Let Control 0 on Interventi 1jZ be the indicator of intervention assignment for thj group ( j = 1 , k) in the trial. Further, let S( Z ) be the indicator of participati on status after group randomization Z = 1 or 0. In a twolevel randomized trial, S= (g jS,p ijS) is a two dimensional binary indicator of participation at both second level and first le vel. This is a new notation
PAGE 36
25 introduced specifically in this dissertation to describe twole vel randomized trials. In this notation, the second component p ijS indicates the par ticipation level of individual subject i from group j and the first component g jS represents participation level for group j Therefore, S= (1, 1) represents partic ipation at both levels and S = (0, 0) represents nonparticipation at neither level. In the following we use the notation ()(j g jZS,p ijS(jZ)) to make S dependent on random assignment Z Using unique combinations of potential mediator outcomes (g jS(0), p ijS(0), g jS(1), p ijS(1)) we are able to defines subsets of individuals according to their potential partic ipation status, thereby forming a principle stratification of the subjects. Table 4 summa rizes the stratification with a total of 16 possible principal strata: (g jS(0), p ijS(0), g jS(1), p ijS(1) = (0, 0, 0, 0), (0, 0, 0, 1), .., (1, 1, 1,1). In this context, the notation fo r enumerating all possible principal strata in a twolevel randomized trial is new, and these are summarized in Table 4. Column 1 of Table 4 is the name of each principal stratum as defined by potential mediator value and intervention condition at both group level (column 2, column 4) and individual level (column 3, column 5). The superscript p indicates the potential mediator outcome of individual level a nd the superscript g indicates the potential me diator outcome of group level. For example, individual subjects who are in the group of nevertakers but are individual complier are denoted by pgcn. The causal effects can be defined at either individual level or group level within each principal stratum. With a twolevel trial, we consider that an individual in a group can have two potential outcomes, Y (jZ= 0) if the group was assigned to control and Y (jZ= 1) if
PAGE 37
26 under intervention. For the thi individual at the thj group, the corresponding notation of potential outcomes is ijY(0), and ijY(1) if assigned to control or intervention condition, respectively.
PAGE 38
27Table 4: Principal Stratification base d on Potential Mediator Outcome in Twolevel Randomized Trials Note: g jS( Z ) = Group level Potential Mediator Outcome; p ijS( Z ) = Individual level Potential Mediator Outcome Potential Mediator Outcome Control Intervention Principal Stratum by Level of Group and individual Participation pgCC g jS(0) p ijS(0) g jS(1) p ijS(1) Group Nevertakers Individual Nevertakers (pgnn) 0 0 0 0 Group Nevertakers Individual Complier (pgcn) 0 0 0 1 Group Complier Individual Nevertakers (pgnc) 0 0 1 0 Group Complier Individual Compliers (pgcc) 0 0 1 1 Group Nevertakers Individual Defier ( pgdn) 0 1 0 0 Group Nevertakers Individual Alwaystakers ( pgan) 0 1 0 1 Group Complier Individual Defier (pgdc) 0 1 1 0 Group Complier Individual Alwaystakers (pgac) 0 1 1 1 Group Defier Individual Nevertakers ( pgnd) 1 0 0 0 Group Defier Individual Complier ( pgcd) 1 0 0 1 Group Alwaystakers Individual Nevertakers ( pgna) 1 0 1 0 Group Alwaystakers Individual Complier ( pgca) 1 0 1 1 Group Defier Individual Defiers (pgdd) 1 1 0 0 Group Defier Individual Alwaystakers ( pgad) 1 1 0 1 Group Alwaystakers Individual Defier ( pgda) 1 1 1 0 Group Alwaystakers Individual Alwaystakers (pgaa) 1 1 1 1
PAGE 39
28 Denote by )(jjZW the average potential value for the j th group, with jW(0) = jN i jji jZ Y N1)0( 1 being the average pote ntial value under c ontrol condition and jW(1) = jN i jji jZ Y N1)1( 1 under the intervention. So we can in princi ple define grouplevel causal effect for each principal stratum defined in Table 4 as 1,1) 1, (1, .., 1), 0, 0, (0, 0), 0, 0, (0, },))0()1({( pgpgpgpgccccCCCCWWE. The assumptions for randomized trials at individuallevel (section 2.1) are restated here. Some, however, need modificatio n in order to be applicable in twolevel randomized trials. Randomized Intervention Assignment. All groups have the same probability of assignment to intervention, and that Z is drawn independently of all baseline vari ables prior to the assignment. Stable Unit Treatment Value Assumption. (Cox 1956, Rubin 1978). Potential outcomes for each group are unr elated to the intervention status of other groups; stratum membership of one group is unchanged regardless of other groups assignment. Note, however, groups assigned to contro l condition may not always have the same probability as groups in the interventi on condition to participat e active intervention.
PAGE 40
29 Moreover, information on indi vidual or group level partic ipation cannot always be observed because of the nature of randomi zation and potential out come. As the degree and amount of information about participati on and compliance status vary from one study to another, the structure of pr incipal strata also differs a nd so does the analysis. In the next section we discuss four such examples and illustrate four distinct models. 2.4 PS Models for TwoLevel Randomized Tria ls with Active Intervention versus Control This section focuses on four different models through four examples. In these examples intervention is assigned at the group le vel and participation status is determined by a combination of both group and individual level compliance. Across these examples, the amount of observable information on participation at each level varies. Table 5 summarizes the four examples and the associated models for analysis. In discussing these models, we present the fundamental characters and illustrate of each the examples from the literature. Model 1 represents the situation where participation status is observed at individual level only, and is ignored (i.e., unmeasured) but can always be determined at group level. There, if individuals in a group assigned to control condition have a zero probability to receive the intervention, then Blooms model for trials of single level randomization can be directly used. However, if individuals have the possibility to receive the intervention even when their group is assigned to control condition, the single level AIR model may be used to estimate individual level CACE (complier average
PAGE 41
30 causal effects) with additional assumptions. Fo r illustration of single level AIR model, we consider the example of the Flu shot study (Hirano, el at., 2000). Model 2 represents the situation where pa rticipation status is available at group level only, but ignored at individual level. There, when assigned to the intervention condition some groups would r eceive the intervention while others decline intervention and receive the same condition as if assigned to controls. In such cases, we have extended classic Blooms model or AIR model to be applicable. For convenience, the extended models are called GB (GroupBloom) model or GAIR (GroupAIR) model. We use the example of GAgatekeeper study (Wyman, el at 2008) to illustrate the GB model. When intervention is only available to i ndividuals whose group is assigned to and participates in intervention, we consider model 3 (GBPB model), a combination of GB model and individuallevel Bl oom model (PB). When indivi duals in any group have the same probability to receive the intervention, we consider model 4 (GAPA model), a combination of GAIR and indi vidual level AIR model. We use the example of Good Behavior Game (Ialongo et al., 1999) to il lustrate both model 3 and model 4.
PAGE 42
31 Table 5: Summary of Models for Twolevel Randomize d Trials with Group and Individual Level Participation Restriction Example Model Group Level Participation Individual Level Participation Number of Principal Strata Individual & Group Randomization Intervention Participation Outcome Blooms Model Only possible in intervention group 2 principal strata: pc (Participation), pn (Nonparticipation), (individual level) Elementary school students in randomly assigned classrooms Individual : students; Group : classroom Parent training, only available to families in assigned classrooms Parent attendance in parent training (Ialongo et al., 1999) Child aggressive behavior 1 (participation status is available at individual level only) AIR Model Unmeasured None 4 principal strata: pa, pc, pn, pd Patients within physicians Individual : patients; Group : physicians Physicians encouraged to have their patients get a flu shot Patients can receive flu shot regardless of physician behavior (Hirano, el at. 2000) Did patient get the flu? GBlooms Model* Only possible when assigned to intervention Unmeasured 2 principal strata: gc (Participantion), gn (Nonparticipation), (group level) Middle/High schools randomly assigned to the intervention Individual : school staff; Group : schools Gatekeeper training program provided to all school staff in assigned schools School staff receive training within intervention schools (Wyman, el at., 2008) Frequency that school staff refers students for suicidal behavior 2 (participation status is available at group level only) GAIR Model None Unmeasured 4 principal strata: ga, gc, gn, gd Counties randomly assigned to one of Community development team model Level of implementatio n achieved by Number of foster care placements
PAGE 43
32two implementation strategies for an evidencebased practice Individual : foster care family; Group : county versus individual county implementati on intervention county within 18 months (Chamberlain, el at. 2006) made by county 3 GBPB Model* Only possible when assigned to intervention Only possible when the group participates 4 principal strata: pgnn, pgcn, pgnc, pgcc 4 GAPA Model* None None 16 principal strata: pgnn, pgcn, pgnc, pgcc,pgdn,pgan,pgdc, pgac,pgnd, pgcd, pgna, pgca,pgdd, pgad, pgda, pgaa Elementary school students randomly assigned to classrooms and classrooms randomly assigned to intervention Individual : parent; Group : classroom Parent training in behavior management Level of program fidelity delivered by school counselors; Family attendance in training sessions (Ialongo et al., 1999) Child aggressive behavior Note*: G = Group, B = Bloom, A = AIR
PAGE 44
332.4.1. AIR Model The flu shot study mentioned in Chapter 1 is an example to which the single level AIR model may be applied under additional assumptions about individual level compliance as presented in Hirano et al. (2000) Physicians were assumed to completely comply with their intervention assignment (g roup level). As a result, all patients were classified into four principal strata based on their compliance status (individual level) with their physicians (see Table 6). In order to extend the AIR model to the flu shot study, Hirano et al. (2000) applie d specific assumptions, which we state below, Monotonicity Assumption. The monotonicity assumption at the individual level is the probability that defier is zero ()0()1( j p ij j p ijZSZS). In the case of the flu shot example, this simply eliminates the possibility that there are patients who w ould not want a flu shot if their physician would receive an encouragement letter, but otherwise would want a flu shot. In terms of probability, the assumption dictates P r (0)0)1)1(( & 1)0)0(( g j p ij g j p ijSS SS, hence effectively removing the stratum of individual defier Hirano et al. (2000) argued for the reasonableness of the monotonicity assumption. However, there would be unwanted consequences of the AIR model, as is shown later, wh en the monotonicity assumption is untrue. Exclusion restriction. The Average Causal Effect among alwaystaker (pgac) and nevertaker (pgnc) are zero. Consider Y to be a measured
PAGE 45
34 outcome at the individual level. We have under this exclusion restriction that 0 ))0(())1(( ) )0(() )1(( )(01 aa ac ac pgpg pgpg pgZYEZYE acCCZYEacCCZYEacCACEpg pg and 0 ))0(())1(( ) )0(() )1(()(01 nn nc nc pgpg pgpg pgZYEZYE ncCCZYEncCCZYEncCACEpg pg where a1 and a0 are the population means of potentia l outcome for alwaystaker under intervention and control condition, respectively; n1 and n0 are the population mean of nevertaker under interv ention and control condition, respec tively. In our example, the first relation implies that the chance that a su bject gets the flu given that they would have gotten the flu shot is the same regardless of wh ether their physician was sent the letter to encourage her to recommend her patients to have the flu shot. In the second class, those patients who would not get the flu shot regardle ss of their physician being encouraged or not, would also have the same chance of getting the flu. Let us now assume the individual leve l monotonicity assumption holds for a particular trial. Due to the monotonicity assu mption, we are able to ignore the stratum of defier, ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 1, 1, 0), and consider only the average causal effects within the stratum of complier ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 0, 1, 1). The Complier Average Causal Effect is naturally defined as
PAGE 46
35CACE = cc01 where) )1((1pgpg cccCCZYE and ) )0((0pgpg cccCCZYE are the population mean of complier under interven tion and control condi tion, respectively. Note that the overall popul ation means of potential outcome under intervention and control can be expressed as ))1((1 1 1 1aanncc + + = ZYE aanncc+ + = ZYE0 0 0 0))0(( (2.7) respectively, and c n and a are the population proportion (probability) of complier, nevertaker, or alwayst aker. It follows that )( )()()( ) () (01 01 01 01 0 0 0 1 1 1 01ccc aaannnccc aannccaanncc + + + + + = CACE= c cc 01 01 (2.8) In situations where 0d in contrast with the montonicity assumption, i.e. there are individuals whose compliance beha vior would be oppos ite to intervention assignment, the overall population means of potential outcome would be given by 1 1 1 1 1 ddaanncc * + + = d da anncc * + + = 0 0 0 0 0 (2.7a) It follows that + + = dddccc dddaaann nccc)()( )()()()(01 01 01 01 01 01 01
PAGE 47
36 and c ddd ccCACE )(01 01 01 (2.8a) If intervention has causal effect s among both complier and defier, cc 01 and dd01 are of opposite signs. As a result, CACE* > CACE, and CACE under the monotonicity assumption would be an underestimat e of the true causal effects. In practice compliance behavior is not observable, so some practical assumptions are useful for estimating CACE*. For example, if we assu me causal effects of the intervention with )(01 01 cc dd and 0 the estimate for CACE* may be approximated by **CACE= dc cc 01 01. (2.8b) In the case where intervention effects are opposite among compliers and defiers, the estimator in (2.8) will be an overestimate of CACE*.
PAGE 48
37Table 6: Principal Stratifi cation of Flu Shot Study Note: g jS( Z ) = Group level Potential Mediator Outcome; p ijS( Z ) = Individual level Potential Mediator Outcome Potential Mediator Outcome Control Intervention Principal Stratum by Level of Group and individual Participation pgCC g jS(0) p ijS(0) g jS(1) p ijS(1) Group Complier Individual Nevertakers (pgnc) 0 0 1 0 Group Complier Individual Compliers (pgcc) 0 0 1 1 Group Complier Indivi dual Defiers (pgdc) 0 1 1 0 Group Complier Indi vidual Alwaystakers (pgac) 0 1 1 1
PAGE 49
382.4.2. GBlooms Model The GA gatekeeper study to be discussed in Chapter Three is an example of GBlooms model. In this study, randomization occurred at the school level, intervention was implemented at the school level by training of school staff to identify signs of suicide and to ask the youth whether they felt suicid al. Group level participation was based on whether or not training at the school occurred by a certain time, and later we examine other characteristics, such as the proportion of school staff that ar e trained as a function of time. Youth outcomes included whether they were referred to th e school support staff to deal with suicide and other life threatening behaviors, whether they were suicidal, and whether they attended mental health treatmen t. These data were all collected at the student level in a deidentified fashion. Researchers also measured the percentage of staff who had been trained and the time of staff receiving training at each school in aggregate instead of which individual staff member was tr ained. It our first l ook at this problem, it is assumed that participation status of staff within school is the same as the participant status of their school. Also, no one in a school could receive the interv ention if the school was a control (Blooms model) and all youth were exposed to the effects of the training if the school participated in the intervention. It is also assumed that the participation status of a school, which is only observed for those randomly assigned to be trained, was independent of its assigned intervention condition. In the gatekeeper study, due to the requirement that control schools had a zero probability to r eceive intervention, participation status was only observable among the intervention schools but not control
PAGE 50
39 schools. Therefore, all schools were classified into two principal strata formed according to participation status at th e group and individual levels, i. e., never taker and complier. Table 7 illustrates the design of the intervention. Using the notation of Table 4, g jS(1) represented the participation status of the j th school if it had been assigned to the intervention condition. Participation was de termined by the time at which the school would implement the gatekeeper program. Ther efore the level of participation could be defined as the time the school started the gate keeper program. A simpler approach is to dichotomize the timing of intervention into early (gS=1) or later (gS= 0). This is the case in Table 7. If assigned to control school participation g jS(0) = 0 because it had no chance to participate, hence information on the level of participation was missing. This is seen from Table 6 as there is only one row, corresponding to))0(),0((p ij g jSS = (0, 0). Once the schools participation status was determined, the staffs participation status had to be the same as the schools. Therefore )1()1(p ij g jSS is given in the only two columns of Table 7. The design resulted in only two strata, nevertaker (lateradopter) (pgnn) or complier (earlyadopter) (pgcc). For this example, the common assumptions warrant slightly different interpretation. Monotonicity Assumption. 0)1)0(Pr(g jS and 0)1)0(Pr(p ijS. That is, no school in the control condition he nce no staff in a control school can receive active intervention. For now we consider the situation with no covariates at the group level to predict participation status, and therefore a ll groups have the same probability ( ) of
PAGE 51
40 participating if they are assigned to the intervention condition. Consider W to be a measured outcome at the group level. Additi onally, Average Causal Effect is defined within complier (ear lyadopter) stratum ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 0, 1, 1) as )) 0(())1(()adopter)(early complier ( ZWEZWE ACEpg pgcc cc Note that we have used the subscript pgcc to indicate expectation within the stratum of complier (earlyadopter) while omitting )(j p ijZSat the same time. Because membership in the stratum of complier (earlyadopter) among those assigned to control ( Z = 0) is not completely observable, the fo llowing exclusion rest riction is used to provide identifiability. Exclusion restriction. The Average Causal E ffect among nonparticipants (nevertaker(lateradopter): ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 0, 0, 0) is zero, ACE(nevertaker (lateradopter)) = )) 0(())1(( ZWEZWEpg pgnn nn= 0 Note that ) (0))(()())0(())0((cPZWEnPZWEZWEr cc r nnpg pg Because a subject can only be either a complier (earlyadopt er) or a nevertaker (lateradopter), randomized intervention assignment dictates that )(cPr and 1)(nPr. The preceding equation gives )1())0(())0(( ))0(( ZWEZWE ZWEpg pgnn cc
PAGE 52
41 = )1())1(())0(( ZWEZWEpgnn As a result, the Average Causal Effect among the compliers (earlyadopter) is given by ACE= ) )1())1(())0( ( ))1 ( ZWEW(ZE (ZWEpg pgnn cc (2.9) Table 7: GBlooms Mode: GA Gatekeeper Study Note: g jS ( Z ) = Group level Potential Mediator Outcome; p ijS ( Z ) = Individual level Potential Mediator Outcome S= 0: later intervention S= 1: early intervention Potential Mediator Outcome Control Intervention Principal Stratum by Level of Group and Individual Participation pgCC g jS(0) p ijS(0) g jS(1) p ijS(1) Group Nevertakers Individual Nevertakers (pgnn) 0 0 0 0 Group Complier Individual Compliers (pgcc) 0 0 1 1 Group Defier Individual Defiers (pgdd) Not Possible Not Possible Not Possible Not Possible Group Alwaystakers Individual Alwaystakers (pgaa) Not Possible Not Possible Not Possible Not Possible
PAGE 53
42 In equation (2.9) all expectation is based on observed data. For example, )) 0(( ZWE is based on those assigned to the control c ondition. Equation (2.9) allows for unbiased estimation of ACE based on moment estimators, assuming that ) 0(ZW, ) 1( Z Wpgnn, and )1(Z Wpgcchave the same variance. Note that the moment estimators k j jW k )ZW1)0( 1 0( nj j nnW k Z Wpg)1( 1 )1(1, and k cj j ccW k Z Wpg)1( 1 )1(2 are unbiased estimators of )),0(( Z W E ))1( ( Z WEpgnn, and ))1 ( (Z WEpgcc. Of the latter two estimators, the summation is over the set of nevertaker (lateradopter) and the subset of complier (earlyadopter) w ithin the control a nd intervention groups, respectively, and 1k and 2k are the number of groups in each set, respectively. Finally, an unbiased moment estimator of the causal effect within the stratum of complier (earlyadopter) is ECA = ) )1() 1(0)(Z ()1( Z W W Z Wpg pgnn cc (2.10)
PAGE 54
43 The standard error (SE) of ECA is approximately equals to: Z W W Z Wpg pgnn cc2 2 ))1((ra v) 1(0))(Z(ra v ))1((ra v (2.11) Had the groups under the control condition had a possibility to rece ive the intervention, then there would be four principal strata and AIR model would appl y. Results presented in section 2.4.1 can be re adily extended to group leve l analysis by using the group average instead of the individual average. 2.4.3. GBPB Model/GAPA Model With GBPB/GAPA models, we consider th e situation where participation status is observed at both group level and individual level. An ex ample is the FamilySchool Partnership intervention for firs t graders in Baltimore, MD (Ialongo et al., 1999). In that study, children were randomly allocated to one of three classrooms in first grade; the three classrooms were randomized to a classroomcentered intervention, a parent training intervention, or a control, respectively. The parent training in tervention and control conditions are a pair of conditions considered in an example for these two models. In the parent training intervention, a counselor in the school was traine d to provide parent training in the childs behavior management and a home environment to support school achievement. These parent trainings were pr ovided weekly in school. However, the class receiving the control condition cannot partic ipate in the intervention, at least by assumption that there is no contamination acro ss classes. Thus it is possible to examine
PAGE 55
44 participation at two levels, both at the cla ss level represented by the school counselor, who may or may not deliver the program with fu ll fidelity, and at the level of the family, who may or may not attend an adequate number of sessions to receive the benefit of this intervention. One of the critical issu es with twolevel participation is the interrelationship between the two levels. We consider and examine two situations below: individuals cannot participate wh en their group does not particip ate, or not restricted at both individual level participation and group level participation. Under the first situation wh ere the school counselor can deliver the intervention with fidelity or not, and pare nts of first graders cannot par ticipate in the intervention if their class is assigned the cont rol condition, subjects are classified into 4 principal strata, pgnn, pgcn, pgnc, or pgcc (Table 8a), The resultant principal strata are assured by the follow modified monotonicity assumption. Monotonicity Assumption (Angrist et al., 1996). )0()0(p ij g jSS= 0. In other words, Pr (g jS (0) = 1) = 0 and Pr(p ijS(0) = 1) = 0. When an individuals participat ion status is always the same as the associated groups participation status (2 principal strata), GB model is appropriate. When this is not the case, we consider the GBPB model in whic h we assume there is no group level causal effect among nonparticipant gr oups (Angrist et al., 1996). Exclusion Restriction. The Average Causal Effect among nonparticipants is zero. That is,
PAGE 56
450))0,0(())0,1(()0(g j g j g jSZWESZWESACE. This in turn can be expressed in two parts: )0)1(,0)1(,0)0(,0)0(()(g j p ij g j p ij pgSSSSACEnnACE 0))0,0,0(())0,0,1((p ij g j p ij g jSSZWE SSZWE and )1)1(,0)1(,0)0(,0)0(()(g j p ij g j p ij pgSSSSACEcnACE 0))0,0,0(())1,0,1((p ij g j p ij g jSSZWESSZWE. We note that the exclusion restriction 0 )(pgcnACEmay be replaced by adding)1()1(p ij g jSS, or equivalently 0)1)1(&0)1(Pr( p ij g jS S, to the preceding monotonicity assumption. We also note an additional possible condition for exclusion restriction: 0))0,0,0(())0,1,1(( )0)1(,1)1(,0)0(,0)0(()( p ij g j p ij g j p ij g j p ij g j pgSSZWESSZWE SSSSACEncACE The preceding equation implies that th ere are no causal effects among those nevertaker individuals. The us e of the additional restricti on would impact the estimation of causal effects as is seen below in our discussion. The average causal effects in the compiler stratum (both group and individual complier pgcc) is given by: )1)1(,1)1(,0)0(,0)0((p ij g j p ij g jSSSSACE =))0,0,0(())1,1,1(( p ij g j cc p ij g j ccSSZWESSZWEpg pg. (2.12)
PAGE 57
46 Let 1 be the group level participation rate under the intervention condition, and 2 be the individual level participation rate within groups assigned to the intervention condition. Because of randomization, the propor tion of participation at group level is the same as that of complier, i.e., 1)(g rcP, and subsequently, 11)(g rnP. By the same argument, the proportion of participati on among individuals is th at of participation when assigned to an intervention class, 2)(p rcP and 21)(p rnP. Note that ))0(())1(( ZWEZWE = ))0()1((21 ZWZWEpg pgcc cc + )) 0()1(()1(2 1 ZWZWEpg pgnc nc + )) 0()1(()1(21 ZWZWEpg pgcn cn + ))0()1((()1)(1(2 1 ZWZWEpg pgnn nn Under the exclusion restriction, ))0(())1(( ZWEZWE= )) 0()1((21 ZWZWEpg pgcc cc + )) 0()1(()1(2 1 ZWZWEpg pgnc nc It follows )}0,0,0()0,1,1({)1())0()1(( )(21 2 1 p ij g j p ij g j pgSSZW SSZWE ZWZWE ccCACE Under the additional exclusion restriction 0)(pgncACE it reduces to 21))0()1(( )( ZWZWE ccCACEpg (2.13)
PAGE 58
47 Because the group )0,0,0(p ij g jSSZ is not differentiable from the other strata, ))0,0,0((p ij g j ccSSZWEpgis not directly computable. We thus use the exclusion restriction to reexpress ))0,0,0((p ij g j ccSSZWEpgas follows. ))0,0,0((p ij g jSSZWE = ) ())0,0(()())0,0((g g j c g g j ncPSZWEnP SZWEg g Similarly, we have 2 2)0,0,0()1()0,0,0( )()0,0,0()()0,0,0( ))0,0(( p ij g j cn p ij g j nn p p ij g j cn p p ij g j nn g j nSSZEW SSZEW cPSSZEWnPSSZEW SZWEpg pg pg pg g 2 2)0,0,0()1()0,0,0( )()0,0,0()()0,0,0( ))0,0(( p ij g j cc p ij g j nc p p ij g j cc p p ij g j nc g j cSSZEW SSZEW cPSSZEWnPSSZEW SZWEpg pg pg pg g Using three equations from the excl usion restriction, we can show 21 12 21 1)1))(0,1,1(( )1))(0,1(())0,0,0(( ))0,0,0(( p ij g j nc g j n p ij g j p ij g j ccSSZWE SZWESSZWE SSZWEpg g pg in which all terms can be estimated because data are observable within the respective strata. For example, ))0(( ZWE and ))1(( ZWEis based on those assigned to the control and intervention conditions, respectiv ely. Equation (2.13) allows for unbiased
PAGE 59
48 estimation of CACE based on moment estimators, assuming that ) 0(ZW, and )1(ZWhave the same variance. Note that the moment estimators k j jW k )ZW1)0( 1 0( k j jW k )ZW1)1( 1 1(, are unbiased estimators of )) 0(( Z W E and )) 1(( Z W E Finally, an unbiased moment estimator of the complier average causal effect is: ECCA = 21 0)(Z1)(Z W W (2.14) The standard error (SE) of ECCA is approximately W W2 2 2 1 0))(Z(ra v1))(Z(ra v
PAGE 60
49 Table 8a: Baltimore Good Behavior Game: GBPB Model for Intervention of First Graders in Baltimore (with restriction on individual level) Potential Mediator Outcome Control Intervention Principal Stratum by Level of Group and Individual Participation pgCC g jS(0) p ijS(0) g jS(1) p ijS(1) Group Nevertakers Individual Nevertakers (pgnn ) 0 0 0 0 Group Nevertakers Individual Complier (pgcn) 0 0 0 1 Group Complier Individual Nevertakers (pgnc) 0 0 1 0 Group Complier Individual Compliers (pgcc ) 0 0 1 1
PAGE 61
50 However, if more complicated exam ples under which both group level and individual level can have the alternative to re ceive the intervention, then all subjects will be classified into 16 principal strata based on assigned in tervention status, group level participation status and indivi dual level participation status (GAPA Model, Table 8b). Analyses will be done at either the group level or individual level. For example, if we assume that an individuals participation stat us is the same as their group participation status the causal models can be analyzed by using the same method as the AIR model or GAIR model. The number of principal strata will be reduced from 16 to 4. Otherwise, more assumptions are required for analysis in order to reduce number of status beside monotonicity assumption and exclusion restri ction assumption. The resultant principal strata are assured by the follow modified monotonicity assumption. Monotonicity Assumption (Angrist et al., 1996). The monotonicity assumption excludes the probability of having defiers from both group level and individual level, which assume)0)0(()1)1(( g j p ij g j p ijSS SS and )0()1(g j g jSS In our example, it simply eliminates the possibility that there are parents who would not wa nt attend to training sessions if the school counselor would deliver the program with full fidelity, but otherwise would want attend to traini ng sessions. In terms of probability, the assumption dictates 0)0)1)1((&1)0)0((( g j p ij g j p ijrSS SSP or 0)1)0(&0)1(( g j g jrS SP,
PAGE 62
51 hence effectively removing the stratum includi ng any individual defier or group defier. The number of principal strata will be reduced from 16 to 9 (Table 8b). Table 8b: Baltimore Good Behavior Ga me: GBPB Model for Intervention of First Graders in Baltimore (without restriction) Potential Mediator Outcome Control Intervention Principal Stratum by Level of Group and Individual Participation pgCC g jS(0) p ijS(0) g jS(1) p ijS(1) Group Nevertakers Individual Nevertakers (pgnn ) 0 0 0 0 Group Nevertakers Individual Complier (pgcn) 0 0 0 1 Group Complier Individual Nevertakers (pgnc) 0 0 1 0 Group Complier Individual Compliers (pgcc ) 0 0 1 1 Group Nevertakers Individual Alwaystakers ( pgan ) 0 1 0 1 Group Complier Individual Alwaystakers (pgac) 0 1 1 1 Group Alwaystakers Individual Nevertakers ( pgna ) 1 0 1 0 Group Alwaystakers Individual Complier ( pgca ) 1 0 1 1 Group Alwaystakers Individual Alwaystakers (pgaa) 1 1 1 1
PAGE 63
52 2.5 Incorporating Individual Charac teristics through Regression Models In general, intervention effects could be im pacted by a subjects characteristics as well as other factors. As a result, individua ls response varies. To reflect the betweenindividual variation in response, we can in corporate individual leve l covariates in the stratum mean of the Bloom m odel, as Little & Yau (1998) did. This can be done through adopting the linear model for the stratumspecific mean: iY = p i p i p i Xn p i Xc p i Zc p n p cncnXcXcZncp p p p p ' (2.14) where iY is a continuous response variable of individual i, (pc = 1 & pn= 0 ) if individual i is a complier (participation), and (pc = 0 & pn = 1) if individual i is a nevertaker (nonparticipation). Further,pc is intercept for compliers and pn is intercept for nevertaker; iZ =1 indicates intervention condition and iZ =0 is control condition; Z is the coefficient of intervention effects, representing average effect of intervention; iX is a 1 p vector of individual le vel characteristics and 'X is 1 p vector of associated coefficients. The complieraverage causal effect (ACE) is equal to pZc as given in equation (2.14). The indicators p pnc and are for the individuals principal strata membership. Recall that this class membership is independent of the intervention assignment and is only observed on those who are assigned to the in tervention condition. However, the membership status for those under the control conditi on can be predicted under an assumption that Z is independent of the class membership and the participation rate is the same under both intervention and control conditions, the case for a welldesigned randomized trial. Furthermore, it is possible to allow for participation
PAGE 64
53 (compliance) rate to vary with individual ch aracteristics. Take a si ngle level randomized trial for example. The log odds of particip ation (among those assi gned to intervention) can be expressed as a linear function of the individual leve l covariates via a logistic regression model as suggested by Little and Yau (1998): pic pp i pp iX cC cC' 10) ) Pr(1 ) Pr( log( (2.15) wherepicX is individual level covariates predictiv e of participation (compliance). Assume that the distribution of Y is normally distributed with variance 2 and mean pn for nevertaker (n onparticipation), 0pc for complier (participation) assigned to the control condition, and 1pc for complier (participation) assigned to the intervention condition. The likelihood based on the observed data then has the form: )],( ),([ ),( ),( )data(2 0 2 )0,0{ 2 1 )1,1{ 2 )0,1{ p p p ii p p p ii p ii pc i c n i SZi n c i c SZi n i SZi nyg yg yg yg L (2.16) where 1 p pcn and ),,,,, ( 2 01p ppppccncn is set of parameters in the model, and ) ,(2yg denotes the probability density of a normal distribution with mean and variance 2. Maximum likelihood estimates of the parameters are obtained by maximizing this mixture of likelihood f unctions of two prin cipal strata of pc(compliers) andpn (nevertakers). In particular, the estimation of pc is via the logistic regression model of (2.15). The complieraverage causal effect (ACE) is then estimated by 0 1p pcc pZc where pZc is the maximum likelihood estimates of pZc
PAGE 65
54 However, equation 2.14 requires the excl usion restriction as sumption under which the causal effect is zero for nevertaker. This may not hold in certain cases. If the nevertakeraverage causal effect does exist, then e quation 2.14 can be rewritten in a full model: iY = p i p i p i Xn p i Xc p i Zn p i Zc p n p cncnXcXnZcZncp p p p p p ' (2.17) where pZc is complieraverage causal effect and pZn is nevertakeraverage causal effect. In reality,p pXn Xc and may vary with individual characteristics. The causal effect can be estimated by using marginal maximum likelihood estimates form the models defined in equations 2.14 and 2.15. However, these mixture likelihood functions involve individuals cl ass membership, which is unobservable for subjects in the control condition. Thus the estimation principal stratum membership is a key issue. To this end, the mixture model ties together the unobserved class membership in the control condition with individual level covariates and the marginal effects of intervention, and then maximizes the marginal maximum likelihood to estimate the regression coefficients. Although this mixtur e likelihood approach has been used in analysis of single level randomized studies, its use in multilevel randomized trials is relatively new and is considered in this dissertation. The following sections discuss the mixture likelihood approach in analysis of twolevel randomized trials.
PAGE 66
55 2.6 Mixtures and Marginal Maximum Like lihood Approach for TwoLevel Randomized Trials In this dissertation we have defined a twoleve l randomized trial as one where random assignment of intervention condition occu rs at a group instead of individual level. Further, intervention status may change over time during the study peri od. Principal strata are defined by the combination of assigned intervention condition and the compliance of the subjects at both group and individual levels. The full likelihood function is the mixture of those associated with all possible principal strata. The mixture then leads to a marginal likelihood function on which maximum likelihood estima tion is feasible. In the following we illustrate the regression m odel and its corresponding mixture likelihood function for the AIR models. 2.6.1 TwoLevel Randomized Trials with Active Intervention versus Control We assume that in the AIR model ijY, has same distribution in all principal strata ),,, (p ij g j p ij g j p ij g j p ij g j p ij g jdcacncccCC In principle, individual membership in a principal stratum may not be observed, but can be predicted under a multinomial distribution in conjunction with individu allevel covariates. In this case, equation 2.14 can be extended as )()()()( ' ' p ij g j dc p ij g j ac p ij g j cc p ij g j nc ijdc ac cc ncYp ij g j p ij g j p ij g j p ij g j )( )( )( )(' )( )( )( )( p ij g j dcxdc p ij g j acxac p ij g j ccxcc p ij g j ncxncdcx acx ccx ncxp ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j )( )( )( )(' )( )( )( )( p ij g jj Zdc p ij g jj Zac p ij g jj Zcc p ij g jj ZncdcZ acZ ccZ ncZp ij g j p ij g j p ij g j p ij g j
PAGE 67
56 )()()()()( )( )( )( p ij g j bdc p ij g j bac p ij g j bcc p ij g j bncdc ac cc ncj p ij g j j p ij g j j p ij g j j p ij g j )()()()()( )( )( )( p ij g j wdc p ij g j wac p ij g j wcc p ij g j wincdc ac cc ncij p ij g j ij p ij g j ij p ij g j j p ij g j (2.18) where defier}1&0&0&0{ takeralways}0&1&0&0{ complier }0&0&1&0{ takernever}0&0&0 &1{ p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g j p ij g jdcacccnc dcacccnc dcacccnc dcacccnc 'p ij g jnc, 'p ij g jcc ,'p ij g jacand 'p ij g jdcare intercepts for nevertaker, complier, alwaystaker and defier, respectively; p ij g jncx, p ij g jccx, p ij g jacx and p ij g jdcxare individuallevel characters for nevertaker, complier, alwaystaker and defier; xs are coefficients of x covariates; Zs represent the intervention effect. Errors j p ij g jbnc )( (nevertaker), j p ij g jbcc )( (complier) j p ij g jbac )( (alwaystaker) and j p ij g jbdc )( (defier) are assumed to be normally distributed with zero mean and the between groups variance 2 )(j p ij g jbnc, 2 )(j p ij g jbcc, 2 )(j p ij g jbacand 2 )(j p ij g jbdc. Similarly, errors ij p ij g jwnc )( ij p ij g jwcc )( ij p ij g jwac )( and ij p ij g jwdc )( are assumed to be normally distributed with zero mean a nd the within groups variance 2 )(ij p ij g jwnc 2 )(ij p ij g jwcc, 2 )(ij p ij g jwacand 2 )(ij p ij g jwdc. The principal strata membership can be predicted by using a multinomial logistic model: },,,{ 10 10) exp( ) exp( ) Pr(pgpgpgpgpg pg pgdcncaccctt tit tit p ij g j p ij g jX X ttCC (2.19) Where } ,,,{pgpgpgpgpgdcncaccctt and pgtitXis individual level covariates predictive of principal strata membership.
PAGE 68
57 Similarly, assume that the distribution of Y is normal distributed with variance 2 and mean pgnc for nevertaker, pgac for alwaystaker, 0pgcc for complier assigned to the control condition, and 1pgcc for complier assigned to the intervention condition, 0pgdc for defier assigned to the control condition, and 1pgdc for defier assigned to the intervention condition. The likelihood based on the observed data then has the form for AIR model: )],( ),([ )],( ),([ )],( ),([ )],( ),([ )data(2 0 2 )1,0{ 2 0 2 )0,0{ 2 2 1 )1,1{ 2 1 2 )0,1{ pg pg pg ii pg pg pg pg ii pg pg pg pg pg ii pg pg pg ii pgdc ij dc nc ij SZi nc cc ij cc nc ij SZi nc ac ij ac cc ij cc SZi dc ij dc nc ij SZi ncyg yg yg yg yg yg yg yg L (2.20) where 1 pg pg pg pgdcacccnc and ),,,,,,, ,,, ( 2 0 1 0 1pg pgpg pg pgpgpgpgpgpgdcdcacccccncdcacccnc is a set of parameters in the model, and ) ,(2yg denotes the probability density of a normal distribution with mean and variance 2. Maximum likelihood estimates of the parameters are obtained by maximizing this function with respect to the parameters The complieraverage causal effect (CACE) is then estimated by 0 1pg pgcccc where )(p ij g jcc are maximum likelihood estimates of )(p ij g jcc
PAGE 69
58 Under the assumptions of monotonicity and exclusion restricti on, defier does not exist and a causal effect will not exist for th e nevertaker and alwaystaker class. Then equation 2.18 becomes ijY = )()()( ' p ij g j ac p ij g j cc p ij g j ncac cc ncp ij g j p ij g j p ij g j )( )( )(' )( )( )( p ij g j acxac p ij g j ccxcc p ij g j ncxncacx ccx ncxp ij g j p ij g j p ij g j p ij g j p ij g j p ij g j )(' )( p ij g jj ZccccZp ij g j )()()()( )( )( p ij g j bac p ij g j bcc p ij g j bncac cc ncj p ij g j j p ij g j j p ij g j )()()()( )( )( p ij g j wac p ij g j wcc p ij g j wincac cc ncij p ij g j ij p ij g j j p ij g j (2.21) The CACE as defined in equation 2.8 is equal to CACE= )( 01 01p ij g jcc c cc Given the linear mixed regression models a nd its corresponding like lihood function, the AIR model above can be fitted using the Mplus software. The sandwich type estimators are used in Mplus to adjust for any correla tion among responses of different individuals characterized by selected characters. In the case of GBloom Model, equation 2.18 is simplified to ijY = )( )( )()( )( )( ' p ij g j ccxcc p ij g j nnxnn p ij g j cc p ij g j nnccx nnx cc nnp ij g j p ij g j p ij g j p ij g j p ij g j p ij g j j p ij g jj Zcc p ij g jj ZnnbccZ nnZp ij g j p ij g j)( )(' )( )( )()()()()( )( )( )( p ij g j wcc p ij g j winn p ij g j bcc p ij g j bnncc nn cc nnij p ij g j j p ij g j j p ij g j j p ij g j (2.22) due to group variation may exist, we define jb p i g jCC ~ N(0, )2 b is the random effect of the group. The principal strata membersh ip can be predicted by using a multinomial logit model with a group variation:
PAGE 70
59 ) exp(1 ) exp( ) Pr(' 1 10 1 10 j ccw w ccb b j ccw w ccb b p i g j p i g jpg ij pg j pg ij pg jX X X X ccCC (2.23) where jbXis a vector of betw eengroup covariates; ijwX is a vector of withingroup covariates and betweengroup residual j predictive of principal strata membership. j causes the logistic va lue vary across groups which mean ing the proportion of compliers differs across groups. The likelihood based on the observed data then has the form: j 2 0 2 )0,0{ 2 1 2 1 )1,1{ 1 1 2 )0,1{)]),( ),([ ( )),( ( )),( ( )data( pg p pg jj pg pg p pg jj pg jj pgcc ij c nn ij SZi nn k j cc ij cc SZi k j nn ij SZi nnyg yg yg yg L (2.24) where 1k and 2k are the number of groups in each principal stratum in intervention condition and 1 pg pgccnn 2.6.2 TwoLevel Randomized Trial with Random Time of Crossover fro m Control to Active Intervention We now consider a specia l case of twoleve l intervention study in which groups are originally assigned to the control conditi on and at a later time ch ange to intervention condition. We call this a dynamic waitlisted design or timed assignment of intervention, and its properties of ITT analyses have been investigated elsewhere (Brown et al., 2006). We introduce this study design in preparation fo r the application to be discussed in the next chapter.
PAGE 71
60 Let Control 0 on Interventi 1jtZ be the indicator of intervention assignment for j = 1 , k, the number of groups in the trial at time t Randomized Intervention Assignment We assume that the jtZ is exchangeable, that is, all groups have the same probability of assignment to intervention at each different time interval, and that Z is drawn independent of all the other random variables in the study. Monotonicity Assumption Pr (g jtS(0) = 1) = 0. That is, no one in the control group at time t can receive the active intervention before that group receives the intervention. Stable Unit Treatment Value Assumption (Cox 1956, Rubin 1978). All potential outcomes for each individua l are unchanged regardless of the assignment of all other units. However, the principal stratum to which each group belongs will not change over the time. Under these assumptions, a regression m odel can be developed by including in equation 2.22 the effects of timed entran ce into active intervention due to timedassignment: ijtY = )( )( )()( )( )( ' p ij g j ccxcc p ij g j ncxnn p ij g j cc p ij g j nnccx nnx cc nnp ij g j p ij g j p ij g j p ij g j p ij g j p ij g j )( )(' )( )( p ij g jj Tcc j p ij g jj ZccccT bccZp ij g j p ij g j
PAGE 72
61 ) ()( )()()( )( )( )( p ij g j twcc p ij g j twnn p ij g j tbcc p ij g j tbnncc nn cc nnij p ij g j j i p ij g j j p ij g j j p ij g j (2.25) where )( tccp ij g jrepresents the intervention effect on compliers due to the delay in receiving active intervention, errors tbnnj p ij g j)( (nevertaker) andtbccj p ij g j)( (complier) are assumed to be normally distributed with zer o mean and the between groups variance 2 )( tbnnj p ij g j and 2 )( tbccj p ij g j for those groups assigned to start intervention at time t Similarly, errors twncj i p ij g j)( and twccij p ij g j)( are assumed to be normally distributed with zero mean and the within groups variance 2)( t j i w p ij n g j c and 2 )(twccij p ij g jfor individuals star ting intervention at time t Equation 2.23 can be applied here to pr edict the principal strata membership for each group because we assume that the membership of each principal stratum will not change over time. The likelihood function will become: t cc ij c nn ij SZi nn t k j cc ij cc SZi t k j nn ij SZi nnpg p pg jj pg pg p pg jj pg jj pgyg yg yg yg L ))]),( ),([ (( ))),( (( ))),( (( )data(j 2 0 2 )0,0{ 2 1 2 1 )1,1{ 1 1 2 )0,1{ (2.26) In Chapter Three we apply the new methods developed in this chapter to a twolevel randomized trial, the QPR gatekeeper study.
PAGE 73
62 Chapter Three Using the T woLevel Principle Stra tification Method to Evaluate the QPR Gatekeeper Training Program Four main models, which can be possibly formed at a twolevel randomized trial by using Principal Stratification method, were developed and discussed in the previous chapter. Both the method of moment estim ator and the marginal maximum likelihood estimator for mixtures have been discussed fo r these four models. This chapter will apply these models to a specific twolevel randomized trial, the Georgia Gatekeeper Study. It is the first randomized trial of the gatekeeper tr aining in a schoolbase d setting, and it is a crossover design trial in whic h schools have a random time to change from control to intervention. The time when school s started to receive training varied due to participation at the first level. This allows us to ex amine whether intervention effects varied by participant status and whether the effects continued over time. However, based on the nature of the study design, in which the pr obability of all subjects within a group under control condition received in tervention was zero and grouplevel numbers of student referred to receive mental health professiona l assessment were collected for each school due to deidentified issue, GBloom model w ill be applied to evaluate an intervention causal effect. The exclusion restriction assu mption was investigated because there were time intervals where the intervention conditi on of schools was changed from control to active intervention. In particular, strong a nd weak exclusion restri ction assumptions will
PAGE 74
63 be introduced for this type of data. These two assumptions will be compared in the following analyses. This chapter will start with an introduction of the intervention that was used in the randomized trial, Gatekeeper Training Program. Then it will be followed by an introduction of study design. In the analysis section, Principal Stratification method will be used to evaluate the effect of the Gatekeeper Training Program. Analyses will be conducted for two situations, the first time period only and th e entire four study periods. The results will be compared with two tradi tional methods, IntenttoTreat (ITT) and AsTreated (AT). Similar analyses will be conducted separately by school type as well. 3.1 Introduction Nearly 4,000 people aged 15 die by suic ide each year in the United States. The gatekeeper training program is one type of intervention that has been designed and conducted to prevent youth suicides. The school gatekeeper training program is a schoolbased program that is designed to train all sc hool staffs, who act as gatekeepers, in order to improve early identification of students at high risk for suicide and to facilitate timely referrals for mental health services. School gatekeepers can include any adults in the school (e.g., counselors, teachers, coaches, administrators) who are in a position to observe and interact with students. The main purpose of the gatekeeper training program is to increase awareness and knowledge about youth suicide risk, to directly ask troubled youth if they are suicidal, and to help suicidal youth to receive appropriate mental health services. Increased knowledge of risk factors for suicide, and changing attitude towards
PAGE 75
64 asking troubled youth if they are suicidal have the poten tial for incr easing referral behavior and may promot e early identification of suicidal students. Although the gatekeeper training program has been widely applied to various communities under different social and environm ental settings, it has not been rigorously tested and evaluated. Most studies simply reported that the traini ng program is helpful (Nelson, 1987, Barrett, 1985, Spiritto, et al., 1988). Based on the reported finding so far, the program basically increases awarene ss of suicide warning signs, knowledge of treatment resources, and willingness to make referrals to mental health professionals among gatekeepers (Shaffer et al., 1988). However, the studies to examine the eff ects of the program on actual number of referrals were rare. One randomized trial wa s conducted in Cobb County, Georgia, to evaluate the training effect of the QPR ( Question, Persuade, Refer) (Quinnett, 1995) gatekeeper training program on knowledge of suicide, appraisal including willingness to assume a gatekeeper role for suicide prev ention, selfreported intervention behaviors with students, and improvement of early dete ction. The QPR training has been shown to clearly increase knowledge of suicide warning signs, intervention behaviors, appraisals including gatekeeper efficacy, and service access, as tested on adults by an intenttotreat analysis (Wyman, el at., 2008). To date th e effects on youth referrals have not been reported. The analyses in this chapter will exam whether the QPR can increase the number of middle and high school students referred to receive mental health assessment. Of special interest is the fact that there was variation in the timing and completeness of the training of adults in schools. These may be due to selfselection factors that are
PAGE 76
65 relatively stable in the schools, or this variation may be due to more transitory or even spurious factors in the schools. The tran sitory and permanent referral behavior for different schools will also be investigated here. 3.2 Method The QPR gatekeeper training occurred with school staffs in the Cobb County School District in Georgia. Funding for this study came from a National Institute of Mental Health grant. The trial started at January 8, 2004. It is the first randomized trial of the gatekeeper training in a schoolbased se tting within the U.S. The intervention was used in this trial is called QPR gatekeeper training program. This dissertation will examine the QPR training effect on the outcome of the trial, which was measured by the number of middle and high school students wh o have been referred to receive mental health assessment. This section will start with an introduction of the QPR training, followed by description of how the randomized trial has been conducted, and how the method of analyses has been chosen and what analysis strategy is. 3.2.1 Study Design and Pa rticipant Population 3.2.1.1 Intro duction of the QPR Gatekeeper Training The QPR ( Question, Persuade, Refer ) Gatekeeper training program is designed to train school staffs directly. It increases knowledge on suic ide among school staffs in order to help potential suicidal students to access professional serv ices. School staffs will learn three basic lifesaving intervention skills to provide suicide prevention among youth.
PAGE 77
66 The three intervention skills are: question a person who shows warning signs about suicide, persuade the person to pursue professional help, and refer and direct the person to appropriate resource s (Quinnett, 1995, 1999). The QPR is designed based on the belief or theory that thos e individuals would like to talk about their distre ss to someone around them or whom they trust about their feelings if they are at risk for selfdestruction and violence. It is also believed that suicide warning signs can be recognized by some one who possesses enough suicide knowledge and is trained professionally. Professional staffs in the districts Prevention/Interv ention Center (PI/C), school counselors, and school staffs in Cobb County School District, GA recei ved three levels of the QPR training program respectively. In order to play a role as a trainer and evaluator, professional staffs received more than 12 hour s training before the study started. Then a counselor from each school that was assigned to the training program received more than 6 hours training from PI/C professional staffs after the study star ted. Finally, professional staffs and the counselors coled a one and onehalf hour gatekeeper training session for school staffs (gatekeepers) in assigned schools. The training covered important knowledge about youth suicide and how to identify students at highrisk. The training also taught school staffs how to ask a student about suicide, to persuade a student to obtain help, and to refer a stude nt to receive professional help About a halfyear after the initial training, the school staffs were invited for a 30minute refresher training.
PAGE 78
67 3.2.1.2 Study Design The random assignment of the time that each school would be trained occurred at the school level. Thirtytwo middle and high schools in th e Cobb County School District participated in the study. Three schools were excluded from the trial since they already received training before the trial began. All th irtytwo schools were stra tified into 4 strata by school type (middle vs. high school) and the rates of student crisis referrals during the 20022003 schoolyear which preceded the trial (low vs. high). Within each stratum, one half of the middle and high schools were rando mly selected to receive the QPR training during the 20032004 schoolyears For the 16 schools on the w aiting list, training was planned to start during the fo llowing school year, since the sc hool district felt strongly that all schools should receive the training. This classic waitlisted design offered the opportunity to compare referral rates for suicide that were reported to a central district office, among trained and untrained schools. No schools w ithdrew from the study. However, it was apparent to the research team immediately, that additional information could be obtained by extending th e time of the trial while continuing to schedule the remaining schools at random times to be trained. The investigators prepared and published a technical paper documenting the advantage in power resulting from this continued random assignment of crossover times for training (Brown et al., 2006). With the approval of the Data Safety and Monitoring Committee and the funding agency, the trial design was modified so that it could be extended. In summary, by the end of 20042005 schoolyear, all 16 early tr aining schools received the training. The study used dynamic waitlisted or rollout design for the remaining 16 schools on the waiting
PAGE 79
68 list after the first year of the study (Table 10). These remaining 16 schools on the waiting list were stratified into 4 strata by schoo l type and the level (rat e) of student crisis referral during the previous year. One school from each schoolsize/referral rate stratum was randomly selected and assigned into a block. The 4 schools in each block were promptly scheduled to receive training one afte r another. This formed 5 different training periods within this 2year study. The Consort Diagram for Study Design (Figur e 2) provides detail information of the study design, such as numbers of middle and high schools received training, and proportion of school staffs have been trained at each design time period.
PAGE 80
69 Table 10: Study Design of the QP R Gatekeeper Training Program Note: one middle school was scheduled training at period 4 but r eceived training a few days into peri od 5; one high sch ool was scheduled to be trained in peri od 5 but received training at the end of period 4 Year Time Block Classic WaitListed Design Dynamic WaitListed Design Period QPR Trained WaitListed QPR Trained WaitListed 1 Spring 04 1 16 Trained: 14 16 Trained: 0 Fall 04 Spring 05 1 Trained: 2 Trained: 0 2 2 20 Trained: 4 12 Trained: 0 3 24 Trained: 4 8 Trained: 0 4 28 Trained: 3 12 Trained: 1 5 32 Trained: 4 12 Trained: 0
PAGE 81
70 Figure 2: Consort Diagram for the Study Design of the QPR Gatekeeper Training Program 35 middle/high schools 10 Middle Schools Low Referrals 0203 10 Middle Schools High Referrals 0203 6 High Schools Low Referrals 0203 6 High Schools Low Referrals 0203 3 schools excluded due to prior gatekeeper training 4,849 Staff and 48990 Students 0304 Year 4783 Staff and 50227 Students 0405 Year 4715 Staff and 49569 Students 0506 Year Randomized 1stTraining Period Jan 2004 to Aug 2005Train in Period 1 Later Training 10 middle and 6 high schools 2858 Staff 0304, 2463 Staff 0405 23138 Students 0304, 23805 Students 0405 10 middle and 6 high schools 1991 Staff 0304, 2320 Staff 0405 25852 Students 0304, 26422 Students 0405 2007 (86.51%) staff trained 2ndTraining Period Aug 2005 to Oct 2005 Randomize Randomized Previously Trained Later Training Train in Period 2 8 middle and 4 high schools 1706 Staff 0506 16602 Students 0506 2 middle and 2 high schools 615 Staff 0506 6669 Students 0506 402 (65.37%) staff trained 10 middle and 6 high schools 2394 Staff 0506 26298 Students 0506
PAGE 82
71 Randomize Previously Trained Train in Period 3 4 middle and 4 high schools 1398 Staff 0506 12971 Students 0506 4 middle schools 380 Staff 0506 3631 Students 0506 246 (64.74%) staff trained 12 middle and 8high schools 2909 Staff 0506 32967 Students 0506 3ndTraining Period Oct 2005 to Jan 2006 Later Training Randomize Previously Trained Train in Period 4 2 middle and 2 high schools 626 Staff 0506 6254 Students 0506 2 middle and 2 high schools 648 Staff 0506 6717 Students 0506 423 (65.28%) staff trained 16 middle and 8 high schools 3289 Staff 0506 36598 Students 05064ndTraining Period Jan 2006 to Feb 2006 Later Training Previously Trained Train in Period 5 2 middle and 2 high schools 626 Staff 0506 6717 Students 0506 443 ( 65.28% ) staff trained 18 middle and 10 high schools 3937 Staff 0506 42736 Students 05065thTraining Period Feb 2006 to Jun 2006 Note: one middle school was scheduled training at period 4 but received training at period 5, one high school was scheduled training at period 5 but received training at period 4.
PAGE 83
72 3.2.1.3 Participant Population The Cobb County School District is the second largest school system in Georgia. Its student population grows by nearly 2000 ev ery school year. Over 97,000 students, in grades Kg 12 were enrolled in the school system during the 20012002 schoolyear. 60% of them were White, 25% were African American, and 8% were Hispanic (Table 12). A total of 48,990 students were in the schools that participated in the study during the 2003 2004 schoolyear. While these participants were all in grades 6 12, they have a similar distribution to the full population in terms of population ethnicity. The students were 56% White, 29% were African Ameri can, and remaining 15 % were Hispanic, Asian and others (Table 11). At the base line, there were no significant differences between the 16 early training schools and the 16 waitlisted schools on race/ethnicity, gender, and grade level (Table 12). Table 11: Ethnic Distribution of Cobb County School Students Total Student Population (20012002 schoolyears) (N =97,343 ) Total Student Participants (2003 2004 schoolyears) (N = 48,990 ) N % N % Ethnicity White 58,747 60.35 27,370 55.9 African American 24,267 24.98 14,295 29.2 Hispanic 7,953 8.17 4,164 8.50 Asian 3,572 3.67 1,767 3.61 MultiRacial 2,541 2.61 1,277 2.61 American Indian 224 0.23 117 0.24
PAGE 84
73 Table 12: Demographic Distribution of Students from the 32 Study Schools (2003 2004 schoolyears) Among 13,080 staffs in the Cobb Co unty schools during the 20012002 schoolyears, the majority (84.95%) were White, mo re than 12% was African American, and less than 2% were Hispanic (Table 13). Approximately twothirds of the staffs were located in middle or high schools. This was the ta rget population who took the QPR gatekeeper training course. Of 13,080 staffs who held jobs in the 32 study schools, 4,853 received QPR training. However, we obtai ned training information only on 4,403 of them. Among those, Training School(N =25,852 ) WaitListed School(N = 23,138 ) N % N % Ethnicity White 14,135 54.67 13,235 57.20 African American 8,035 31.08 6,260 27.06 Hispanic 2,126 8.22 2,038 8.81 Asian 871 3.37 896 3.87 MultiRacial 622 2.41 655 2.83 American Indian 63 0.24 54 0.23 Gender Female 12,720 49.20 11,273 48.72 Male 13,132 50.80 11,865 51.28 Grade 6 grade 3763 14.56% 3737 16.15% 7 grade 3934 15.22% 3893 16.83% 8 grade 3954 15.29% 3845 16.62% 9 grade 4230 16.36% 3575 15.45% 10 grade 3635 14.06% 3011 13.01% 11 grade 3451 13.35% 2747 11.87% 12 grade 2885 11.16% 2330 10.07%
PAGE 85
74 128 were administration staffs, 187 were suppor ting staffs, and the remaining 4,100 staffs were teachers. There were more fulltime s upporting staffs in training schools than in waitlisted schools ( p = 0.002) (Wyman, el at., 2008). Among those school staffs, there were no significant differences in terms of gender, ethnicity, and years of experience between 16 early training schools an d 16 waitlisted schools (Table 14). Table 13: Ethnicity Distribution of Cobb County School Staffs Total Staff Population (20012002 schoolyears) (N =13,080 ) Total Staff Participants (2003 2004 schoolyears) (N = 4,393 ) N % N % Ethnicity White 11,111 84.95 3693 84.08 African American 1,631 12.47 581 13.22 Hispanic 191 1.46 72 1.64 Asian 94 0.72 19 0.43 MultiRacial 41 0.31 21 0.48 American Indian 10 0.08 7 0.16
PAGE 86
75 Table 14: Demographic Distri bution of School Staffs from the 32 Study Schools (2003 2004 schoolyears) 3.2.1.4 Measures Within the school district, all crisis re ferrals were sent to the Districts PreventionIntervention Center (P/IC), which assessed the need for professional evaluation (SameDay Assessment). A centraliz ed record keeping system for reporting and referring youth for life threatening behavi or, either suicidal or homicidal ideation or behavior, had been in place for the past 15 years, and a crisis protocol system was in Training School(N =2,263 ) WaitListed School(N = 2,130 ) N % N % Ethnicity White 1,843 81.44 1,850 86.87 African American 362 16.00 219 10.28 Hispanic 42 1.86 30 1.40 Asian 6 0.27 13 0.61 MultiRacial 9 0.40 12 0.56 American Indian 1 0.04 6 0.28 Gender Female 1,575 69.60 1,481 69.21 Male 688 30.40 649 30.79 Years of Experience < 1 year 143 6.30% 132 6.10% 1 10 years 1187 52.40% 1,099 51.10% 11 20 years 504 22.30% 503 23.40% 21 30 years 354 15.60% 344 16.00% more than 30 years 75 3.30% 62 2.90%
PAGE 87
76 place to respond to the needs of these youth. An evaluated documentation was then completed by P/IC center staff for each referred student. Due to deidentified issue, school level numbers of student referred to receive mental health professional assessment were collected. School level covariates such as gender, grade, and race/ethnicity were collected as well. 3.2.2 Methodology The Principal Stratification m ethod has been used to examine the QPR training effect. Due to the nature of the study design, specifically, the GBloom model has applied in this example. The reason that the Princi pal Stratification method has been chosen is because (1) group level data have been coll ected; (2) sixteen schools had been selected and assigned to be trained in the QPR duri ng the first period; (3) some schools under the QPR training condition started to receive training quickly wh ile training in other schools was slow, and (4) the remaining sixteen sc hools had been assigned to the waitlisted condition. None of the 16 schools on the wait inglist had any chance to attend training. The purpose of using this method is to compare the effectiveness of training by the intensity of the schools participation in training which could only be measured among the 16 schools selected for early trai ning. We intended to estimate how many students among all referred from a school were due to the effect of QPR training. We also wanted to examine whether the QPR training has a different effect on the schools starting training early vs. later.
PAGE 88
77 The evidence given by Figure 3 shows the SameDay Assessment rate per 1000 people per month during the en tire study period. Sixteen sc hools assigned to training group at first period were dichotomized as either EarlyAdopter school or LaterAdopter school based on when they received tr aining. A school is defines as an EarlyAdopter school if its first training started within 81 da y after beginning of the study (January, 2004) at the first period under tr aining condition, otherw ise, a school is a LaterAdopter school. This cut point is the median value of the first training time so that the observed compliance rate ( ) is 50%. Therefore, thirtytwo schools have been divided into 6 groups. Both the Earlyadopt er group and the Lateradopter group consist of 8 training schools which received their fi rst training within or after 81 days after training started. The schools in Block1 (Fi gure 3) are 4 schools selected to receive training during the second period. Block2, Block3 and Block4 each include 4 schools selected to receive training during the third, forth and fifth period respectively. The 6 lines in Figure 3 represent the variations of the SameDay Assessment rate per 1000 people per month over the time for 6 different schoolgroups. The solid line denotes the schoolgroups under training condition, and th e dash line indicates those under control condition. Note that the Ea rlyadopter group has continuou s solid lines while all the other schools begin with dashed lines and th en convert to solid lines when they begin training. The Earlyadopter group (black line) appears to have a higher SameDay Assessment rate than the Lateradopter group (pink dashed and solid line) except at the last time point when referral rates are compar atively low overall (Figure 3). This suggests selfselection factors that are persistently different between the early adopter and later
PAGE 89
78 adopting schools. Another indication of the persistent differences across schools is the fact that the patterns of refe rrals are generally similar and parallel over time. Except for the schools in Block3, all schoolgroups show a similar temporal pattern in their SameDay Assessment rates, which increases at th e beginning and reaches the maximum at the second time point, then decreases towards the third time point, at which the rates are still higher than at the first time point. The rate s continuously drop towards the fourth time point, and increase somewhat towards the fifth time point, and show certain departure from parallelism at the 6th time point. Cont rarily, the SameDay Assessment rate for Block3 schools decreases from the beginning until the third time point, then follows a similar pattern as other schoolgroups. Figur e 3 does show that there is a continuing differential rate of referrals throughout for the early adopte rs in Block 1, and it also displays the dramatic difference in referral rates across time. When we examine whether there are changes in referral rates as a function of training ti me, that is when the curves change from dotted to solid lines, we do not perceive any major shifts that occur as a function of training. There is heterogeneity in the timing of trai ning and number of staffs trained in the QPR schools that were selected to be traine d during the first period. One implication of this heterogeneity is that it may be a selfs election factor; some concerns, which may be due to the principal, school counselor, or the climate of the school, may lead some schools to adopt training much more readily and intensiv ely than did other schools.
PAGE 90
79 Figure 3. SameDay Assessment Rate s of 6 Block Schools across Time Sameday Assessment Rate0 5 10 15 20 25 10/6/20034/23/200411/9/20045/28/200512/14/20057/2/2006Training TimeRate (10000people/month) Block1 EarlyAdopters Block1 LaterAdopters Block3 Schools Block4 Schools Block5 Schools Block6 Schools
PAGE 91
80 3.2.2.1 Method The moment estimate and mixtures a nd marginal maximum likelihood estimate notations of the causal effect within the st ratum of complier (ear lyadopter) ha ve been developed in Chapter Two. Those notati ons can be extended for this example. interval time64 ..... 2, 1, period 5 4, 3, 2, 1, level grade andcity race/ethni gender,by sector school in the students ..,. 2, 1, schools 32 ..., 2, 1, Let t T j Ni jth j t j t j Zth th jt at time condition training QPR the toassigned wasschool theif 1 at timelist waiting the toassigned wasschool theif 0 early training started school theif 1 later training started school theif 0 th th g jj j S Note that T represents the design study period and t represents the actual time point or day when a school rece iving its training. In our anal yses we have broken all the time intervals into subperiods when any additi onal staff training occurred in any school. The posttreatment variable,g jS, is only observed if Zj= 1 and p ijSis always equal to g jS under an intervention condition since we are una ble to observe the training status of each individual school staff. Ther efore, all schools will be classified into two grouplevel principal strata but the princi pal stratum membership of each school will not change over time. The two grouplevel principal strata are g jC= g jc, complier (earlyadopter) stratum ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 0, 1, 1), and g jC = g jn, nevertaker (lateradopter) stratum ))1(),1(),0(),0((p ij g j p ij g jSSSS = (0, 0, 0, 0).
PAGE 92
81 However, the principal stratum membership is always missing for schools in the waitlisted condition and can be treated as a mixture problem either without school level covariates used as predictors of principa l stratum membership or with school level covariates used as predic tors (Little & Yau, 1998). Figure 4 shows a simplified schematic drawi ng of the model analysis presented in the study. The count variable, SameDay Asse ssment, is regressed on the school level covariates ggrade, ggender, grace/ethnicity and the intervention variable training. The categorical latent variable g jC(Early/Later adopter gr oups) is the trainingreceiving status of a school with class 1 referring to Earlyadopter group and class 2 referring to Later adopter group. This variable is observa ble for the training condition, but unobservable for the waitlisted condition. The arrow from Early/Later adopter groups to the SameDay Assessment indi cates that the inte rcept of SameDay Assessment across the classes of g jC (Early/Later adopter gr oups). The arrows from Early/Later adopter groups to training indicate that the slopes in regression of SameDay Assessment on trainin g vary across the classes of g jC (Early/Later adopter groups). The arrows from ggrade, ggender, and grac e/ethnicity to g jC (Early/Later adopter groups) in Figure 4 re present the multinomial logistic regression of Early/Later adopter groups on these covariates.
PAGE 93
82 Figure 4. A Simplified Schematic Drawing of the Model Analysis for the Georgia Gatekeeper Project Sameday Assessment Training GGender GGrade GRace/ Ethnicity Early/Lateradopter groups
PAGE 94
83 In this study, the response variable is the number of same day assessments from each school within each interval of time. Let us define ) ( xUgjtzc as the number of same day assessments that are indexed by school le vel characteristics, i.e., in our case, x represent the cross classificat ion by school, gender, race/ethnic ity, and grade level. Note also that this total is indexed by time t as well as principal stratum gC and school j Then we define tx j jUW to be total number of referred students for thj school. Note that the moment estimators developed from Chapter Two are: k j jW k )ZW1)0( 1 0( nj j nW k Z Wg)1( 1 )1(1, and k cj j cW k Z Wg)1( 1 )1(2 The unbiased moment estimator of the causa l effect within the stratum of complier (earlyadopter) is: ECA = ) )1() 1(0)(Z ()1( Z W W Z Wg gn c The standard error (SE) of ECA is approximately equals to: Z W W Z Wg gn c2 2 ))1((ra v) 1(0))(Z(ra v ))1((ra v
PAGE 95
84 However, students have been referred may vary by their characteristics. By applying the mixtures and marginal maximum likelihood estimate method, ) ( xUgjtzc is assumed to have a Poisson distribution with a mean of gjtxzc The overall model considered is given by the general set of predictors of this Poisson rate. x offsetx jtxzC jtxzCg g )log( ZT ZC ZXzt g zc zxg jtbb (3.1) On the first line of Equation 3.1, it in cludes an offset term corresponding to the number of students in that school with covariates x times the duration of the time for interval t the effect of known covariates, and intercept. On the second line, three fixed effects have been listed as interactions between assignment and covariates, assignment and strata, and assignment and time. On the last line, two random effects have been listed to take into account of vari ation across time and schools. Bo th of these latter terms can include known covariates, i.e., the effect of longer periods of time and the contrast between middle and high schools. As mentioned at the beginning of this chapter, the exclusion restriction assumption may be violated in the study due to different intervals in time where schools training status changes from control to active intervention. Therefore, two additional assumptions, Strong Exclusion Restricti on and Weak Exclusion Restriction are introduced. The Strong Exclusion Restriction imp lies that no causal effect among later adopter groups exist at any time point. The W eak Exclusion Restriction implies that there
PAGE 96
85 is a causal effect among la ter adopter groups. Specifi cally, for all time periods T all schools j all categories of covariates x and for late adopters gC = gn ifgnjtx1 = gnjtx0 we say that the strong exclusion restri ction holds. If on th e other hand, this relationship only holds until the school is formally trained, or njtx 1 = njtx0 for time t where school j is converted to a fully trained cond ition, then we say the weak exclusion restriction applies. 3.2.2.2 Hypotheses By applying the GBloom model of the Principal Stratificatio n method to examine the QPR training effect. The following questions can be answered: Hypothesis 1: the QPR training has an effect on increase of student SameDay Assessment rates at the first training period, and, the effect is different for the schools start the training early vs. later. Hypothesis 2: the QPR traini ng effect varies by students characteristics such as gender, race/ethnicity, and grade level at the first tr aining period, and, the effect is same for all schools regardless their starting time of training. Hypothesis 3: the QPR training effects pe rsist over time, varies by school starting time of training (training early vs. later), a nd varies by students gender, race/ethnicity, and grade level. Hypothesis 4: the QPR training effects vary by school type, middle school vs high school.
PAGE 97
86 3.2.2.3 Model Selection Strategy In order to answer the questions liste d above, the table below illustrates the specific sets of models that have been exam ined relative to the options in Equation 3.1 (all contain the same offset, which is ignored in this table). The analys is starts with strong exclusion restriction. Step 1: fit a model that contains all main effects resulting from school levels covariates, gender, race/ethnicity, and grade, and training; Step 2: set equal slopes on school levels covariates, gender, race/ethnicity, and grade for Earlyadopter group and Lateradopter group, then compare with the model from step 1 to test whether main effect s on school levels characteristics are same between the Earlyadopter groups and Lateradopter groups (referred as Model A in Table 15); Step 3: remove school levels covariates gender, race/ethnicity, and grade from the model in Step 2 one at a time, then comp are it with the Step 2 model to test whether the number of students who have been referr ed are associated with those covariates (referred as Model B in Table 15); Step 4: add the interaction effect between training and school level covariates to the model from Step 2, then compare it with the Step 2 model to test whether any interaction effect exists between school leve ls characteristics and training for the Earlyadopter groups and Lateradopter groups (referred as Model C in Table 15);
PAGE 98
87 Step 5: add the interaction effect between training and time to the model from Step 2 then compare it with the Step 2 model to test whether there are any training effect over time (referred to Model D in Table 15); Step 6: add other interested covariates su ch as percentage of training staffs of each school to the model from Step 5, then compare it with the Step 5 model to test whether those covariates are significant (referred as Model E in Table 15); Step 7: reach a final model that contains main effects of school levels covariates, gender, race/ethnicity, and grade with equal slopes for the Earlyadopter groups and the Lateradopter groups, interac tion effect between training an d time, and main effect of training. During this step, e qual intercepts between the Earlyadopter groups and the Lateradopter groups are tested to examine whether a baseline variation exists between different adopter classes (referr ed as Model F in Table 15). All comparison tests listed above are based on likelihood ratio tests. Repeat all those seven steps under weak exclusion rest riction which assume there may have a training effect among Lateradopter group. With in each step, the results will be compared parallelly for different types of exclusi on restriction assumption to examine which assumption is appropriate to the data.
PAGE 99
88Table 15. Model Selection Strategy Models Main Effect of Training Main Effects of School Level Covariates Interaction Effects between Individual Level Covariates and Training Other Main Effects Moderation Effects Effects of time Options in Model 3.1 Model A Gender, Race/Ethnicity Grade Equal slopes between Early/Lateradopter groups gxc = gxn Model B Gender, Race/Ethnicity Grade Between Early/Lateradopter groups x Model C Training with Gender, Race/Ethnicity Grade Between Early/Lateradopter groups zx Model D Between Early/Lateradopter groups Time period zt Model E Percentage of training Between Early/Lateradopter groups Model F Training Gender, Race/Ethnicity Grade Baseline different between Early/Lateradopter groups Time period gxc = gxn
PAGE 100
89 3.3 Analysis Mplus version 5.0 with TW OLEVEL, RANDOM, and MIXTURE analysis type was used to calculate the intervention causal effect. Due to the first training period being the longest period within the twoyear study, our current an alysis will examine the QPR training effect at the first pe riod and the entire study period separately. This also allows us to examine whether the QPR traini ng effect is persistent over time. 3.3.1 Analyses Limited to the First Study Period After m odel selection, the final mode l for the first study period only was constructed under the weak excl usion restriction assumption a nd contains main effects of gender, grade level, and race /ethnicity, and random effect in school by training time. The results show that there is a significant differen ce on baseline between the Earlyadopter groups and the Lateradopter groups, with intercept values of 10.497 (SE = 0.200) and 16.049 (SE = 0.858) (Table 16), respectively. The slopes of the SameDay Assessment on the main covariate effects are not signifi cantly different between the Earlyadopter groups and the Lateradopter groups. Table 16 shows that female students have a mean of SameDay Assessment rate less than 30% ( = 0.327, SE = 0.128), which is higher than male students, and the difference is significant ( p = 0.011). On average, the SameDay Assessment rate for middle schoo ls is higher than high schools. The 8th Grade has the highest SameDay Assessment rate ( = 0.445, SE = 0.198) within middle schools. Meanwhile, the SameDay Asse ssment rate increases with increase of grade level within middle schools. However, this association is opposite within high
PAGE 101
90 schools, where the 9th Grade has the highest Sam eDay Assessment rate ( = 0.247, SE = 0.255). The White has a higher SameDay Assessment rate than the other race/ethnicity categories, except for Multirace ( = 0.155, SE = 0.290). The SameDay Assessment rates for African American and Hi spanic are significantly different from the White ( p =0.004, p = 0.227). The mean intervention e ffect is 0.596 (SE = 0.215, 95% CI =(1.026, 0.166)) for the Earlyadopter group, which suggests that the SameDay Assessment rate in the training schools is al most 60% lower than the rate in the waitlisted schools. In contrast, the Lateradopter group has a mean intervention effect of 4.7 (SE = 0.852, 95% CI = (2.996, 6.404)), indica ting that the SameDay Assessment rate for the training schools in this group is almost 100 times higher than the rate in the waitlisted schools. The reason that the Lateradopter group has a highe r intervention effect than the Earlyadopter group may be due to the facts that their school systems and school staffs were well prepared to participat e in the training and a ll training instructors have received feedback from the early traini ng period. Finally, the re sults show that the SameDay Assessment rates are highly associat ed with school level characters such as gender, race/ethnicity, and grade level, a nd the training has different impact on the Earlyadopter group and the Lateradopter group. The following section will apply the Poisson regression model to the entire four time periods. The results will also be compar ed with two traditional methods, IntenttoTreat (ITT) and AsTreated (AT).
PAGE 102
91Table 16: Results of the Poisson Re gression Model under Weak Exclusion Restriction Assumption during the First Period of Study Comparison Estimate S.E. Est./S.E. PValue Gender Male vs Female 0.327 0.128 2.556 0.011 Grade 7th vs 6th 0.300 0.187 0.1603 0.109 8th vs 6th0.445 0.198 2.258 0.024 9th vs 6th0.247 0.255 0.972 0.331 10th vs 6th0.204 0.255 0.717 0.473 11th vs 6th0.317 0.268 1.183 0.237 12th vs 6th0.935 0.369 2.534 0.011 Race/Ethnicity Asian vs White 0.384 0.357 1.076 0.282 African American vs White0.501 0.174 2.873 0.004 Hispanic vs White0.492 0.222 2.218 0.027 Multirace vs White0.155 0.290 0.534 0.593 EarlyAdopter Group Training Status 0.596 0.215 2.774 0.006 Intercept10.479 0.200 52.432 < 0.0001 LaterAdopter Group Training Status 4.700 0.852 5.520 < 0.0001 Intercept16.049 0.858 18.714 < 0.0001
PAGE 103
92 3.3.2 Entire Study Period All of the an alyses that have been done so far are only applied to the first training period, which is the longest tr aining period within the twoye ar study. During this period, the schools in the waitlisted condition did not have a chance to participate in the training. The compliance status for the training schools can be defined by their first start time of training. However, the study design has been changed in order to increase the study power (Brown et al., 2006) a nd increase participation rate within each school; training time for each training period has been narrowed down. The 16 remaining schools have been randomly assigned to receive training at four different time blocks. Eventually, all 32 schools finished their training within thes e five time periods. The following analyses will focus on the first four time periods to evaluate the QPR Gatekeeper training program effect over time. The two traditional methods, ITT (Intenttotreat) Analysis and AT (Astreated) Analysis, were used first to examin e the intervention effect over time. Then, the advanced method (Principal Stratification method), was used under both strong exclusion restriction and weak exclusion restriction assumptions to compare with ITT and AT analyses to investigate whether we can gain more information from this advanced method and are able to better examine the intervention effect over time for this study.
PAGE 104
93 3.3.2.1 Summary of Analyses for All 32 Schools Comparison of all estimates obtained from different methods show that the ITT method underestimates on almost all paramete rs (Table 17). Th e estimators on gender give a consistent message that male students have 17% ( = 0.19, SE = 0.11) lower SameDay Assessment rates than female students by using different methods. The estimators on grade levels show that middle schools have a higher SameDay Assessment rate than high schools on average. The SameDay Assessment rates increase along with grade level in middle sc hool but they show an opposite direction in high schools. The 8th and 9th grade students have the highest SameDay Assessment rates among middle schools and high schools respectively. The multirace students have the highest SameDay Assessment rate am ong all race/ethnicity students. White students are the majority group in the school district. Their SameDay Assessment rate is the second to the highest. Compared to ITT analysis, AT analysis has a positive estimate on intervention effect which means that the QPR Gatekeep er training program increased the SameDay Assessment rate in the training group compared to the waitlisted group. Contrarily, the resu lts from the Principal Stratification method show that the intervention has a positive im pact on Lateradopter groups rather than Earlyadopter groups, regardless of the ITT or AT training status. Time has a strong positive impact on both Lateradopter groups and Earlyadopte r groups. Period 2 has the highest SameDay Assessment rate. The Principal Stratification method w ith AT status under weak exclusion restriction has the best fit due to the smallest value of BIC. The results show that female
PAGE 105
94 students have a mean SameDay Assessment rate that is 17% higher than do male students ( = 0.19, SE = 0.11). By grade, the 8th grade students have the highest referral rate ( = 0.44, SE = 0.17). Among different race/ethnicity groups, the Multirace students have the highest referral rate ( = 0.17, SE = 0.25), and Hispanic students have the lowest ( = 0.46, SE = 0.19). The mean intervention effect is 0.52 (SE = 0.25, 95% CI = (0.82, 0.02)) (Table 18) for the Earlyadopter gr oup, which suggests that the SameDay Assessment rate in the training sch ools is almost 60% lower than the rate in the waitlisted schools. However, the Lat eradopter group has a mean intervention effect of 0.67 (SE = 0.31, 95% CI = (0. 05, 1.29)), indicating that the SameDay Assessment rate for the training schools in this group is almost 2 times higher than the rate in the waitlisted schools.
PAGE 106
95Table 17: Estimate Comparison fo r All 32 Schools over Four Periods Method PS(ITT) PS(AT) ITT AT Strong Weak Strong Weak Comparison Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Gender Male vs Female 0.47 (0.10)*** 0.19 (0.11) 0.19(0.11) 0.19 (0.11) 0.19 (0.11) 0.19 (0.11) Grade 7th vs 6th 0.63 (0.15)*** 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 8th vs 6th 0.43 (0.15)* 0.44 (0.17)* 0.44 (0.17)* 0.44 (0.17)* 0.44 (0.17)* 0.44 (0.17)* 9th vs 6th 0.64 (0.18)*** 0.17 (0.20) 0.24 (0.22) 0.23 (0.21) 0.24 (0.21) 0.24 (0.21) 10th vs 6th 0.90 (0.20)*** 0.15 (0.22) 0.10 (0 .23) 0.10 (0.23) 0.10 (0.23) 0.08 (0.23) 11th vs 6th 0.96 (0.18)*** 0.27 (0.22) 0.21 (0 .23) 0.21 (0.23) 0.21 (0.22) 0.20 (0.22) 12th vs 6th 1.14 (0.18)*** 0.82 (0.28)* 0.75 (0.29)* 0.76(0.29)* 0.75 (0.29)* 0.75 (0.29)* Race/Ethnicity Asian vs White 0.68 (0.35)** 0.26 (0.30) 0.26 (0.30) 0.26 (0.30) 0.26 (0.30) 0.27(0.30) African American vs White 0.65 (0.14)*** 0.35 (0.14)* 0.34 (0.14)* 0.34 (0.15)* 0.34 (0.14)** 0.36 (0.14)* Hispanic vs White 0.75 (0.20)*** 0.44 (0.19)** 0.43 (0.19)** 0.44 (0.19)**0.44 (0.19)** 0.46 (0.19)** Multirace vs White 0.63 (0.42) 0.17 (0.25) 0.17 (0.25) 0.17 (0.25) 0.17 (0.26) 0.17 (0.25) EarlyAdopter Group Intercept 11.02 (0.28)*** 14.72 (4.85)*** 10.88 (0.21)*** 10.73 (0.19)* LaterAdopter Group Intercept 11.42 (0.25)*** 10.66 (0.19)*** 11.49 (0.23)*** 11.89 (0.29)*** pvalue < 0.01; ** pvalue < 0.05; *** pvalue < 0.0001
PAGE 107
96Table 18: Effect Estimates for all 32 Schools over Four Periods pvalue < 0.01 ** pvalue < 0.05 *** pvalue < 0.0001 Effect Estimate (SE) Method EarlyAdopter group LaterAdopter group Intenttotreat (ITT) 0.04 (0.13) Astreated (AT) 0.02 (0.17) Principal Stratification (ITT) Strong Exclusion Restriction 0.09 (0.30) 0 Weak Exclusion Restriction3.61 (4.84) 0.77 (0.23)** Principal Stratification (AT) Strong Exclusion Restriction 0.35 (0.25) 0 Weak Exclusion Restriction0.38 (0.23) 0.69 (0.32)**
PAGE 108
97 3.3.2.2 Summary of Analyses for 20 Middle Schools The same logic described above has been applied to test the training effect on middle schools only. The results show that the SameDay Assessment rates in middle schools have similar slopes for both gender and race/ethnicity between EarlyAdopter groups and LaterAdopter groups. The re sults also show that the SameDay Assessment rates are highly associated with race/ethnicity and grade level but not with gender. The weak exclusion restriction assumption has a better f it than the strong exclusion restriction assumption with ITT training status, but not with AT training status. ITT training status has a lower BI C value than AT training status. Overall, the Poisson regression model w ith ITT training status under the weak exclusion restriction shows that male students haves a 16% ( = 0.18, SE = 0.15) lower rate than female students (Table 19). The SameDay Assessment rates increase as students get older. The 8th grade has the highest SameDay Assessment rate within middle schools which is 54% higher ( = 0.43, SE = 0.17) than 6th grade and 25% higher than 7th grade. All race/ethnicity categorie s have a lower SameDay Assessment rate compared to Whites. The SameDay Assessment rates for African American ( = 0.84, SE = 0.18) and Hispanic students ( = 0.48, SE = 0.23) are significantly different from the Whites. African American student s have the lowest rates among middle school students. Their rates are less than half of t hose of White students. The overall intervention has a strong positive effect on the Earlyadopt er group and a negative effect on the Lateradopter group. The SameDay Assessmen t rate has been increased more than
PAGE 109
98 2.7 times ( = 1.02, SE = 0.50) (Table 20) in the training schools than waitlisted schools for Earlyadopter groups. However, the SameDay Assessment rate has been decreased by 56% ( = 0.82, SE = 0.38) in the Lateradopter training schools. The intervention effect persist over time
PAGE 110
99Table 19: Estimate Comparison for 20 Middle Schools over Four periods Method PS(ITT) PS(AT) ITT AT Strong Weak Strong Weak Comparison Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Gender Male vs Female 0.18 (0.15) 0.18 (0.15) 0.18 (0.15) 0.18 (0.15) 0.18 (0.15) 0.18 (0.15) Grade 7th vs 6th 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 0.25 (0.17) 8th vs 6th 0.43 (0.17)* 0.43 (0.17)* 0.43 (0.17)* 0.43 (0.17)* 0.43 (0.17)* 0.43 (0.17)** Race/Ethnicity Asian vs White 0.59 (0.44) 0.60 (0.44) 0.59 (0.44) 0.60 (0.44) 0.60 (0.44) 0.60 (0.44) African American vs White 0.84 (0.18)*** 0.81 (0.18)*** 0.84 (0.18)*** 0.84 (0.18)*** 0.81 (0.18)*** 0.81 (0.18)*** Hispanic vs White 0.48 (0.23)** 0.47 (0.23)** 0.48 (0.23)** 0.48 (0.23)** 0.47 (0.23)** 0.46 (0.23)** Multirace vs White 0.02 (0.33) 0.02 (0.33) 0.02 (0.33) 0.02 (0.33) 0.02 (0.33) 0.02 (0.33) EarlyAdopter Group Intercept 11.50 (0.39)*** 13.45 (2.12)*** 11.01 (0.23)*** 11.01 (0.22) LaterAdopter Group Intercept 10.98 (0.30)*** 10.13 (0.23)*** 11.17 (0.285)*** 11.55 (0.43)*** pvalue < 0.01 ** pvalue < 0.05 *** pvalue < 0.0001
PAGE 111
100Table 20: Effect Estimates for 20 Middle Schools over Four Periods pvalue < 0.01 ** pvalue < 0.05 *** pvalue < 0.0001 Effect Estimate (SE) Method EarlyAdopter group LaterAdopter group Intenttotreat (ITT) 0.35 (0.21)* Astreated (AT) 0.06 (0.21) Principal Stratification (ITT) Strong Exclusion Restriction 0.57 (0.39) 0 Weak Exclusion Restriction1.02 (0.50)* 0.82 (0.38)* Principal Stratification (AT) Strong Exclusion Restriction 0.04 (0.27) 0 Weak Exclusion Restriction0.14 (0.28) 0.58 (0.56)
PAGE 112
101 3.3.2.3 Summary of Analyses for 12 High Schools Overall, the Poisson regression model with ITT training status under weak exclusion restriction shows that female students have more than 20% higher ( = 0.22, SE = 0.17) SameDay Assessment rate than male students in the high schools (Table 21). The 12th grade has the lowest SameDay Assessment rate ( = 0.97, SE = 0.25) among students. The highest rate among race/e thnicity groups in high schools was found for American Indian students ( = 0.95, SE = 1.00). African American students have a consistently higher rate than White students in both EarlyAdopter group and LaterAdopter group. Similar to middle schools, Hispanic students have a consistently lower rate than White students in both EarlyA dopter group and Later Adopter group. The overall intervention has a negative effect on the EarlyAdopter group but a positive effect on the Lateradopter group. The Earlyadopter training schools show a 18% decrease in the SameDay Assessment ra te compared to the waitlisted schools ( = 1.73, SE = 0.51) (Table 22); while the Lat eradopter training schools have a SameDay Assessment rate 3 times of th e Lateradopter training schools ( = 1.23, SE = 1.57) (Table 22). The impact of the interv ention increases over time in both the EarlyAdopter group and th e LaterAdopter group.
PAGE 113
102Table 21: Estimate Comparison fo r 12 High Schools over Four Periods Method PS(ITT) PS(AT) ITT AT Strong Weak Strong Weak Comparison Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Estimate (SE) Gender Male vs Female 0.29 (0.15) 0.20 (0.15) 0.20 (0.162) 0.22 (0.167) 0.20 (0.16) 0.21 (0.17) Grade 10th vs 9th 0.57(0.18)* 0.81 (0.21)*** 0.32 (0.20) 0.30 (0.20) 0.31(0.20) 0.30(0.20) 11th vs 9th 0.73 (0.19)*** 1.02 (0.23)*** 0.43 (0.21)** 0.42 (0.21)** 0.44 (0.21)** 0.42 (0.21)** 12th vs 9th 1.39 (0.28)*** 1.58 (0.31)*** 0.98 (0.25)*** 0.97 (0.25)*** 0.96 (0.25)*** 0.96 (0.25)*** Race/Ethnicity American Indian vs White 0.94 (5.07) 0.50 (2.94) 0.92 (0.99) 0.95 (1.00) 0.95 (0.99) 0.98 (1.00) Asian vs White 2.94 (8.26) 1.99 (2.19) 0.07 (0.40) 0.10 (0.41) 0.12 (0.40) 0.09 (0.40) Africa American vs White 0.09 (0.17) 0.41 (0.23) 0.13 (0.21) 0.19 (0.21) 0.196 (0.215) 0.19 (0.21) Hispanic vs White 0.98 (0.42)* 1.01 (0.38)** 0. 39 (0.34) 0.45 (0.33) 0.44 (0.33) 0.46 (0.33) Multirace vs White 0.67 (0.87) 0.88 (0.97) 0.37 (0.39) 0.41 (0.40) 0.43 (0.40) 0.44 (0.40) EarlyAdopter Group Intercept 10.49 (0.17) 10.45 (0.16) 10.17 (0.44)*** 9.35 (0.30)*** 10.31 (0.43)*** 9.77 (0.33)*** LaterAdopter Group Intercept 11.84 (0.31)*** 12.80 (1.58)*** 11.51 (0.23)*** 11.84 (0.31)*** pvalue < 0.01; ** pvalue < 0.05; *** pvalue < 0.0001
PAGE 114
103 Table 22: Effect Estimates for 12 High Schools over Four Periods pvalue < 0.01 ** pvalue < 0.05 *** pvalue < 0.0001 Effect Estimate (SE) Method EarlyAdopter group LaterAdopter group Intenttotreat (ITT) 0.23 (0.19) Astreated (AT) 0.003 (0.23) Principal Stratification (ITT) Strong Exclusion Restriction 0.80 (0.35)** 0 Weak Exclusion Restriction1.73 (0.51)* 1.23 (1.57) Principal Stratification (AT) Strong Exclusion Restriction 0.72 (0.37)** 0 Weak Exclusion Restriction1.34 (0.57)** 0.63 (0.38)
PAGE 115
104 3.4 Conclusion Overall, the analyses show that the P r incipal Stratification method with a weak exclusion restriction assumption is the best model because of the nature of the study design, variation in starting ti me of school training, and the characteristics of schools (e.g. whether willing to receive the training earli er or later). Moreover, the training had a different impact on EarlyAdopter and Late radopter schools. The intervention effect is also different for middle and high schools. The training had a stronglong term positive impact on SameDay Assessment rates. The SameDay Assessment rate is also highly associated with school le vel characteristics. However, there are still limitations for this study. First, the outcom e, the number of SameDay Assessments from schools, was rare. Secondly, limited information was collect ed from student level which was used to predict the adopter class fo r the waitlisted schools.
PAGE 116
105 Chapter Four Conclusion This study has extended current causal inference procedures from singlelevel principal stratification to tw olevel stratification in random ized trials. It applied the developed models on a twolevel randomized trial in which stratification is determined at two levels to evaluate the causal effect of the intervention. This chapter will summarize the new development arising from this study, the contribution of this study in both methodology and application, the limitations, and the research goals in the future. 4.1 Methodological Contributions Most previous appli catio ns of Principal Stratific ation have focused on studies where participation or compliance status is determined by a single posttreatment variable, even though randomization can be at either group or individual leve l (Vinokur et al., 1995) (McDonald et al., 1992). Moreover analyses of the causal effects have been done at the individual level even in the case of mu ltilevel randomized trials such as the flu shot study (McDonald et al., 1992). Such an approach has ignored the group level participation status. In this dissertati on, methods were developed for multilevel randomized trials where principal stratific ation membership may be determined by multilevel posttreatment variables. The following summarizes the main contributions of this dissertation.
PAGE 117
106 The new method defines multidimensiona l (2) posttreatment variable (S) which allows us to combine the part icipation status from both the group level and the individual level and to list all possible principal strata. Several possible models that can be applied to twolevel randomized trials where particip ation can be at the individual level only, group level only, or both indi vidual and group levels have been discussed. The new method modifies assumptions that underl y general causal inference for different situations in principal stratification. Moreove r, it discusses how to obtain a moment or mixture and marginal likelihood estimate of causal effect for all possible models by listing relative equations and maximum likeli hood functions. A very useful feature of mixture and marginal likelihood estimation is th at it takes account of covariate effects. Understanding the complex influence of c ovariates is essential to understanding intervention mechanisms because those cova riates may confound the intervention. In general, including covariates that are good pr edictors of complian ce increases precision in estimation of compliance status, increases the power to detect complier average causal effect (CACE), decreases sensitivity if CACE estimates to violation of underlying assumptions, and increases identifiability of CACE when critical indentifying assumptions are relaxed (Booil, 2008). Finally, the idea of defining th e posttreatment variable (S) to multiple dimensions permits the method to be applie d to multilevel randomized trials with more than two intervention arms or with more than two levels of randomization. For example, study subjects can be classified into more than 16 principal strata because higher dimensional posttreatment outcome can be created.
PAGE 118
107 4.2 Limitations However, this study did not consider the im pact of intraclass correlation (ICC) on variance inflation in the estimation of CACE in multilevel randomized trials because it has been discussed in other literature (Booil, 2008). Power issues have not been explicitly discussed in this study, even though power is a great concern in planning randomized trials because sample size will be restri cted due to the limitation of group level randomized trials (Booil, 2008). 4.3 Application of Findings The prim ary example used in this diss ertation, the Georgi a Gatekeeper Study, started with a classic randomized design. Howe ver, due to variation in implementation of groups selected to participate in the intervention, the study design switched to the dynamic waitlist design which implied varying e xposure level to schools that adopted the intervention at different times. This variation invited the application of the Principal Stratification method. This variation in expos ure could be caused by the organization of the school system or the attitude or w illingness of school administration regarding participation in the intervention. There was evidence of baseline variation between the Earlyadopter group and the Lateradopter group among all 32 middle and high schools. Similarly, the baseline variations between the E arlyadopter group and the Lateradopter group also existed in both middle schools and high schools when analyses were done separately.
PAGE 119
108 As to the outcome of QPR (Question, Persuade, Refer) (Quinnett, 1995) services, females were 17% more likely to be referred than males. Middle schools had higher rates of referral than high school on average. Seventh, 8th, and 9th grades had higher rates of referral then 6th, 11th and 12th grades did. Multirace and White students were more likely to be referred than Americans Indians. Females had higher referral rates than males in both middle schools and high schools. The referral rates increased in middle schools but decreased in high schools by grade level. In this example, schools were broken down into a smaller unit within each training period in order to ma intain the training schedule. 4.4 Further Discussion Evaluation o f the intervention effect is the objective of clinical trials. However, compliance is an important issue existing in prevention trials and ot her research studies. Evaluation of causal effect, taking account of noncompliance, is what Principal Stratification method aims to achieve. Appl ying the Principal Stra tification method to multilevel randomized trials is current a nd important among researchers because it controls selection bias. The Principal St ratification method can be applied in any scientific fields which are able to conduct randomized trials, especially, in social science and mental health because group level randomized trials are common designs used in mental health prevention studies or co mmunity social behavior studies. There are some alternative methods whic h address noncompliance issues such as astreated analysis and perp rotocol (PP) analysis. However, the randomization may be
PAGE 120
109 broken by using these two analyses, because astreated analysis is the case where the subjects in the control cond ition are not measured for participation and the astreated analysis may compare the participants in the intervention condition with a nonintervention group, which combines the nonparticipants in the intervention condition with all subjects in the cont rol condition. The PP analysis focuses on the effect of compliance to the assigned treatment pr otocol (Ten Have, et al., 2008). When participation can be measured in the c ontrol condition, a PP analysis may compare participation in the intervention condition with the participation in the control condition. The exclusion of nonparticip ation under the PP approach distinguishes it from the astreated method. In the case where the control condition is not measured for participation, the PP analysis may contrast the participat ion in the interventi on condition with all subjects in the control cond ition, excluding the nonpartic ipants in the intervention condition. Principal Stratification method and propensity score method are two other methods used to reduce selection bias by equating groups based on a set of known covariates. They both estimate interventi on causal effect by comparing potential outcomes under control and intervention conditi on and reduce overt bias which may be attributable to observed confounders by adjust ing these covariates (Ten Have, et al., 2008). The propensity score method was intr oduced by Rosenbaum and Rubin (1983). It is defined as the conditional probability of intervention given background variables. However, the m ain difference between these two methods is that the Principal Stratification method is also able to adjust for hidden bias which is unobserved
PAGE 121
110 confounders such as compliance behaviors ba sed on randomization, so it relies on having some "instrument" (what subjects were ra ndomized) that affects the intervention level that subjects receive. The propensity score method does not assume anything was randomized, but instead relies on an assump tion of unconfounded treatment assignment: it assumes that there is no hidden bias betw een the intervention and control conditions, and it assumes that only observed variables a ffect the intervention level that subjects receive (Angrist, et al., 1996; Rubin, 2001; Posner, et al ., 2001; Landrum & Ayanian, 2001). Therefore, the Principal Stratification method can help better identify meaningful relationships between treatment complianc e and noncompliance with respect to the effect of treatment on outcome. The follo wing table (Table 23) summarizes the comparison of these alternative methods. Further improvement on the modified Prin cipal Stratification method presented in this research is still warranted due to the li mitations discussed previously. Future studies should focus on how to make the Principal St ratification method more flexible for use on multilevel randomized trials due to variations of participation status at different levels, complex memberships of principal strata, and requirements of more sophisticated models. To apply the Principal Stratifi cation method to longitudinal tr ials where the participation status of an individual subject or group may change over time further development of the method would also be needed. Future studies sh ould also address the issues of power and power calculation requirements of the Principal Stratificati on method, as currently there are no theoretical and practical bases for su ch exercises. Expansion of the Principal Stratification method to other types of analyses (e.g., survival analysis) and other
PAGE 122
111 research fields (e.g., pharmaceutical) may also draw extensive interests. It may also be attractive to seek alternative methods to control the selection bias when designing similar community trials such as those studies described in this research. We do believe that the Principal Stratification method ha s the potential to be applied to other relevant research fields and cover more situations and is thus a viable method, if appropriate development is conducted.
PAGE 123
112Table 23. Comparison of Alternative Methods Methods Strength Limitation Intenttotreat (ITT) Includes all randomized subjects Does not control selection bias AsTreated (AT) Considers noncompliance Broken randomization PerProtocol (PP) Considers noncompliance Broken randomization Propensity Score Reduces selection bias Control observed confounders Does not require randomization Principal Stratification Reduces selection bias Controls observed and unobserved confounders Needs additional assumptions to limit the numbers of principal strata
PAGE 124
113 References Angrist, J. D ., Imbens, G. W., & Rubin, Donald B. (1996). Identification of causal effects using instrumental variables Journal of the American St atistical Association, Barnard, J., Frangakis, Constantine E ., Hill, J., & Rubin, Donald B. (2003). Principal Stratificati on Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City (with discussion) Journal of the American Statistical Association. Brown, C. Hendricks, Liao, Jason (1999). Principles for Designing Randomized Preventive Trials in Mental Health: An Emerging Developmental Epidemiology Paradigm American Journal of Community Psychology. Brown, C. Hendricks, Wyman. Peter A, Guo. Jing, Pea. Juan. (2006). Dynamic waitlisted designs for randomized trials: ne w designs for prevention of youth suicide. Clinical Trials. Bloom, Howard S. (1984). Accounting For NoShows In Experimental Evaluation Designs Evaluation Review. Cox, D.R.(1958). The Planning of Experiments John Wiley and Sons, New York. Frangakis, Constantine E., Rubin, Donald B. (2002). Principle Stratification in Causal Inference Biometrics.
PAGE 125
114 Frangakis, Constantine E., Rubin, Donald B., Zhou, X. H. (2002). Clustered encouragement design with individual noncompliance: Bayesian inference and application to advance directive forms. Biostatistics. Grilli L., Mealli F. (2008). Nonparametric Bounds on the Causal Effect of University Studies on Job Opportuni ties Using Principal Stratification. Journal of Educational and Behavioral Statistics. Have, Thomas R. Ten., Normand, SharonLise T., Marcus, Sue M., Brown, C. Hendricks, Lavori, Philip., and Duan, Naihua (2008). Intenttotreat vs. Nonintenttotreat Analyses under TreatmentNonadherence in Mental Health Randomized Trials. Psychiatric Annals. Hirano, Keisuke, Imbens, Guido W., Rubi n, Donald B., Zhou, XiaoHua (2000). Assessing the Effect of an Influenza Vaccine in an Encouragement Design Biostatistics. Hong, G. & Raudenbush, S. W. (2006). Evaluating kindergarten retention policy: A case study of causal inferenc e for multilevel observational data. Journal of the American Statistical Association Ialongo, Nicholas S., Werthamer, Lisa, Kellam, Sheppard G., Brown, C. Hendricks, Wang, Song B., & Lin, Yu H. (1999). Proximal impact of two firstgrade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599641. Jo, Booil (2002). Estimation of Intervention Effects with Noncompliance: Alternative Model Specification Journal of Educational a nd Behavioral Statistics.
PAGE 126
115 Jo, Booil, Asparouhov, Tihonmir, Muthen. Bengt O. (2008). IntentiontoTreat Analysis in Cluster Randomized Trials with Noncompliance. Statistics in Medicine. Jo, Booil, Asparouhov, Tihonmir, Muthen. Bengt O., Ialongo, Nicholas S., Brown, C. Hendricks. (2008). Cluster Randomized Trials W ith Treatment Noncompliance Psychological Method. Jo, Booil. (2008). Causal Inference in Randomized Experiments With MediationalProcesses Psychological Method. Landrum, M.B. and Ayanian, J.Z. (2001). Causal effect of amulatory specialty care on mortality following myocardial infarc tion: A comparison of propensity score and instrumental variable analyses Health Services and Outc omes Research Methodology Posner, M.A., Ash, A.S., Freund, K.M., Moskowitz, M.A., and Shwartz, M. (2001). Comparing standard regression, propensity score matching, and instrumental variables methods for determining the infl uence of mammography on stage of diagnosis. Health Services and Outcomes Research Methodology. Quinnett, P. (1995). QPR: Ask a question, save a life Spokane, WA: QPR Institute and Suicide Awareness/Voices of Education. Rosenbaum, Paul. R., & Rubin, Donald B. (1983a). Assessing sensitivity to an unobserved binary covariate in an obse rvational study with binary outcome Journal of the Royal Statistical Society, Series B (Methodological). Rosenbaum, Paul. R., & Rubin, Donald B. (1983b). The central role of the propensity score in observationa l studies for causal effects Biometrika.
PAGE 127
116 Rubin, Donald B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology. Rubin, Donald B. (1978). BayesianInference for Caus al Effects Role of Randomization Annals of Statistics. Rubin, Donald B. (1978). Randomization Analysis of Experimental Data the Fisher Randomization Test Comment. Journal of the American Statistical Association. Rubin, D.B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation Health Services and Outcomes Research Methodology. Stuart, Elizabeth A. & Green, Kerry M. (2008). Using Full Matching to Estimate Causal Effects in Nonexperimental Studi es: Examining the Relationship Between Adolescent Marijuana Use and Adult Outcomes Developmental Psychology. Sobel, Michael E. (2006). What Do Randomized Studies of Housing Mobility Demonstrate?: Causal Inference in the Face of Interference Journal of the American Statistical Association. Vinokur, Amiram D., Price, Richar d H., and Schul, Yaacov. (1995). Impact of IOBS Intervention on Unemployed Wor kers Varying in Risk for Depression American Journal of Community Psychology. Wyman, Peter A., Brown, C. Hendricks Inman, Jeff, Cross, Wendi, SchmeelkCone, Karen, Guo, Jing, Pena, Juan B.(2008). Randomized Trial of a Gatekeeper Program for Suicide Prevention: 1Ye ar Impact on Secondary School Staff. Journal of Consulting and Clinical Psychology.
PAGE 128
117 Appendices
PAGE 129
118 Appendix A: Compute a Chisquare Difference Test Following are the s teps needed to compute a chisquare difference test based on loglikelihood values and scaling correction f actors obtained with the MLR estimator. 1. Estimate the nested and comparison models using MLR. The printout gives loglikelihood values L0 and L1 for the H0 a nd H1 models, respectively, as well as scaling correction factors c0 and c1 for the H0 and H1 models, respectively. For example, L0 = 2,606, c0 = 1.450 with 39 parameters (p0 = 39) L1 = 2,583, c1 = 1.546 with 47 parameters (p1 = 47) 2. Compute the difference test scaling correction where p0 is the number of parameters in the nested model and p1 is the number of parameters in the comparison model. cd = (p0 c0 p1*c1)/(p0 p1) = (39*1.450 47*1.546)/(39 47) = 2.014 3. Compute the chisquare difference test (TRd) as follows: TRd = 2*(L0 L1)/cd = 2*(2606 + 2583)/2.014 = 22.840 4. Splus Code: ChisqTest
PAGE 130
119 Appendix B: Mplus Code ITT Model 01: M ain effects within first training period TITLE: ITTModel01_median_time; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 131
120 USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE ( Period EQ 1 ); using period 1 data only ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#2% !early training group CPI on Male G7 G12 R2 R6 LPopDur @1; %BETWEEN%
PAGE 132
121 %OVERALL% CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 133
122 Appendix C: Mplus Code ITT Model 02: Ma in ef fects with equal slops within first training period TITLE: ITTModel02_median_time_slops_eq; DATA: FILE IS J:\data\cpi.numericB.txt; t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 134
123 USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE ( Period EQ 1 ); using period 1 data only ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 LPopDur @1; CPI on Male (M1); define the slops on gender are same between two !groups CPI on R2 (b7); defi ne the slops on race/ethnict y are same between two groups CPI on R3 (b8); CPI on R4 (b9);
PAGE 135
124 CPI on R5 (b10); CPI on R6 (b11); CPI on G7 (c7); define the slops on grade level are same between two groups CPI on G8 (c8); CPI on G9 (c9); CPI on G10 (c10); CPI on G11 (c11); CPI on G12 (c12); %c1#2% !early training group CPI on Male G7 G12 R2 R6 LPopDur @1; CPI on Male (M1); define the slops on gender are same between two groups CPI on R2 (b7); define the slops on race/ethnicty are same between two !groups CPI on R3 (b8); CPI on R4 (b9); CPI on R5 (b10); CPI on R6 (b11); CPI on G7 (c7); define the slops on geade level are same between two groups CPI on G8 (c8); CPI on G9 (c9); CPI on G10 (c10); CPI on G11 (c11); CPI on G12 (c12); %BETWEEN% %OVERALL% CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 136
125 Appendix D: Mplus Code ITT Model 09: Main ef fects with interactions within first training period TITLE: ITTModel02_median_time_slops_eq; DATA: FILE IS J:\data\cpi.numericB.txt; t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 137
126 USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2 SM SR2SR6 SG7SG12 t1 t2; LPopDur = log (population*durati on of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE ( Period EQ 1 ); using period 1 data only ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 SM SR2SR6 SG7SG12 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 SM SR2SR6 SG7SG12 LPopDur @1; CPI on SM (M); CPI on SR2 (R2); CPI on SR3 (R3); CPI on SR4 (R4); CPI on SR5 (R5); CPI on SR6 (R6);
PAGE 138
127 CPI on SG7 (G7); CPI on SG8 (G8); CPI on SG9 (G9); CPI on SG10 (G10); CPI on SG11 (G11); CPI on SG12 (G12); %c1#2% CPI on Male G7 G12 R2 R6 SM SR2SR6 SG7SG12 LPopDur @1; CPI on SM (M); CPI on SR2 (R2); CPI on SR3 (R3); CPI on SR4 (R4); CPI on SR5 (R5); CPI on SR6 (R6); CPI on SG7 (G7); CPI on SG8 (G8); CPI on SG9 (G9); CPI on SG10 (G10); CPI on SG11 (G11); CPI on SG12 (G12); %BETWEEN% %OVERALL% CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 139
128 Appendix E: Mplus Code ITT Model 13: Main ef fects within first training period with weaken condition TITLE: ITTModel13_median_time_weaken; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST
PAGE 140
129 TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA; USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE ( Period EQ 1 ); using period 1 data only ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#2% !early training group CPI on Male G7 G12 R2 R6 LPopDur @1;
PAGE 141
130 %BETWEEN% %OVERALL% CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S; weaken assumption: training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 142
131 Appendix F: Mplus Code AS Model 28: Main effects within first training period with weaken con dition under Astreat status TITLE: ITTModel28_as_median_time; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group AS = 1; if NotYet eq 1 then AS = 0; !create a Astreated training status G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = AS *M; AS gender SG7 = AS *G7; AS grade SG8 = AS *G8; SG9 = AS *G9; SG10 = AS *G10; SG11 = AS *G11; SG12 = AS *G12; SR2 = AS *R2; AS race/ethnicity SR3 = AS *R3; SR4 = AS *R4; SR5 = AS *R5; SR6 = AS *R6;
PAGE 143
132 VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA; USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE ( Period EQ 1 ); using period 1 data only ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#2% !early training group
PAGE 144
133 CPI on Male G7 G12 R2 R6 LPopDur @1; %BETWEEN% %OVERALL% CPI on AS; c1#1 on AS @0; class level is in dependent with training status %c1#1% CPI on AS @0; strong assu mption: no training effect is !allowed %c1#2% CPI on AS; only testi ng training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 145
134 Appendix G: Mplus Code ITT Model 88: m edian time 4period TITLE: ITTModel88_median_time_4period; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 146
135 USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G12 R2 R6 LPopDur @1; %c1#2% !early training group CPI on Male G7 G12 R2 R6 LPopDur @1; %BETWEEN% %OVERALL% CPI on S;
PAGE 147
136 c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 148
137 Appendix H: Mplus Code ITT Model 132: A S main effects 4period ms TITLE: ITTModel132_AS_m ain_effects_4period_ms; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 149
138 USEVARIABLES ARE CPI Stat us ST Male LPopDur G7 G8 R3 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G7 G8 R3 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE (Middle EQ 1 AND RACE NE 1); ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G7 G8 R3 R6 LPopDur @1; %c1#1% later training group CPI on Male G7 G8 R3 R6 LPopDur @1; %c1#2% !early training group CPI on Male G7 G8 R3 R6 LPopDur @1; %BETWEEN% %OVERALL%
PAGE 150
139 CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 151
140 Appendix I: Mplus Code ITT Mode l 176: AS m ain effects 4period hs TITLE: ITTModel176_AS_main_effects_4period_hs; DATA: FILE IS J:\data\cpi.numericB.txt; Define: t1 =1; t2 =1; if Tcom eq 0 then t2=0; early training group if Tcom eq 1 then t1=0; later training group G7 = ( Grade EQ 7); create dummy variables for grade level G8 = ( Grade EQ 8); G9 = ( Grade EQ 9); G10 = ( Grade EQ 10); G11 = ( Grade EQ 11 ); G12 = (Grade EQ 12); R2 = ( Race EQ 1); create dummy variables for race/ethnicity R3 = ( Race EQ 2); R4 = ( Race EQ 3); R5 = ( Race EQ 4); R6 = ( Race EQ 5); create dummy variables for interaction terms SM = S*M; status gender SG7 = S*G7; status grade SG8 = S*G8; SG9 = S*G9; SG10 = S*G10; SG11 = S*G11; SG12 = S*G12; SR2 = S*R2; status race/ethnicity SR3 = S*R3; SR4 = S*R4; SR5 = S*R5; SR6 = S*R6; VARIABLE: NAMES ARE School Male Grade Race Period Status Middle HiLevel Duration Timing TimeInt CPI LPopDur PopDur ST TimSchTr com Tcom Perc First NotYet TNA TA AFirst ANotYet ATNA ATA;
PAGE 152
141 USEVARIABLES ARE CPI Status ST Male LPopDur G10 G12 R2 R6 t1 t2; LPopDur = log (p opulation*duration of training period ST = School*Time sequence COUNT is CPI; means CPI is a count data and follow a Poisson distribution CLUSTER = ST; !random effect for School timing sequence WITHIN = Male G10 G12 R2 R6 LPopDur ; main effect for Gender, Grade, and Race/ethnicity, offset = log (population duration) BETWEEN = Status; Status = training status Class = c1(2); two levels TRAINING =t1 t2 ; later trai ning group will have a similar !distribution as early training group Missing is ALL (999); USEOBS ARE (Middle EQ 0); ANALYSIS: TYPE = TWOLEVEL random MIXTURE; MODEL: %WITHIN% %OVERALL% CPI on Male G10 G12 R2 R6 LPopDur @1; %c1#1% later training group CPI on Male G10 G12 R2 R6 LPopDur @1; %c1#2% !early training group CPI on Male G10 G12 R2 R6 LPopDur @1; %BETWEEN%
PAGE 153
142 %OVERALL% CPI on S; c1#1 on s @0; class level is independent with training status %c1#1% CPI on S @0; strong assumption: no training effect is !allowed %c1#2% CPI on S; only test ing training effect on early training !group OUTPUT: TECH1 TECH2;
PAGE 154
About the Author Jing Guo originally came from Beijing, P. R. China. She received a Bachelors Degree in Applied Electronic Technology from Beijing Union University, Beijing, China and a M.S. in Statistics from University of Georgia, Athens, U.S. She entered the Ph.D. program at University of South Florida in 2004. While in the Ph.D. program at the Univers ity of South Florida, Ms. Guo also work as a research staff at the Prevention Science and Methodolog y Group and a statistical data analyst at Policy and Services Research Data Center, Department of Mental Health Law & Policy, Louis de la Parte Florida Mental H ealth Institute, University of South Florida. She, as a biostatistician, has participated in many clinical trials and administrative studies, and provided critical st atistical support.
