Randomized Controlled Trials (RCTs): Further Explanation
The non-experimental evaluation techniques of Covariate Regression Adjustment and Heckman Selection Bias Correction have notable drawbacks for generating valid and reliable results. Covariate Regression Adjustment can control only for observable characteristics, while in reality, motivation and other hard-to-observe characteristics may play big roles in group outcome differences. Heckman Selection Bias Corrections depend on strong parametric assumptions about error terms and on availability of suitable Z variables .
RCTs avoid these problems by using randomization to produce groups that are expected to be balanced on all outcome-relevant characteristics, both observed and unobserved. A well-run RCT for program evaluation uses recruitment strategies such as advertising to generate excess demand for available program slots, and then employs randomization as a fair way to allocate scarce places in the program. In some contexts, such as children’s enrollment is magnet schools, randomization is employed routinely whether or not an evaluation is underway.
To ensure that treatment and control groups are balanced before any treatment takes place, it is essential for an RCT to collect baseline data on characteristics that could offer alternative explanations for treatment-control differences. Passing statistical tests of the hypothesis of no pre-existing differences in observed characteristics offers assurance that there are no pre-existing differences in relevant unobserved characteristics either.
If there are slight differences in pre-existing characteristics, the technique of Covariate Regression Adjustment can address them. When used with RCT data, the amounts of regression adjustment needed to balance treatment and control groups ordinarily is minuscule compared to the amounts of adjustment required by non-experimental data.
The term “ITT impact” is used to denote a treatment-control difference in average outcomes in RCT data. “ITT” stands for “Intent To Treat.” Research status for individuals is measured by whether or not they were offered treatment, rather than whether or not they actually received or completed treatment. Calculating ITT impacts using linear regression is the same as calculating differences in average group outcomes using ANCOVA. Baseline covariates X even can be omitted entirely; in that instance, the regression procedure is known is ANOVA (ANalysis Of VAriance) rather than ANCOVA. If ANCOVA is used to calculate RCT impacts, it is good practice to publish ANOVA results as well for the most important outcomes. Doing so assures report readers that ANCOVA covariate specifications have not been manipulated to tilt results in one direction or another.
