We address the problem that occurs when inferences about counterfactuals -- predictions, "what if" questions, and causal effects -- are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data are more model-dependent, and then develop easy-to-apply methods to evaluate how model-dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence.
See also our "Empirical vs. Theoretical Claims about Extreme Counterfactuals: A Response," in a Political Analysis symposium on this article (Paper: PDF | Abstract: HTML). You may also be interested in open source software we developed, called WhatIf, that implements all the methods in this paper. Also available is and an overlapping, companion paper to this one, entitled "When Can History be Our Guide? The Pitfalls of Counterfactual Inference," International Studies Quarterly, (Paper: PDF | Abstract: HTML): it excludes the mathematical proofs and other technical material, and has less general notation, but it includes additional examples and pedagogically oriented material (as well as a symposium in ISQ). Also see our approach to matching to deal with model dependence and other related research.