7. Common Biases in Sampling Methods

k. describe the issues regarding selection of the appropriate sample size, data-mining bias, sample selection bias, survivorship bias, look-ahead bias, and time-period bias.

What is data-snooping bias? Data-snooping bias is the bias in the inference drawn as a result of prying into the empirical results of others to guide your own analysis.

What is data-mining bias? Data-mining is the practice of finding forecasting models by extensive searching through databases for patterns or trading rules (i.e., repeatedly “drilling” in the same data until you find something). It has a very specific definition: continually mixing and matching the elements of a database until one “discovers” two or more data series that are highly correlated. Data-mining also refers more generically to any of a number of practices in which data can be tortured into confessing anything.

How do you avoid data mining bias? You hold out some data to test the derived trading rules against.

What is sample selection bias? Sample selection bias occurs when data availability leads to certain assets being excluded from the analysis.

What is survivorship bias? Survivorship bias occurs when studies are conducted on databases that have eliminated all companies that have ceased to exist (often due to bankruptcy). The findings from such studies most likely will be upwardly biased, since the surviving companies will look better than those that no longer exist.

What is look-ahead bias? Look-ahead bias exists when studies assume that fundamental information is available when it is not.

What is time-period bias? Time period bias occurs when a test design is based on a time period that may make the results time-period specific. Even the worst performers have months or even years in which they look wonderful. After all, stopped clocks are right twice a day.