Statisticians use the Greek letter α (alpha) to indicate the probability of rejecting the statistical hypothesis tested when, in fact, that hypothesis is true. Before conducting any statistical test it is important to establish a value for α. For most psychologists, and for many other scientists, it is customary to set α at 0.05.

This is the equivalent of asserting that you will reject the hypothesis tested if the obtained statistic is among those that would occur only 5 out of 100 times that random samples are drawn from a population in which the hypothesis is true. If your obtained statistic leads you to reject the hypothesis tested, it's not because you believe that the obtained statistic could not have occurred by chance.

It's that you are asserting that the odds of obtaining that statistic by chance only are sufficiently low (one out of twenty) that it's reasonable to conclude that your results are not due to chance. Could you be in error? Of course you could, but at least you know the probability of such an error. It is exactly equal to the value you have previously established for α.