How Often is a Temperature Trend Significant?

I've finally learned enough Python in my spare time to calculate something I've wanted to do for awhile: begin to answer how often past temperature trends have been signficiant.

That is, if you calculate the all possible N-year linear trends for a dataset, how often are their statistical significance 95% or higher, for all N?

Not as often as you might think.

I used HadCRUT4 surface data, which begins in January 1850. So for N=2 (2 years), you can calculate a linear trend from 1/1850 to 12/1851, from 2/1850 to 1/1852, from 3/1850 to 2/1852, etc., up to 24 months before the present.

Of these 1,949 trends for N=2, how many have been statistically significant at the 95% level?

50%, if you don't include any autocorrelation. Obviously, with autocorrelation it will be smaller, because the error bars on the trend will be bigger.

I did this for N from 2 years to 99 years. Here's my result:

So 10-year linear trends of HadCRUT4 have only been statistically signfiicant 65% of the time. 20-year trends 76% of the time. Etc. (This assumes, as usual, that the residuals of the linear fit are normally distributed.)

And, mind you, this is without any autocorrelation factored in, not even rank-1 autocorrelation between a temperature point and its nearest neighbor, let alone the exponential decay method of calculation autocorrelation described in Foster and Rahmstorf (2011) and used by the Skeptical Science trend calculator.

I'll get to those soon. But in the meantime, even this basic calculation shows that talking about 10- or 15 or 18-year trends, like Werner Brozek likes to do at WUWT, is meaningless, just mindless numerology.

Nor, of course, is there anything magical about 95%. It means the result is significant to about 2 standard deviations, and it's the common benchmark for medical studies, but high-energy physicists (say) often more, as, for example, the 5-sigma requirement for the discovery of the Higgs boson. In climate science, which isn't an experimental science (viz. you can't do as  many repetitive experiments as you want), you can't get that kind of data -- you take the data you can get. So like we're supposed to take action if a warming trend is 96%, but not if it's 94%,or 80%, or 60%? That's absurd, of course.

Numerology is not a substitute for thinking. And short intervals are not meaningful.