Friday, November 20, 2009

The CRU Hack

I've been busy and have only been able to spend about 2 hours reading about the CRU hack, and some of the documents themselves.

A few comments:
  1. Can we assume all this is genuine? Some of it certainly seems to be, from what people have said. And there's surely too much to totally invent. But all of it? I don't know. It's easy to imagine the hackers making word changes and sentence additions, etc., before releasing it back out.
  2. Nothing I've seen so far gets me very excited. I don't see any evidence of a worldwide conspiracy to enslave us all into gray, communist conformity, nor any type of scientific collusion. What I see it just the ordinary exchange that takes place among...people, scientists included. They have their own inside language, thoughts, and jokes, and words that some conspiracy theorists are picking at are clearly just inside language that scientists use and that laypeople are not going to understand. They say things imprecisely and informally and off the cuff, like all of us, and don't imagine their emails are going to be splashed across the Web any more than you imagine your own emails will be for endless scrutiny by your enemies. Which of us could not be made to look completely evil if anyone were given access to all the emails we sent and received?
  3. Nor do I have any concerns about inside groups peer-reviewing other's papers. For any subspecialty in science these days, there are only a few people/groups who are going to understand it and who are potential peer-reviewers. When my advisor and I wrote our first paper together when I was in grad school (on using jets to detect quark-gluon plasmas), we got the (anonymous) reviewer's comments and immediately knew who they were -- one of the few groups to be able to properly evaluate the work and put it into context. We had read all their papers and knew all about their work. Graciously they suggested ways in which we might extend our results, but we went ahead and published without doing that and sure enough a few months later this reviewer and his student published the augmentation he had suggested. We weren't surprised at all. Nor did I feel like he did us any favor. In fact, in such small specialities there is often more competition than usual, and you have to work even harder to impress your colleagues (especially when they get the chance to be somewhat anonymous). Scientists love to tear one another down, as anyone who has ever attended a Friday afternoon department seminar knows. Then afterward they go out on the patio and have a beer together.
  4. I saw that my name popped up in a few places in the emails. I don't have much to say about them (which is neither a confirmation or a denial of whatever was claimed) except I do not at all recall ever threatening Sonia Boehmer-Christiansen (editor of Energy & Environment) with "litigation" (mail file 1068239573.txt) I can't imagine doing that as a journalist, and even if I wanted to I don't know what I'd threaten, unless it was to tell her I was going to file a FOIA, which I certainly did not do in this case since (a) I didn't think that that any of her internal journal correspondence was subject to a FOIA, as it was unrelated any government function, and (b) I don't even know if an American can file whatever the equivalent of a FOIA is in Britain. I do, though, plead guilty to having called her multiple times, whether it annoyed her or not. That's my job.
  5. The Arctic is still melting, glaciers are still receding, sea-level is still rising, tree lines are moving north, and both land and sea are getting warmer.
If there are excerpts of the emails you think I've missed, please point them out in the comments.

31 comments:

bob said...

The problem is a lot of the deniers don't understand how science works.

To them all data should be released to the public, including any stationary used to write that data. Anything short of that isn't science in their opinion.

They set the bar of expectation far too high like they do for models. If anything is below their far too high bar then they write it off as worthless.

I haven't worked out if they set the bar so high deliberately to prevent anything getting over the bar, or whether they truly believe it should be that high.

I think it would be fairly easy to answer this by getting them to apply their silly bar to other fields of science, but they often refuse to be examined and just gish gallop.

*Oh but I suppose anything skeptical they don't have a bar at all. Beck's co2 reconstruction is often given a free pass. So I guess they are just setting the bar high so nothing can get past.

My next question is are they doing this knowingly or subconsciously?

I suspect we are dealing with minds alien to us, people whose decisions and behavior is solely defined by their political ideology.

Anonymous said...

David,

Please give me an opinion on this

http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/the_warmist_conspiracy_tthe_emails_that_really_damn_professor_jones/

charlesH

Anonymous said...

i'll try again

http://blogs.news.com.au/heraldsun/andrewbolt/

"These are the emails that should have Professor Phil Jones most worried about his future."

charlesH

bob said...

Anonymous,

Andrew Bolt is simply making a lot of bad assumptions when he interprets the emails.

For example the data Jones has said is lost is a subset of raw station data. Whereas Jone's email was about deleting email correspondence, which was specifically to do with a request for IPCC related private emails.

Andrew Bolt confuses the two to claim that Jones was talking about deleting station data and then claimed they were lost.

This is just one such example of skeptics fabricating cases of dishonesty.

Another one I immediately spot is the claim of tax avoidance. I believe this one relates to an email by some russian collegues who requested funding be sent in a certain way so it wouldn't be taxed and more could be spent on work. This email wasn't even sent to Phil Jones. But Bolt mentions it in passing anyway as if Phil Jones had been dodging tax.

rhhardin said...

This seems to be a nice list link.

They sound like normal scientific model builders that I've worked with, except with a defensive ring around the model rather than the usual sense of irony that model builders take up.

I guess the difference is where the defensive ring is constructed, around the model or around the modeller.

Irony does the latter.

Model building, aka pulling equations out of your ass, has always been with us.

It is not marked by curiosity.

David Appell said...

Model building is hardly "pulling equations out of your ass."

In any case, please tell us how you propose to calculate future climate without models.

rhhardin said...

Sure, use the Navier Stokes equations.

Too hard to calculate with them? Pieces of information missing?

Right. You can't do it.

My own approach has always been Baysean.

What are the odds that computers become widely available at exactly the moment a dire warning is needed?

As opposed to, on the other hand, the Earth going along and making out somehow, under some sort of apparent stability, and thus not needing warnings.

I'd suggest going back to more elementary physics, like what causes wind waves to grow, a hot topic in the 70s in the JGR as I recall, back when science was science.

David Appell said...

People certainly started with elementary physical models -- they predict warming.

Solve the N-S equation? Sure. Go ahead and show us. They can barely be solved in simple fluid situations. In fact, it's not even known if they HAVE closed solutions in all situations in 3 dimensions.

How are you going to factor in land use changes, other GHGs, ocean currents, evaporation, cloud physics, solar changes, aerosols, etc into the N-S equations? Even in principle?

> Too hard to calculate with
> them? Pieces of information
> missing?
>
> Right. You can't do it.

Actually they can do it, as post-predictions of the 20th century show.

In any case, just throwing up one's hands and saying "you can't do it" is not sufficient, as too much depends on the answer.

rhhardin said...

If you can't do it, you can't do it.

There's lots of things you can't do.

This is one.

You can get nice numerical solutions of the Navier Stokes equations. It's well worth doing once, so you can get a feel for how far out of doable the climate is.

"How far out" means huge orders of magnitude.

So you have to leave physics pretty much behind. Enter model builders, unaffected by the these considerations.

rhhardin said...

You can fit any past data easily.

It has zero predictive power, however.

Imagine just running a polynomial through all the points, if it helps.

It predicts the past perfectly.

It predicts the future not at all.

Models are an obscure version of a polynomial fit.

Zoot said...

Are you kidding? I've read them! Threatening to delete data to avoid freedom of information. We deserve a fully open public review of the raw unedited data to be looked over by the scientific community. If they have nothing to hide than this shouldn't be a problem. I do not deny "climate change" I do however know a thing or two about science and "tricking" data is a big no, no?. Denying access to third parties, shame on them. If I took massaged data to a client I could be charged for fraud. Editing data and deleting information. This is not unbiased science.

Anonymous said...

"Sure, use the Navier Stokes equations."

Excuse me but what are those? Can they be solved in reasonable time when applied to a large and complex system? If not, how do you know they work in such a system? And please don't say they work because they work in smaller systems, since you cannot be sure. Why are you wasting everybody's time writing of something that you do not know if it works in such a system? Excuse me for the bluntness, but what are the error limits if you try to solve those in such a system?

David Appell said...

> Threatening to delete data to
> avoid freedom of information

Yes, that (if its true, and not added in by the hackers) is not acceptable.

Most journals have (or now have) policies that require authors to archive their data someone and make them publically available. AGU journals do. Science does.

rhhardin said...

"Sure, use the Navier Stokes equations."

Excuse me but what are those? Can they be solved in reasonable time when applied to a large and complex system?


The core equations of fluid dynamics.

They're hard to solve for several reasons, but they're the least that you have to solve.

One reason being that disturbances in three dimensions cascade to smaller scales, so your grid will never be adequate for long.

David Appell said...

> You can fit any past data easily.

Models do not "fit" data -- they predict values. My understanding is that most models today do not use so-called flux corrections (ref: IPCC 4AR WG1 Ch 8.2.7 p 607)--they are based purely on physics.

> Models are an obscure version
> of a polynomial fit.

That's wrong. They have their basis in physical laws, not statistics. The equations used are physical equations, not statistical equations. You should read one of the manuals that describes a model --

David Appell said...

> You can get nice numerical
> solutions of the Navier Stokes
> equations

And what do you think a climate model is?

It's numerical solutions to the underlying PDEs, including the NS equations. Except they incorporate a huge amount of other physics and physical processes that you can't get by just naively solving the NS equations.

David Appell said...

> So you have to leave physics
> pretty much behind. Enter model
> builders, unaffected by the these
> considerations.

That's just absurd.

Read a book like Pierrehumbert's "Principles of Planetary Climate," and you'll see some of the serious physics that form the basis for climate models. Read any of the documents describing a specific climate model, and you'll find it full of all kinds of physical equations.

rhhardin said...

I assure you they don't solve the Navier Stokes equations.

rhhardin said...

You try your model out on past data, which it is no good if it doesn't fit.

You model has parameters.

You fuss with the parameters.

Then the model fits.

That's a statistical procedure; you can formalize it easily and do it automatically.

It will work as well as a polynomial fit.

rhhardin said...

There's lots of interesting natural physics.

I always liked the guess that the way wind produced large scale ocean waves was by maser action, with the smaller waves breaking perferentially on the tops of larger waves and thus adding energy in the right places.

Whoever did that didn't have in mind funding for a bogus model. He was just curious where the hell the long waves come from, and thought this might be it.

That's the kind of area where actual physics can do something.

Model builders are a branching out towards a career path in management, as we used to call it.

Funding is everything there.

David Appell said...

> You model has parameters.
> You fuss with the parameters.
> Then the model fits.

Please tell us exactly what parameters modelers are "fussing with" to make sure their model fits the data.

You can pick whichever of today's models you want.

David Appell said...

> I assure you they don't solve
> the Navier Stokes equations

That's not what I said.

I said that they solve the underlying PDEs in exactly the same way you propose the NS equation be solved -- numerically.

You talked like you had some new approach to the problem. But it turns out that all you meant was...to do what climate models are already doing.

rhhardin said...

Being a PDE doesn't make it underlying physics!

I pointed to the Navier Stokes equations as an example of what actually governs the problem, not as an example of a PDE.

A parameter might be, oh, say "effective viscosity." V sub eff.

Looks scientific, doesn't it.

It's available to make the model that it's part of fit the data.

This is not anything new.

One interesting exercise is counting the degrees of freedom in stuff like that, and comparing how much data is being matched.

rhhardin said...

One of the advertised features of Kalman filters is the ability to determine parameters from data, no matter what the model is.

Not that there's magic in Kalman filters, but it's a handy formal way to compute what could be done other ways.

Feed in data and get out the best current estimate of V sub eff and all the other parameters; they can even be described by PDEs.

That just makes explicit the direction the implications really go, from data to parameters, in models.

David Appell said...

> Being a PDE doesn't make it
> underlying physics!

Of course not. But almost all the relevant physics equations needed to model climate dynamics are PDEs -- the equations of fluid dynamics and heat conduction and thermodynamics and radiation transfers.

David Appell said...

> A parameter might be, oh,
> say "effective viscosity."
> V sub eff.

So you complaint is that climate scientists don't determine effective viscosity from first principles, but instead use measured values?

That's very, very lame. That scenario is true for every single branch of physics I can imagine. There are many, many fundamental parameters that physics can't calculate from first principles but instead use measured values.

So what is your point?

David Appell said...

> It's available to make the model
> that it's part of fit the data.

Bullshit.

Climate modelers don't choose whatever value of the effective viscosity is needed to somehow predict future warming. They use the values that scientists have measured for it, as a function of temperature, pressure, density, and/or other relevant parameters. This is true for all areas of applied physics, including ones you trust your life to. Do you think that aeronautical engineers, when they model the airplane wings we all ride all the time, calculate Reynolds numbers from first principles?

David Appell said...

> One of the advertised
> features of Kalman filters is the
> ability to determine parameters
> from data, no matter what the
> model is.

Throwing around a big word like "Kalman filters" just convinces me more about that you're full of bullshit. I have yet to see any evidence that you know what climate modelers are actually doing, or that you've ever taken the time to read a document on the underlying design of a climate model. At best you've taken a course in PDEs and maybe introductory EE, about 30 yrs ago I suspect. You make a lot of big bullshit claims like you just need to solve the NS equations to determine future climate, which is clear utterly simplistic and something I'm highly sure physicists tried in about, maybe 1935. Stop giving this impression like you're somehow all-wise and the thousands of PHd climate scientists out there who work on this stuff their whole lives, 10 hrs a day are buffoons who can't even apply what they learned in sophomore calculus.

If your methods are so much superior, where are your results (published or not)?

rhhardin said...

I'm saying you can't solve the Navier Stokes equations, and you need to solve the Navier Stokes equations. So you can't predict the climate.

A Kalman filter just maximizes the normal probability of the data given the parameters, as a way to determine the parameters. It happens to be organized to easily add new data.

It's a way to formalize the model fitting as a statistical problem. It doesn't depend on whether there's physics there or not, just that there are parameters. You're not doing physics, you're doing curve fitting.

Hamming: the purpose of computation is understanding, not numbers. Sort of the opposite of climate modelling.

rhhardin said...

Aeronautical engineers work in thoroughly explored regimes.

That's where "pushing the envelope" comes from, actually testing something where you don't have data to see what happens.

They know when they're doing it. It's an actual science.

jyyh said...

(unscientific comment) So if a trade wind blows from the southeast it doesn't do that according to Navier-Stokes? How fairly interesting, in spite of all geostrophic winds from Antarctic High, may it live for ever.