Pages

Saturday, April 10, 2010

Review of Climate Model Evaluation

Here's an interesting paper on the present-day evaluation of climate models:

"How Well Do Coupled Models Simulate Today’s Climate?" Thomas Reichler and Junsu Kimm, BAMS, March 2008, p. 303-311 (American Meteorological Society)

Their conclusion?
CONCLUSION. Using a composite measure of model performance, we objectively determined the ability of three generations of models to simulate present-day mean climate. Current models are certainly not perfect, but we found that they are much more realistic than their predecessors. This is mostly related to the enormous progress in model development that took place over the last decade, which is partly due to more sophisticated model parameterizations, but also to the general increase in computational resources, which allows for more thorough model testing and higher model resolution. Most of the current models not only perform better, they are also no longer flux corrected. Both improved performance and more physical formulation suggest that an increasing level of confidence can be placed in model-based predictions of climate. This, however, is only true to the extent that the performance of a model in simulating present mean climate is related to the ability to make reliable forecasts of long-term trends. It is hoped that these advancements will enhance the public credibility of model predictions and help to justify the development of even better models.

(Emphasis mine.)

24 comments:

  1. Back when I worked among model-oriented physicists, the model "figure of merit" was the buzz phrase.

    The phrase "objective measure" recalled it for me.

    The chief thing about these people was that they didn't like physics, or anyway didn't do any.

    The figure of merit was good for a couple of internal papers.

    These people were in particular not at all concerned with getting the physics right.

    Which connects with R.W.Hamming's admonition, "The purpose of computation is understanding, not numbers."

    ReplyDelete
  2. Most of the climate modelers I've talked to (usually at conferences) are both physicists and coders.

    ReplyDelete
  3. To be sure, everybody was a physicist.

    The trouble is when the physics is pretty much left behind by simplifications.

    That's what separates one kind of physicist from the other, namely whether they'll follow through with the simplifications, or work on something else entirely that still seems to them like real physics.

    You may be a physicist working with simplifications, but you're not doing physics.

    That has implications when the job description ("physicist") is confused with the work.

    ReplyDelete
  4. > You may be a physicist working
    > with simplifications, but
    > you're not doing physics.

    Of course, EVERY calculation ever done makes simplifications. All physical theories assume an idealized view of the world, and all physical theories are valid only in certain realms (but not all). In fact, you could go so far as to say that physics is about the artful application of simplification, about knowing what to include and what to leave out.

    Examine any open source climate model, like NASA GISS's E model. The physics involved is quite elaborate.

    Besides, you have yet to offer any alternative method of determining future climate. The issue is too important not to use the best available methods.

    How would you do it?

    ReplyDelete
  5. You can't compute it.

    Leave the earth to take care of itself. It's been doing it okay for a while now. Evidently it operates in a region of stability.

    The alarm itself is as bogus as anything.

    The alarm is there because it's profitable.

    On simplicity: you want a simple theory that accounts for complex results. Physics operates by adding to the complexity of the results, not the complexity of the theory.

    It's the ratio of the complexities that's important.

    If you add instead to the complexity of the theory, you're doing curve fitting, not physics.

    You've lost the ratio that guards against error.

    The simplification I was talking of in modelling, though, was the simple replacement of (partly) known but unsolveable component PDEs by some guess about what the solution would look like, which is not physics at all. It's a need for a number.

    ReplyDelete
  6. > The alarm itself is as bogus
    > as anything.

    How do you know, if you can't compute future climate?

    ReplyDelete
  7. How do you know, if you can't compute future climate?

    Because "knowing" holds off the cognitive dissonance that arises when reality isn't going according to the worldview.

    Best,

    D

    ReplyDelete
  8. How do you know, if you can't compute future climate?

    You don't know most things in that phil 101 sense.

    Why are you not worried, would be a better question.

    Because the earth has been getting along fine for a long time, so is evidently stable. If it's near some edge, it would have gone over long ago.

    Because Baysean odds against it are therefore huge.

    Because further the odds of cheap and widely available computation coming along at the exact instant earth needs urgent computational warnings are zero. Multiply that in.

    Because further there's a good account of the warnings, namely that they produce money and power, and the odds of that are very everyday. It does not require bad motives.

    What's arrayed against the denier position is not science, which I have been trying to point out, but the trappings of science.

    If you can't compute it, you can't do science on the matter, even if you want to, or have to.

    ReplyDelete
  9. > Because the earth has been
    > getting along fine for a long
    > time, so is evidently stable.

    Here is exactly the fallacy in your logic. The Earth system has been stable. That does not logically imply that the (Earth + man) system is stable. Man is subjecting the Earth to new forcings never seen before. So all previous analogies are thrown out.

    Moreover: even if the Earth + man system is stable, in the sense that it doesn't run away, it may still (and is) swing quickly in one direction, with consequences.

    So, again: How do you know, if you can't compute future climate?

    ReplyDelete
  10. > Because further the odds of
    > cheap and widely available
    > computation coming along at the
    > exact instant earth needs urgent
    > computational warnings are zero.

    This is outright jibberish.

    It's no different than saying "the odds of the wheel coming along just at the exact instant humans need faster transportation are zero."

    Computation is a tool, nothing more.

    ReplyDelete
  11. So, again: How do you know, if you can't compute future climate?

    Is this a question about knowing or about computing?

    If about knowing, we'd have to get into what certainty is and what knowing is, in particular that these have normal and uses ("How do you know she's coming tomorrow?" "She just called." It's normally a call for such a reason and nothing more. You can imagine a reason for the question quite easily.), and this isn't one of them.

    I gave the normal and usual reason. It's too improbable for a couple of reasons, and nothing justifies worrying about it.

    I do not worry about monsters in the dark, either. -- But there could be monsters! -- Still I don't worry about it.

    If it's a question about computation, I'm saying that this computation isn't going to help you.

    It's more than a matter of taste because of the vast political resources involved, which is why it's worth arguing out.

    ReplyDelete
  12. It's no different than saying "the odds of the wheel coming along just at the exact instant humans need faster transportation are zero."

    But humans didn't urgently need transportation at that instant.

    The earth needs the urgent warning right now.

    It's the coincidence of two independent events, (1) the need for a warning, and (2) the availability of the means of warning, that multiplies down the probability.

    On the other hand the whole thing is quite probable under a different dynamic, namely that as soon as the means of warning are available, warnings will appear if they're encouraged.

    Several means of encouragement are around.

    The simulations that warn will tend to survive and mutate and grow; the simulations that don't warn wind up in the punch card recycle bin.

    It's evolution.

    Which is the probable side?

    By a long way?

    ReplyDelete
  13. Computation is a tool, nothing more

    Computation can be quite beautiful.

    People with that kind of sense spend a lot of time arranging it, whatever they compute.

    If you find somebody like that, hire him.

    ReplyDelete
  14. >> So, again: How do you know, if
    >> you can't compute future climate?

    > Is this a question about knowing
    > or about computing?

    Obviously it's not a problem about knowing. Everyone acknowledges there are significant uncertainties.

    > I do not worry about monsters
    > in the dark, either.

    This is a false analogy. There are very good reasons to worry that increases in GHGs will cause warming. This is, after all, why they're called GHGs. We know that they warm our atmosphere above what it would be without them by about 20 C (see this week's Eos).

    Why shouldn't more of them cause warming -- as is evident from the paleo record?

    Also, as a science journalist, I'm wondering: where have your ideas been vetted and published?

    ReplyDelete
  15. This is, after all, why they're called GHGs.

    My understanding is that greenhouses work by eliminating convection.

    Also, as a science journalist, I'm wondering: where have your ideas been vetted and published?

    You mean about Bayes? Much of life works off of it.

    If you mean about science, I imagine it's just a restatement of Popper.

    The idea that the relative complexities of theory and data define the transition from science to curve-fitting perhaps is more from statistics.

    ReplyDelete
  16. > My understanding is that
    > greenhouses work by eliminating
    > convection

    Your understanding is quite wrong.

    They work by reradiation, regardless of the source.

    ReplyDelete
  17. >> Also, as a science journalist, >> I'm wondering: where have your >> ideas been vetted and published?

    > You mean about Bayes? Much
    > of life works off of it.
    > If you mean about science, I
    > imagine it's just a restatement
    > of Popper.
    > The idea that the relative
    > complexities of theory and data
    > define the transition from
    > science to curve-fitting perhaps > is more from statistics.

    Alright, it's clear that you're just a bullshitter, and probably a troll.

    In any case, you have no intellectually serious responses, and don't hold a candle to the dedicated, erudite, serious scientists I talk to who actually, you know, write papers.

    rhardin: Stop the bullshitting or I will block your IP address. You're wasting everyone's time here. Constructive thoughts are welcome. Your obvious bullshit is not.

    ReplyDelete
  18. > My understanding is that
    > greenhouses work by eliminating
    > convection

    Your understanding is quite wrong.

    They work by reradiation, regardless of the source.


    Wiki

    "The glass used for a greenhouse works as a barrier to air flow and its effect is to trap energy within the greenhouse, which heats both the plants and the ground inside it. This warms the air near the ground, and this air is prevented from rising and flowing away."

    ReplyDelete
  19. rhardin: Stop the bullshitting or I will block your IP address. You're wasting everyone's time here. Constructive thoughts are welcome. Your obvious bullshit is not.

    [bafflement]

    ReplyDelete
  20. rhardin wrote:
    > "The glass used for a greenhouse
    > works as a barrier to air
    > flow and its effect is to
    > trap energy within the
    > greenhouse

    Wow. I mean, wow.

    I am simply amazed that you would quote this.

    I mean, it seriously proves your stupidity.

    For years scientist & journalists have been emphasizing that GHGs do not work, literally, as does a real greenhouse.

    You really don't know this?
    Seriously?

    Real greenhouses do trap conventional heat. Real GHG gases do not. The bad features of this analogy have been pointed out for about... 100 yrs now.

    I cannot believe you do not this.

    Such stupidity invalidates all of your claims.

    ReplyDelete
  21. There are very good reasons to worry that increases in GHGs will cause warming. This is, after all, why they're called GHGs.

    But greenhouses to not work that way.

    So it's not a great reason.

    ReplyDelete
  22. > But greenhouses to not work that
    > way.

    You were the one who stupidly thought GHGs worked like an actual greenhouse, trapping convection, not me.

    ReplyDelete
  23. You were justifying the name greenhouse, so I pointed out that greenhouses don't work that way.

    ReplyDelete
  24. > You were justifying the name
    > greenhouse, so I pointed out that
    > greenhouses don't work that way

    Whatever.

    I'm tired of your slippery word games.

    ReplyDelete