What the Heck is “Calibration” Anyway?

Every year I teach EC607 I arrive at the Real Business Cycle model and run into a problem. No, it’s not struggling to answer “why are you teaching the RBC model if you don’t think it is useful for understanding business cycles?”  No, the problem occurs when I get to the subject of calibration.  I would like to tell my students exactly what we mean when we say that we calibrate parameters.  I can’t tell them however since I don’t really know myself.

In my own work, I do things that I would describe as calibration. I even have an intuitive sense of what I mean when I say that some parameter has been calibrated.  However, I do not have a precise notion of what it means to calibrate a model. In fact, I am not sure anyone has a precise statement of what it means.[1]

Calibration is a way of assigning values to the parameters which determine how our models function. Unlike estimation, calibration does not assign parameter values to make the model fit the data. Some descriptions of calibration suggest that the parameter values should come from separate data sources – separate, that is, from the data that you are analyzing with the model.  In Advanced Macroeconomics, David Romer describes calibration as follows (emphasis added):

The basic idea of calibration is to choose parameter values on the basis of microeconomic evidence and then to compare the model’s predictions [with the data].  

This is a fairly reasonable description of what many people mean when they use the term ‘calibration’ but it is problematic for at least a couple of reasons.  First, economic data typically don’t come with useful labels like “Microeconomic Data: Approved for use with calibrating models” or “Macroeconomic Data: WARNING – DO NOT USE FOR CALIBRATION!” You might think that it’s obvious which is which but it’s not. Certainly panel data like the PSID sounds like data an applied microeconomist might use. What about price (inflation) data? Is that aggregate “macro data”? What about unemployment rate data? What about data on type-specific investment? Is that micro data?

Second, many of the calibrations used in practice seem to come from macro datasets anyway. Take for instance the calibration of the labor’s share parameter in the production function. This calibration is typically justified by calculating the average ratio of total employee compensation to total income – figures which both come from the NIPA.

Romer also says that we should choose the parameter values before comparing the model with the data.  I hear sentiments like this a lot though again it doesn’t really hold up when we look at standard practice. The labor’s share parameter is again a case in point. We are setting that parameter based on fitting a single moment of the data (we are going to match the model average labor share with the observed labor share). Another example concerns a standard calibration of investment adjustment costs in business cycle models. These parameters are sometimes calibrated to match the model’s predicted investment volatility with observed investment volatility. These examples make calibration sound suspiciously like estimation. (Hopefully, calibration isn’t just estimation without bothering to report a standard error.)

Nevertheless, even though I don’t really have a precise definition of what I mean by ‘calibration’, I believe that it may indeed have an important role to play in economic analysis. In particular, calibration might work quite well in situations in which we believe the model is wrong. (I know what you’re thinking – we always think the model is wrong! True. This means that calibration may indeed be very valuable.)

Let’s take a specific example. Suppose we have data on wages and employment and we have a labor supply / labor demand model which we propose to explain the observations. Suppose further that all of the changes in employment are driven by shifts to labor demand.  The only thing missing is the labor supply elasticity parameter.  An estimation based approach would do the following: we would invoke the null hypothesis that the model is correct and then estimate the missing labor supply elasticity from the observed data (just run OLS for instance). A calibration approach would not assume that the model is correct. Instead, a calibrated model would (somehow) obtain a parameter value from elsewhere, plug it into the model and compare the model output with the observed data. Let’s assume that the analyst calibrates the labor supply elasticity at roughly 0.5.

Suppose that (unfortunately for the econometrician) the model is mis-specified. In fact, the wage is stuck above the market clearing wage and there are many workers who are involuntarily unemployed. Every labor demand shift is resolved by simply absorbing available workers at the fixed wage.  The econometrician estimates the model and finds that the labor supply elasticity is very high indeed (near infinity in fact). The analyst using the calibrated model finds that his model predicts virtually no changes in employment.  Notice that it seems that the analyst using the calibrated model is actually on to something. There is a tension between his calibrated labor model and the observables. Moreover, this tension seems to provide an important clue as to how the model needs to be modified.[2] The econometrician on the other hand is happy with his estimates and will go about his business content in the belief that all is well with the model.

Naturally, the missing link in this narrative is the source of this outside information that the calibrated model draws on.  Where does this initial parameterization come from? Perhaps there were some earlier studies that provide some information on the labor supply elasticity? Perhaps the analyst just arrived at the number through sheer introspection. (If I were offered a wage increase, how would I respond?)  In a sense calibration shares a common thread with Bayesian estimation which requires a prior to guide the estimates (like calibration, the exact source of the prior is somewhat mysterious). In fact, many prominent researchers who advocate the use of Bayesian techniques come from backgrounds that embrace calibration (Jesus Fernandez-Villaverde was trained at Minnesota for instance).

One other thing which strikes me is that the researchers who use calibration are often much more interested in the performance and insights generated by the models and much less interested in the parameter values themselves.  Estimation it seems tends naturally to place much more emphasis on the point estimates themselves rather than their consequences.

In any case, calibration will likely continue to be used as an important analytical technique, even if no one knows what it actually is …

 

[1] My coauthor Jing Zhang assures me that calibration does indeed have a specific meaning though she has never articulated what this meaning is.  (Actually, when I asked her what she meant by ‘calibration’ her first reaction was to laugh at me after which she told me that I didn’t have proper training …).

[2] Paul Krugman seems to arrive at a similar conclusion in a past blog post (though you will have to put up with the obligatory “fresh water bashing” before the end of the post).

Advertisements

15 thoughts on “What the Heck is “Calibration” Anyway?

  1. In project management, we take a lot of data to get the parameters that move the estimates around. Then, we take a lot of data to make the estimates themselves. Macro vs micro. The macro data would be your calibration.

    In maintaining a particular system in the military, we calibrated the test equipment by testing an emitted signal without that signal going through the system. After calibration, we hooked the system and the test equipment together. We could see the loss caused by the system. A particular threshold for that loss would trigger maintenance. You can consider calibration to be part of the model. After calibration, you investigate the modeled, rather than the model.

  2. Pingback: Links for 4-07-14 | The Penn Ave Post

  3. I enjoyed this post, and have enjoyed your bog immensely.
    I thought you could have perhaps got some entertainment out of quoting from Kydland’s ‘computational experiment’ paper and Sims withering reply.

  4. In engineering we calibrate an algorithm against what we think a typical data set is with supposedly obvious features that the algorithm is supposed to find. Subsequent uses of that algorithm against data it’s never seen reflect its “performance”.

    The word itself derives from physical hardware that you had to adjust to make it work. Maybe that’s what economists mean 🙂

  5. “In any case, calibration will likely continue to be used as an important analytical technique, even if no one knows what it actually is …” Calibration is a process of comparison, a functional part of metrology, the science of measurement. It is primarily tied to the laws of physics and mathematics and provides a means to measure everything in our physical world using international measurement standards and units of measure agreed by treaty, public laws, policies, procedures etc. etc. The word, calibration, has leaked over into other disciplines by way of popular media and the brilliance of the many good idea fairies that plague the scientific community by looking for new and better methodologies that defy the laws of nature and physics. But, since calibration is already a part of the language in our everyday world, it simply means to compare a measurement system or entity against an equal or better standard. The standard is the reference that is understood to be the most correct. Calibration measures the deviation of the entity under test to the standard and then a determination is made that the tested entity is within the stated tolerance band or not. For example, if the standard is the number 100 and the tolerance is your stated error budget of plus or minus 10 (perhaps derived by uncertainty analysis) then the entity under test passes if it presents a result between 90 or 110. But, to be a true calibration the result must be precise, accurate, and repeatable over time. Finally, in this dangerously over simplified statement, the reference standard must be traceable to a national or international standard of measurement that is maintained by an acknowledged governing body like the National Institute for Standards and Technology (NIST) in the U.S. to be valid. Otherwise, one is simply using a word to describe a process made popular in various Hollywood movies using various strings, wires, lubricants and a bit of smoke with carefully positioned mirrors. Magic is less troublesome and gets good results as well. In econometrics using statistical models, the calibration of a model would be the testing of the model to see if it provides a predetermined outcome within the allotted uncertainty. Next, the model is tested for sensitivity to the variation of different parameters and if the outcomes still fall within the specified error budget, the model is valid. If not, the system model must be adjusted to provide a precise, accurate and repeatable (expected) outcome as referenced to the standard. Hope that helps but if not, research metrology, not calibration, to find the root definitions and the vast world of information regarding it.

  6. I believe that avoiding highly processed foods is the first step to be able to lose weight. They will taste very good, but prepared foods possess very little nutritional value, making you take in more to have enough electricity to get over the day. When you are constantly ingesting these foods, moving over to cereals and other complex carbohydrates will help you to have more electricity while consuming less. Interesting blog post.

  7. I intended to create you a very small observation to help say thanks once again with your gorgeous views you have contributed in this case. It’s so particularly open-handed of people like you to convey without restraint all that a lot of folks could have made available for an ebook to make some bucks on their own, notably seeing that you could possibly have done it in case you decided. The advice in addition acted like a good way to comprehend someone else have the identical desire similar to my very own to know the truth very much more related to this condition. I know there are a lot more enjoyable periods in the future for those who find out your blog.

  8. I’ve recently started a webcomic. After viewing some of my favorite other webcomic sites I noticed that they had a ? in every comic posted. My questions are: Do you have to Copyright a webcomic? Is it as simple as just typing in a ? into every comic or do I have to register something somewhere?.

  9. Pingback: ingenieurs marocains

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s