Noah Smith and Matthew Yglesias both have recent posts in which they argues that because DSGE models have not been adopted by investment bankers and other financial market participants that they have failed the market test. As Noah puts it, “(if) DSGE models work, why don’t people use them to get rich?”
Noah continues: “If you have a model that both A) satisfies the Lucas Critique and B) is a decent model of the economy, you can make huge amounts of money. This is because although any old spreadsheet can be used to make unconditional forecasts of the economy, you need Lucas-robust models to make good policy-conditional forecasts.”
This “market test” argument might sound good but Noah’s critique is actually somewhat off target. The fact that investors do not use DSGE models to make money might says basically nothing about whether DSGE models are useful analytical tools.
Think about simple supply and demand models. Supply and demand models are DSGE models and they will fail the market test that Noah emphasizes. (For those of you who don’t know, DSGE stands for Dynamic Stochastic General Equilibrium.) To be specific, let’s consider supply and demand in the market for oranges. How the market behaves is determined by the elasticities of supply and demand which, respectively, tell us how price sensitive orange farmers and orange buyers are. OK, now suppose that occasionally there are spells of bad weather which make growing oranges difficult. A casual observer would notice that when the weather gets bad in Florida, the price of oranges rises and the quantity produced and purchased falls. When the weather is good, prices are low and quantities are high. This observer will notice these patterns and the patterns will become part of her beliefs about the world she lives in. Of course, the observer may not understand why this pattern exists – she merely understands that the pattern does exist.
Alright, now let’s extend our supply and demand model a bit. Let’s now suppose that weather conditions are somewhat persistent from year to year. If the weather is bad this year then it is likely to be bad next year. In this case, when prices are high one year, they will tend to be high next year (high prices this year means that the weather must be currently bad). Again, our casual observer will incorporate this pattern into her beliefs and again she will not be required to understand why this pattern exists. Suppose we add a financial market which coexists with the orange market. The financial market sells claims on future orange prices. A hypothetical contract might pay one dollar in the event that the price of oranges next year is above the historical average price.
If you are following along, you will realize that we are squarely in DSGE territory. This is obviously an Equilibrium model; the model is Stochastic (due to the recurring random swings in the weather); the model is Dynamic (due to the persistence of the weather conditions), and the model is General (due to the presence of both an orange market and the financial market making bets on the future price of oranges, both of which are in equilibrium). In fact, as I’ve described it, it sounds like the model satisfies the rational expectations hypothesis too.
Suppose now an economist comes up with a model which explains the price and quantity variations in terms of supply and demand. Unbeknownst to this economist, the model is actually true. The model provides a meaningful and accurate description of how the orange market works. However, the model is not particularly useful for predicting future prices. The model says that if there is an adverse shift in supply, then prices should rise and quantities should fall. Given the shift in supply, the amount of the price and quantity change are governed by the two structural parameters (the two elasticities). However, predicting future prices in this environment boils down to predicting the weather, and on that score, the supply and demand model, despite being true, is of little help. In contrast, quantifying the observable patterns in the data is definitely helpful for the purpose of forecasting future prices. In fact, the current price contains valuable information on the likely future price. A simple regression of the current price on the past price will provide financial market participants with enough information to price bets on future prices. (If prices and quantities are measured with error, then the best forecast will make use of both price and quantity to predict the future price.)
In this environment, the financial traders have no use for the DSGE model. Thus this supply and demand system will fail the market test in Noah’s and Matt’s posts. At the same time, the supply and demand model provides key insights into how this market works.
In fairness, Noah does sneak in a slight caveat in his post. He says that a correct DSGE model should do a good job of providing policy-conditional forecasts. Fair enough. If there is a change in policy then the statistical patterns that prevailed in the past might well change (this is an instance of the well known Lucas critique). If there were a subsidy to orange farmers in our example, the economist’s DSGE model would correctly predict that average prices would fall and average quantities would rise and so you might think that having a correct prediction would mean that the model would be valuable in this instance. Are we really to believe that, faced with some new policy, people don’t turn to models like this to refine their predictions? I would think that it would be reasonable to think that for most purposes, investment bankers can simply use purely ad hoc statistical forecasting methods – methods devoid of any structural economic content but which have substantial predictive content — to make market predictions. In the rare instance that there is some important change in policy they might use a structural model to adjust their predictions. Ask yourself this: when the Affordable Care Act was being discussed, how do you think observers and participants in the markets for health care made predictions about what might happen to their industry? If your answer is that they turned to estimated structural economic models, then can you really say that these models are failing the test of the market?
Orderstatistic 
Chris, thanks for the response!
But what you call a “slight caveat” is actually the whole point of my post, and I tried to put it front and center instead of “sneaking it in”. If a structural model gives a policymaker more predictive power about the results of her actions than would other available models, then it should give a financial trader similarly increased predictive power relative to whatever she is currently using.
Very nice post and I like the way you spelled out DSGE, the words do actually mean something.
I find the “market test” pretty unconvincing. The ‘statistical models’ which supported the underwriting of subprime mortgages (some of the products had worked out fine in the past) and suggested that their impact on the economy was limited … made some people in the market very rich before they made a lot more less rich. DSGE models did not predict the financial crisis either but I see no reason why economists should not have a huge, deep toolkit even if not marketable.
At the Fed, I primarily work with partial equilibrium time series models of consumption (conditional on a policy path and other conditioning variables) that get stitched together with judgment and other aggregate demand components to make the staff GDP forecast. Other colleagues run policy simulations and forecasts in large macro models like FRBUS and DSGE models like EDO. We all work hard in preparing numerical forecasts, crafting explanations in words, and highlighting the many limits of our models/data/theory.
As I mentioned to Noah, I can’t do GE in my head (and know few economists who can) so I have a lot of respect for the intuitions that can (with much effort) come out of the GE models. Would I trade my consumption model (which has all kinds of squishy variables like consumer sentiment) for the best DSGE model today for forecasting growth over the next few quarters? No way. But the development of those models is worthwhile. I learn a lot when our various models do not line up and we try to figure out why.
And not just me. The Fed has a system-wide project to build, compare and use these models: Overview of DSGE policy uses: http://philadelphiafed.org/research-and-data/publications/business-review/2013/q2/brq213_dsge-models-and-their-use-in-monetary-policy.pdf and a more technical paper: http://www.newyorkfed.org/research/staff_reports/sr554.html
PS So happy to see you blogging!
…In other words, if DSGE helps policymakers improve their predictions of the effects of their own actions, then it should obviously help financial traders improve their predictions of policymakers’ actions.
Noah, your point is well taken but the market test critique is then limited to only instances when there are actual policy changes (most “actions” by the federal reserve are not policy changes the way you would need them to be …). Also, I suspect that models similar to DSGE models are used in these cases.
the market test critique is then limited to only instances when there are actual policy changes
Definitely. If these simply happen so infrequently that it’s not worth it for companies to keep DSGE modelers around on staff, then the critique fails.
(most “actions” by the federal reserve are not policy changes the way you would need them to be
Really? Even if a policy change isn’t a regime change, shouldn’t most Fed “actions” contain some unexpected, surprise component? And don’t we need structural models to optimally forecast the impulse responses to those shocks?
Also, I suspect that models similar to DSGE models are used in these cases.
That would be interesting to know. So far, no one I’ve talked to has heard of such models. But they may exist…
Noah, in my opinion, if the Fed is doing its job, there should be no (or few avoidable) monetary policy “surprises.” There are many ways for the Fed to communicate policy changes, in a structural sense, to the market and no Wall Street firm should need (or rely on) their own DSGE model to discern the monetary policy regime.
Claudia, what about fiscal policy? You don’t think that is well-communicated, do you?
Pingback: The “devalue and dismiss” fallacy, methodological pluralism, and DSGE models
Some actions by the Federal Reserve do have elements of surprise (though Claudia is quite correct when she emphasizes the Feds modern tendency to communicate their intentions fairly clearly). Even when surprises do occur however, this is almost always just the Fed executing its policy rather than a change in policy. In this case simple inferential forecasts will typically do as well (or better) than DSGE models.
Chris, I don’t understand. If a Fed’s action is “executing its policy” without “changing policy”, but is also a “surprise”, won’t that represent an update in the rational expectations of firms and households with regards to the Fed reaction function? In other words, as Claudia said, doesn’t a true “surprise” without a true “policy change” represent an improvement in the Fed’s communication of its reaction function? And wouldn’t you need a structural model to properly predict the impact of that?
Surprises don’t necessarily imply changes in policy. Take for example, the enforcement of holding penalties in the NFL. On any given play, the refs may or may not penalize a player for a given infraction. The reasons for this randomness could range widely: the ref might or might not have seen the play very well; the ref could simply be in a bad mood, etc. In any case, if the referee penalizes a player for holding this is the execution of an established policy but nevertheless has a strong random element. Similarly for the Fed, given current economic activity and current inflation, the Fed might be inclined to raise interest rates (say). If the choose not to raise rates then this is a surprise but not a policy change. In this case, standard forecasting methods are fine.
For the ZLB cases (or QE or the ACA and the like) we have little past information to go on. In addition, the Fed was probably making up some of its policy choices as it went along. Indeed in these instances, a well constructed DSGE model would be valuable.
In addition, it seems to me that there are reasonable frequent situations where the Fed has to basically invent new policies (like when we went below the ZLB, making a standard Taylor-type nominal interest rate rule infeasible). It seems like in those situations, one will always need a structural model to help guide one’s investment decisions.
Noah I think we are mixing two things: 1) the policy (monetary or fiscal) maker’s reaction function to the world and 2) the changes in the world that feed into that policy function.
Policy functions are determined and (best) communicated by the policy makers, not DSGE models. Sure, policy exercises with DSGE models and many, many other analytical tools may help policy makers determine (and update) their reaction functions. Wall Street and market participants rarely make money by researching ‘better’ policy reaction functions … but many find it helpful to forecast and keep track of the variables that seem to feed into policy functions.
None of this as precise or pretty in reality but again the lack of DSGE models on Wall Street is not a strike against those models per se. Again, all of this is just my opinion.
Broadly speaking, I think Chris is correct in pointing out that DSGE models can have strong explanatory power that can be very useful to policy makers. The problem is that DSGE models are so flexible that many fundamentally different models can be used to explain the same phenomena. The only way to distinguish the best among them would be to see which model had the most predictive capacity. The fact that we do have regular policy changes, such as Dodd-Frank or changes to the European banking regime or Basil III, and there is no evidence that any DSGE models are used by those in a position to profit from predictive models suggests that DSGE models lack predictive power, and thus we are unable to distinguish which among them has the most explanatory power. Which model an individual uses will then be up to their discretion, what they think makes the most sense, but not confirmed by some scientifically verifiable process such as predictive capacity. The consequence of this is a lack of consensus on which models are superior.
I have problems with this post, mainly on an epistemological level. You write:
“Unbeknownst to this economist, the model is actually true. The model provides a meaningful and accurate description of how the orange market works. However, the model is not particularly useful for predicting future prices.
How would one ever know the model is ‘true’, without testable predictions of its main variables in time? Such a model is an theoretical construct to be judged on coherence and consistency, not ’empiricals’. (I believe such models have no place in empirical sciences)
Also, since the weather in your example is not random – i.e. one point in time correlates to another, close point – your model shouldn’t be random or stochastic. If weather can be partly predicted on the basis of former observations, economic data points should likewise be at least weather-predictable.
These are good questions.
On the first, I didn’t say that the model had no testable predictions. It does. For instance if we made some policy change (e.g., a tax on orange consumption) the model will predict a reduction in price and quantity. If we don’t observe the reduction in prices and quantities following the tax then this would be grounds for rejecting the model. Ultimately the final judgment for the model must be empirical based.
On the second question, it sounds like you are understanding the setting correctly. I *am* saying that economic data will be weather-predictable (knowing the weather today tells you something about likely prices tomorrow). Such a forecasting equation would work just fine in this setting. Incidentally, the model is stochastic — this type of correlated random variables show up often in real-world data (what statisticians call “serial correlation”).
I appreciate you responding to my comment. I did not claim that DSGE models do not have testable predictions, simply that they lack predictive power. They can be tested, and the fact that they are not employed by market participants suggests they do not past that test. Some models clearly will have more empirically accurate predicted outcomes than others, but the ease with hedge funds could employ a macroeconomist to utilize these tools in order to profit from them, and the reality that they don’t, suggests they lack significant predictive power. In the absence of this empirical confirmation (perhaps a high standard but the standard of Friedman), we are unable to distinguish or even confirm that DSGE models have explanatory power.
I guess my question is, how do you empirically confirm or judge an economic model besides testing its predictive capacity? When you say empirically based, what does that mean if it is not predictive capacity? I take the lack of use of these models in the private sector as evidence they lack predictive capacity, but is there strong evidence otherwise?
Pingback: The Latest Macro Dust-Up | Orderstatistic