In 2008, an article in Proceedings of the National Academy of Sciences (PNAS) reported that lithium had slowed the progression of amyotrophic lateral sclerosis (ALS), which is always fatal. This article describes several attempts to confirm that effect of lithium. Three studies were launched by med school professors. In addition, patients at PatientsLikeMe also organized a test.
One of Nassim Taleb’s complaints about finance professors is their use of VAR (value at risk)Â to measure the riskiness of investments. It’s still being taught at business schools, he says. VAR assumes that fluctuations have a certain distribution. The distributions actually assumed turned out to grossly underestimate risk. VAR has helped many finance professionals take risks they shouldn’t have taken. It would have been wise for finance professors to wonder how well VAR does in practice, thereby to judge the plausibility of the assumed distribution. This might seem obvious. Likewise, the response to the PNAS paper revealed two problems that might seem obvious:
1. Unthinking focus on placebo controls. It would have been progress to find anything that slows ALS. Anything includes placebos. Placebos vary. From the standpoint of those with ALS, it would have been better to compare lithium to nothing than to some sort of placebo. As far as I can tell from the article, no med school professor realized this. No doubt someone has said that the world can be divided into people focused on process (on doing things a certain “right” way) and those focused on results (on outcomes). It should horrify all of us that med school professors appear focused on process.
2. Use of standard statistics (e.g., mean) to measure drug effects. I have not seen the ALS studies, but if they are like all other clinical trials I’ve seen, they tested for an effect by comparing means using a parametric test (e.g., a t test). However, effects of treatment are unlikely to have normal distributions nor are likely to be the same for each person. The usual tests are most sensitive when each member of the treatment group improves the same amount and the underlying variation is normally distributed. If 95% of the treatment group is unaffected and 5% show improvement, for example, the usual tests wouldn’t do the best job of noticing this. If medicine A helps 5% of patients, that’s an important improvement over 0%, especially with a fatal disease. And if you take it and it doesn’t help, you stop taking it and look elsewhere. So it would be a good idea to find drugs that only help a fraction of patients, perhaps a small fraction. The usual analyses may have caused drugs that help a small fraction of patients to be considered worthless when they could have been detected.
All the tests of lithium, including the PatientsLikeMe test, turned out negative. The PatientsLikeMe trial didn’t worry about placebo effects, so my point #1 isn’t a problem. However, my point #2 probably applies to all four trials.
Thanks to JR Minkel and Melissa Francis.
VaR doesn’t assume fluctuations have a normal distribution. It is of course possible to set up a VaR calculation with that assumption, or any other, and maybe a certain laziness leads people to too often assume normality when other choices aren’t obvious, but normality is not an intrinsic assumption of the model by any means. In practice normality is not how I see it being done (rather, the distribution is imputed from historical time-series – which is also a problematic approach, because until 2007 none of those time-series included a subprime/credit crisis, then suddenly in 2007 they did, then as we roll long enough past 2007 the subprime/credit crisis will drop back out of the time-series again..etc. Though there is an attempt to fix that by forcing time-series to include some really bad shocks whenever they occurred…).
It is true of course that VaR is a misleading guide to risk and that this is partially because of the arbitrariness in how fluctuations are assumed to be distributed (normal or not) and the fact that whatever distribution they choose is likely to fail to capture whatever shock is coming.
But it’s also because VaR simply depends on too many variables and inputs to ever be able to calculate correctly. If VaR were as advertised the way it would work would be the risk/control group saying ‘hey you’re over your VaR limit’, and this would genuinely mean the desk had too much risk, and the desk would then proceed to get back under their VaR limit by genuinely reducing their risk. In practice the way it often works is 1. ‘hey you’re over your VaR limit!’, 2. the desk sifts through the VaR calculation and points out an error in it (there is always an error somewhere, and the desk is of course more likely to point out an error whose cure will reduce not increase their VaR), it turns out that if fixed they aren’t over their limit after all, and 3. the risk group says ‘oh yeah’ and goes back to the drawing board. Or, the desk can actually attempt to identify nonsense/spurious trades they can put on (by essentially reverse-engineering VaR) which won’t necessarily generate any profit but which will predictably reduce their “VaR” as calculated by the risk group simply because of whatever quirks in the model. In either case the method/approach of VaR is only tangentially related to genuine risk limits/controls, its more immediate effect is to divert the desk’s resources from revenue generating activity and towards this sort of VaR deconstruction.
So sorry to hijack from your main point. To try to get back to it: you are correct indeed that VaR is a good example of a mentality that focuses on process over results. The risk group is dependent on VaR being accepted as meaningful for their jobs, which are essentially nothing but following Correct process, which if they do, they’re ‘doing a good job’ regardless of results. Upper management likes/uses VaR because they can throw it on charts/graphs showing that they reduced VaR or are keeping their VaR down, and this is meant to be equated with results. Nobody in his right mind who cared about results would actually managing their risk to VaR literally (and small comfort, in practice no one does)…
i don’t know anything about ALS, but i know a bit about VaR.
first, of course finance profs teach VaR, because businesses use VaR. it’s a useful but flawed measure. (any single measure that applies to a portfolio is going to be flawed, but it’s necessary to have some such measures when institutions have large portfolios. think of other economic measures like GDP and unemployment rate — they are heavily flawed but policymakers and economists use them all the time.)
second, VaR does not assume any distribution. you can do VaR with any distribution you like, and people use all kinds of distributions with VaR. if there is an issue with this, it’s that ALL the distributions are potentially wrong because, in finance, there is no guarantee that the past will resemble the future.
sorry, crossposted with Sonic Charmer
q & sonic charmer, thank you for your comments. You are right, VAR doesn’t assume any particular distribution. That is added by the practitioner. I should have said that VAR was used with distributions that turned out to vastly underestimate actual risks. I have changed the post to reflect this.
q, I also think you make a reasonable point that finance profs taught VAR because it was used. I agree that is a good reason to teach something. But, also, finance professors are supposed to think for themselves.
I don’t think your point #2 is of much use unless it’s possible to isolate the group that benefits. If 5% of people with ALS benefit and 5% are harmed (and 90% unaffected), it wouldn’t make sense to treat 100% of that population.
vic, little would change. Everyone would take the drug. After a reasonable length of time, those whom the drug doesn’t help would stop taking it.
seth — finance teachers are supposed to think for themselves, yes, but there are a couple things to remember. finance programs are usually run by business schools even though they masquerade as mathematics, so intellectual rigor goes out the window. (if they were run by math departments, you would expect any connection to reality to be thrown out — and i suppose you have somewhat of a combination between the two.)
in any case, saying that “finance teachers should think for themselves” would indicate that they should still teach VaR but talk about its many limitations and weaknesses. in the one class i took that covered VaR, the professor did this. if people go through these programs and don’t know a lot about VaR they are not going to get jobs and if they do, they are not going to be effective in them.
oh, and while we are on the subject of taleb, the problem i have with him is that, mathematician he is, he criticizes the quants’ mathematics much more than the sociology of the business. sure, the math has holes, but finance isn’t engineering. the fact that people believe things they shouldn’t believe isn’t the fault of the math, and fighting the math with math is besides the point.
q, your criticism of Taleb isn’t clear to me. He’s a philosopher more than a mathematician (in the sense that the issues he discusses are closer to what philosophers have cared about than what mathematicians have cared about) but to me that’s beside the point. I would say that he’s a realist and the people he criticizes have not just made unrealistic assumptions but also claimed their unrealistic assumptions were a big improvement.
I don’t know anything about VaR, but I have learned a bit about ALS since I was diagnosed with it last year.
As far as placebo trials, this is a sore point with the ALS cimmunity due to the small number of subjects. Diverting 1/2 or 1/3 of patients to the placebo arm of a trial reduces the number of trials that can be held. Partly for this reason I know some trials have been done without placebos. They either use a statistical technique called “futility” (don’t know what that means), or they compare against placebo statistics from an earlier trial. I would think it would still be necessary to periodically perform a placebo-like measurement of disease progression, to have an up to date baseline to compare with.
Still I would worry that bias could creep in. Even with a disease like ALS, with a well documented process of neural damage, a placebo effect can occur. It is difficult to objectively measure ALS progression due to the variable nature of the disease, so researchers often rely on patient reports of difficulty performing everyday tasks like eating or dressing. Desire to please researchers, among other effects, could lead patients to unconsciously color their reports. Different research setups and patient relationships could make this kind of placebo effect vary from trial to trial. You’d hate to put resources into an apparently successful treatment when the real effect was just that they had cute coeds doing patient interviews.
Waiting for an objective end point like death can eliminate this problem but even with ALS that means waiting 3-5 years. There is a lot of pressure in the patient community to speed up trials. Nobody wants to wait that long for results.