|
A salmon study...A Case Study in the Cross-Examination of Eco-Liars March 5, 2003 The right to cross-examine witnesses is one of the great rights that citizens of the United States once enjoyed, for cross-examination has been aptly termed the greatest engine for the discovery of truth ever invented. Witnesses of all types routinely shade their testimony, and their creativeness is tempered, if at all, by the fact that they may be placed upon the witness chair and asked to explain themselves. On November 20, 2002, I was privileged to have the opportunity to cross-examine a government expert witness in the case of Columbia Snake River Irrigators Association v. Washington Department of Ecology. The irrigators were suing because the Department had failed to act on their applications to appropriate water from the Columbia River for more than a decade. Most such cases would never go to trial; the Federal courts routinely decide challenges to agency inaction by "summary judgment", in which the courts essentially presume that written statements submitted from government experts are true. But live witnesses often tell a very different story than the one that emerges from papers prepared by Justice Department lawyers. In this case, the Department's excuse for failing to process the water rights applications was imagined fear to salmon from draining a microscopic percentage of the mighty Columbia River. The irrigators were armed with a large number of studies showing that appropriating the water would cause no harm to fish. One such study charted the relationship between river flow and salmon survival as follows:
Now any ordinary citizen can look at this data, and tell that if there is any relationship between flow and survival in this data, it is a negative one: the higher the flow, the lower the survival. The Department's expert, Dr. W. Douglass Shaw, of the University of Nevada, however, saw things differently, provoking the following cross-examination (edited for brevity; the italicized portions are my editorial comments): Q: I believe you said that you agreed that one of the starting points for looking at [the relationship between river flow and salmon survival] would be . . . ordinary least squares analysis comparing flow and survival, correct?
Q: And . . . as part of your practices and procedures you'd look at the raw data of this sort in the course of assisting yourself in designing a particular statistical analysis technique, wouldn't you? A: Right. Q: Okay, and over time you would develop a facility by looking at the raw data to be able to discern whether there's some sort of relationship among the variables, correct -- A: Correct. Q: -- in a general sense? So, for example, if we were to look at
the bottom graph here with 1996, you would agree with me that any
sort of statistical technique that we would apply to this 1996 data
to fit a linear, or indeed, any kind of relationship between these
two sets of data would tend to produce something that drooped down
over the course of the graph, as we went to the right, wouldn't it?
Q: Well, I know that. It is always misleading to look at a tiny subset of the data, and I know that there are hundreds and hundreds of data sets, and in some of them the clouds of dots will point down, and in some of them the cloud of dots will point up, but I'm asking you about the bottom one here [reproduced above], if we fit this with a statistical technique, we would show a decreasing [relationship between] survival and flow. More flow, is less survival, correct? A: Uh, gosh, I just wouldn't do it. I'm not willing to say that. Q: Okay. A: Scatter diagrams are just -- they sometimes can reveal something, but there's enough noise, what I would call noise, in this picture that I would be a little . . . Q: With all due respect, I don't think you're answering the question. The question was more of an abstract econometrics question. You can't sitting here today, fathom some sort of distribution, some sort of estimation technique, any type of thing that you could apply to this particular set of data and get out a positive relationship between flow and survival, can you, [based on] this data alone -- however misleading that may be? A: I just -- I just am not comfortable testifying to that. Q: Well, you're the expert. What kind of statistical distribution or technique would we use to get a positive relationship out of this data? A: Oh, I wouldn't ever embark on a statistical analysis with the mission of getting a particular result. Q: I'm not asking for a mission. A: Yes, you are. You asked me did I -- could I do it to get a positive result. That's saying I know what result I want to get at the outset. I do not know what the relationship is. Q: I don't want you to have any presuppositions at all. I want you to tell me whether you can imagine or point to a specific statistical technique from somewhere in the vast literature of econometrics which you could apply to this data, and get a positive relationship out of it. A: Uh, can I back up a little, and say how I would proceed to use this data? Q: No. I'd rather you answer the question. A: Okay. Then the way you've asked it, yes, I can envision a possible analysis to undertake where it ended up that the outcome was showing some positive relationship. Q: From this data alone? A: Yes.
A: The first thing in assessing the data would be -- I would have to understand what does the block mean. What does percent survival mean? How was that data obtained? How is it that you have a whole bunch of blocks for one year? Now, I don't know the answers to those questions . . . . Q: I believe that Doctor Olson testified that these are from cohort release groups at Lower Granite Dam, and so we can see that over the course of a set of time over this year that is put forth in this graph here, we released a number of cohorts and they experienced different flows and different survival. So, we have that established as a non-hypothetical fact. Q: Do you need more clarification, Doctor Shaw? A: Yes. Q: Okay, what do you need? A: Each block, are they all fish that are counted at one location on the river? Q: I will ask you to assume for the purposes of the hypothetical that they are released in one block at Lower Granite and therefore, counted there, and then detected down the river at different dams. A: Okay. So, the point where they're counted to ascertain the survival rate is at different locations on the river? Q: It may well be, yes. A: Okay. Then the second question, are the blocks obtained, the data that form the blocks, obtained at different points in time during the year? In other words, different months? Different dates? Q: Obviously, yes. A: Okay. Let me think if there's anything else I need to know. Yeah, I would probably ask is the procedure used to count the fish exactly controlled, so that we know it's identical at every single point on the river where the fish were counted? Q: Oh, it's an automated mechanism. They do the best they can. A: Are there errors in their counts? Q: No biological sampling is perfect, as you well know. A: All right. So, each block may have some error associated with it, itself. Q: Indeed. A: Okay. All right. Okay. I think that's giving me enough information now. Q: What particular statistical technique do you apply that you believe would produce a positive flow/survival relationship out of this particular data set? A: Again, I would never answer a question where you said I have to get a positive relationship. If you want to ask me what statistical method would I use to analyze the data, I can answer that. Q: Well, then we've gone around in a big circle here, because the first question I asked you was can you conceive of any type of statistical analysis of this particular data set, that would produce a positive relationship. A: And I said, yes, I could conceive of that. Q: Okay, what is it? A: Okay. In the literature in economics we actually borrow a technique
that is very sophisticated, and we borrow that from biostatisticians
and that's called a survival function approach. Modification to a survival model would also then allow for the possibility that what you have here is a situation where you have essentially, a time series of data. So, for each observation in statistical analysis you measure these things with what we call error. Complicating this problem, almost certainly, is the fact that the errors are gonna be correlated over time. That means, almost certainly, that ordinary least squares regression would be inappropriate. So, I would embark upon -- the methodology I would probably embark on, would be the basic survival model using what we know from time series analysis about possible correlations of the errors throughout time, and that would be a fairly complicated analysis to do. (Here Dr. Shaw tries the common "expert snow job" technique, throwing out a lot of complicated concepts in an attempt to confuse the questioner.)
A: And I just told you what it is. Q: Okay, what is it? Can you express it grammatically? A: I just told you I would use a survival approach with correlated errors, and I'd run the model and see what happened. Q: And what's the function? A: Pardon? Q: What's the function? Survival equals what, as a function of flow? A: Oh. Well, in those models it's not a linear relationship at all, but you're basically -- sure, you're still categorizing the model as the survival probability is a function of the flow. That's the way those models would work. Q: Uh-huh. What's the function? A: Right. Q: It's a nonlinear function you've told us. What is it? A: Oh, you want to know exactly what it is? Q: Yeah. You're the expert. Go ahead, tell me. A: Okay. What you have to do in survival modeling is you have to
do what's called parameterizing the independent variable. The mathematical
function for the probability distribution is probably gonna be exponential.
The reason that it's exponential is that we think most biological
organisms grow exponentially. Therefore what the probability function
does is it links together over the time periods the probability that
the fish can survive in a mathematical form that is an exponential
function. Now, it's not as simple as that, because the next step is
you have to figure out what's called parameterizing the flow, and
when you parameterize the flow you have to use an assumption about
the relationship of flow in terms of the mathematical affect it has
on the survival. They're fairly complicated models.
A: I just told you. Do you want me to get up there and try to write it out? I don't think I can do that right now, but you're asking me to describe on the witness stand, a mathematical function that is fairly complicated. Q: Well, you would agree that what we're engaged in here is attempting to fit a function to the data, correct? A: Right. Q: Can you draw the line that you would draw through the data with such a result? A: Uh, no, because exponential functions take different shapes depending on what we call the scale perimeters in the functions, and so exponential functions actually can have long tails in one direction and the empirical data would help us determine what's called the scale perimeter and the function and then that lets me know does that thing look like this (indicating) or does it look like (indicating), is it flatter than that? They can actually have very, very different looking shapes by the time you estimate them with the data. So, that's actually the beauty of them in a way, because the linear regression model that Doctor Olson used only allows one possibility, and that's a straight line. The exponential function could take any number of different shapes generally in the exponential family, and they can have very different shapes. Q: And they're all curved, right? A: Yeah. Q: Okay. Now, you would agree with me no matter what we're doing if we're fitting a curve to this data it is going to lie somewhere between the top and the bottom, correct? A: Uh, again, my best guess is even if you used an exponential function on the 1996 data that you used, that you depicted here by itself with no other data, you're probably not going to fit it with a nonlinear function and find a very clear picture of any positive relationship, okay? But the -- Q: The shape of the curve is gonna be down, correct? A: It may be flat. Q: Flat or down? A: Flat or down. Drawing the truth out of a witness is a time-consuming and often tedious process, though lawyers typically enjoy it. Even in the unusual procedural circumstances where cross-examination is allowed, most judges do not have the patience to allow the lawyer the very large amounts of time needed to pin down evasive experts on any particular point, and there are many, many points to be nailed down. Even in the rare cases where cross-examination is allowed, an evasive expert will provoke most judges to tell counsel to "move along" before he or she can get anywhere near demonstrating that the government's experts are lying. This case settled, so the Judge never got to write an opinion on the flow/survival issue. As far as I know, no neutral factfinder, whether judge or jury, has ever listened to evidence on Columbia River flows and fish survival, with live testimony of witnesses subject to cross-examination, and then made a decision. We have crippled economic development and wasted literally billions of dollars on measures to manipulate river flows for fish, all the while employing fewer factfinding procedures than we do when a school bully gets expelled. We do this because judges declare that they are supposed to accept the conclusions of government experts, without cross-examination. This has produced a culture where government experts routinely lie, because they know they will never be held accountable for it. Thus we have flow targets that purport to require more river flow
than Nature ever delivers, "endangered" species is no danger
of extinction whatsoever, and a whole host of other measures foisted
upon us by lying experts, accountable to no one. |
