1002 UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLUMBIA ELOUISE PEPION COBELL, et . al. . . Plaintiffs, . Civil Action 96-1285 . v. . . Washington, D.C. DIRK KEMPTHORNE, Secretary . Tuesday, June 17, 2008 of the Interior, et al. . 1:33 p.m. . Defendants. . . . . . . . . . . . . . . . TRANSCRIPT OF TRIAL - AFTERNOON SESSION BEFORE THE HONORABLE JAMES ROBERTSON UNITED STATES DISTRICT JUDGE APPEARANCES: For the Plaintiffs: DENNIS GINGOLD, ESQ. Law Offices of Dennis Gingold 607 14th Street, NW Ninth Floor Washington, D.C. 20005 202-824-1448 ELLIOTT H. LEVITAS, ESQ. WILLIAM E. DORRIS, ESQ. Kilpatrick Stockton, LLP 1100 Peachtree Street Suite 2800 Atlanta, Georgia 30309-4530 404-815-6450 KEITH HARPER, ESQ. JUSTIN GUILDER, ESQ. Kilpatrick Stockton, LLP 607 14th Street, NW Suite 900 Washington, D.C. 20005 202-585-0053 APPEARANCES con't. on next page. Jacqueline M. Sullivan, RPR Official Court Reporter 1003 APPEARANCES, con't. DAVID C. SMITH, ESQ. DANIEL R. TAYLOR, JR., ESQ. Kilpatrick Stockton, LLP 1001 West Fourth Street Winston-Salem, North Carolina 27101 336-607-7392 For the Defendants: ROBERT E. KIRSCHMAN, JR., ESQ. JOHN WARSHAWSKY, ESQ. J. CHRISTOPHER KOHN, ESQ. U.S. Department of Justice 1100 L Street, NW Washington, D.C. 20005 202-307-0010 JOHN STEMPLEWICZ, ESQ. Senior Trial Attorney U.S. Department of Justice Commercial Litigation Branch Civil Division Ben Franklin Station P.O. Box 975 Washington, D.C. 20044 202-307-1104 Court Reporter: JACQUELINE M. SULLIVAN, RPR Official Court Reporter U.S. Courthouse, Room 6720 333 Constitution Avenue, NW Washington, D.C. 20001 202-354-3187 Proceedings reported by machine shorthand, transcript produced by computer-aided transcription. Jacqueline M. Sullivan, RPR Official Court Reporter 1004 1 P R O C E E D I N G S 2 COURTROOM DEPUTY: Please come to order and remain 3 seated. 4 THE COURT: All right, Mr. Dorris. 5 MR. DORRIS: Thank you, your Honor. 6 FRITZ SCHEUREN, Ph.D., WITNESS FOR THE GOVERNMENT, 7 PREVIOUSLY SWORN 8 CROSS-EXAMINATION 9 BY MR. DORRIS (continuing): 10 Q. Good afternoon, Dr. Scheuren. 11 A. Good afternoon to you. 12 Q. Let me move to a little different subject than where we 13 were before the lunch break. You testified some about changes 14 that were made in some of the information that was given to us 15 from -- some information that was provided on May 30th to us and 16 then information that was provided to us on June 4th. Do you 17 recall testifying about that during your direct? 18 A. I do. 19 Q. I'm going to ask that we bring up on the screen Plaintiffs' 20 Exhibit 117. Maybe I've got the wrong number. 21 Can I have just a minute, your Honor? 22 (Pause in the proceedings). 23 MR. DORRIS: I'm sorry, your Honor. I had written 24 down the wrong exhibit number. Oh, here it is. Do you show 25 what exhibit number that is? Okay. Jacqueline M. Sullivan, RPR Official Court Reporter 1005 1 BY MR. DORRIS: 2 Q. I believe it has been marked as 117. And in looking at 3 this I want you to look at the bottom two that are referenced at 4 AR-171R1, the May 30th, and then AR-171R2, which is the June 4. 5 Do you see those? 6 A. Yes. 7 Q. Okay. And if you look over, and this is just revenues, 8 okay, this doesn't address disbursements or other information. 9 Do you see that between the information we were provided on May 10 30th showed total revenues in the pre-1972 time frame of almost 11 $3.4 billion? 12 A. I do. 13 Q. And then in as of June 4, 2008, they'd fallen to about $2.8 14 billion. Do you see that? 15 A. I do. 16 Q. Okay. Now, I don't believe that there were any changes in 17 what we were provided between those two between 1972 and the end 18 of the chart, which I think is 2007. Do you know whether there 19 were any changes in that later time frame? 20 A. To my knowledge, no. 21 Q. Okay. So all of the changes that occurred between these 22 two runs or whatever occurred -- made changes in the pre-1972 23 time frame, fair? 24 A. Yes. 25 Q. Now, you testified, and I tried to follow but I couldn't Jacqueline M. Sullivan, RPR Official Court Reporter 1006 1 exactly, why there were further changes made between May 30 and 2 June 4, 2008. Did I understand you to say that the process you 3 were going through simply wasn't complete on May 30, 2008? 4 A. That is correct. 5 Q. And did you get it completed by June 4th? 6 A. I believe it is. 7 Q. Now, what changes, if you know, were made between those two 8 analyses that would cause about $600,000,000 in revenues to go 9 down during that period, the pre-'72 period? 10 A. We did two things. I already said some of this to you. We 11 looked at the imputation model and we improved it. We were not 12 happy with the -- well, with how good it was working for us, and 13 we improved the model. And then we also looked at the reported 14 data, okay, which we had not factored into the uncertainty in 15 the unreported data at that point. 16 Q. Okay. So you looked at it, the model, because you weren't 17 happy with it, correct? 18 A. That's correct. 19 Q. Why weren't you happy with it? 20 A. The disbursement -- it was getting negative values for the 21 balances, and this is over the whole life of the trust, that 22 couldn't be right, so we tried to find out what it was, and I've 23 already mentioned to you that one of the -- it doesn't sound 24 like it's a big deal, but one of the things that hit us was we 25 had used the value in 1887 as if the trust started in 1887, Jacqueline M. Sullivan, RPR Official Court Reporter 1007 1 which in fact in some senses it didn't. 2 Q. Oh, okay. So that was one thing you did talk about, how? 3 A. This morning. 4 Q. And the model that was run that generated the May 30th 5 information we were given, are you saying that that was 6 presumably a zero balance as of 1887? 7 A. That is my recollection. 8 Q. Okay. You've never produced the analysis for May 30 to us, 9 have you? 10 A. No, I have not. I think I also said this morning that we 11 used only a small number of iterations, only a thousand, in the 12 May 30 data. Like ten thousand. 13 Q. So between May 30 and June 4 we've identified a couple 14 things, and keep me straight. We've identified that you've now 15 put a balance in for 1887 and you ran ten times the number of 16 imputations, correct? 17 A. I didn't put a balance in. I did -- I didn't force a 18 balance of zero, okay, in 1887, which we had done earlier. 19 Q. Okay. You didn't put a balance in, but you let this model 20 that you created generate a value for that missing information? 21 A. Correct. 22 Q. And it generated a value of $5.8 million balance in 1887, 23 correct? 24 A. Yes. 25 Q. Now -- Jacqueline M. Sullivan, RPR Official Court Reporter 1008 1 A. That needs to be footnoted, though. 2 Q. That needs to be footnoted? 3 A. Yes, because of the way we did this. 4 Q. Can I ask this, it's not footnoted on anywhere on what 5 you've done, is it? 6 A. No. 7 Q. Okay. You said it needed to be footnoted. Would you say 8 what you mean? 9 A. What I need to do is explain to you that we actually had to 10 make a starting value, okay, in this area, and grew out of the 11 model that we chose, which we had to go back seven years, as you 12 remember this from this morning, and were not showing all those 13 seven years. We're just showing first of the years, 1887, so 14 that number is coming from, you know, coming from a process you 15 haven't fully seen. 16 Q. Okay. Now, so there was some analysis, further analysis 17 between May 30 and June 4 for a time period between 1880 and 18 1887, correct? 19 A. Yes. 20 Q. Where you were finding that or establishing in some fashion 21 that there was money coming into the government for individual 22 Indian allotted lands, correct? 23 A. I don't know how to answer that. We were not really doing 24 that. We were trying to fix this model so it worked for us. If 25 you look at these numbers here you will see that the collection Jacqueline M. Sullivan, RPR Official Court Reporter 1009 1 numbers for 1887 were 2.2 million and disbursement 2.1 million 2 and that meant there was a balance increase for that year of 2.1 3 million, so where does the other money come from? It obviously 4 comes from the past, okay. It has to come from the past, and 5 this issue is one that we're -- we don't have -- we actually -- 6 it comes out of the process that you haven't seen. 7 Q. Okay. And this process I haven't seen would indicate to me 8 that on average over a course of six or seven years there is 9 approximately $5.8 million balance that is generated through 10 that process, correct? 11 A. That is what is happening, yes. 12 Q. Okay. Am I correct when I look at the top line of 13 Defendants' Exhibit 462, that collections of 2.2 million less 14 disbursements of 2.1 million ends up with a balance at the end 15 of fiscal year 1887 of $5.8 million, am I reading the chart 16 correctly? 17 A. You are correct. 18 Q. So the beginning balance at the end of fiscal year 1886 19 under this would be 5.7 million, correct? 20 A. If it's footed, yes. 21 Q. Now, for those seven years then the average amount would be 22 approximately eight hundred to nine hundred thousand dollars in 23 a buildup of the balance, correct? 24 A. I think I need to give you more information than this, but, 25 yes, that is a logical conclusion, yes. Jacqueline M. Sullivan, RPR Official Court Reporter 1010 1 Q. And then after that there isn't another year where this 2 model that you have constructed would show that much build up in 3 the balance until which year? 4 A. Well, in looking at the data it looks like 1903. 5 Perhaps -- I'm not sure what you mean. 5.8 to 10, what is that? 6 Q. Well, it looks like to me there's a $400,000 difference 7 between collections and disbursements in 1903. 8 A. Yes, but you're not looking at the balances. The 9 difference between the two balances is $3 million. 10 Q. Oh, okay. So you're saying, well, look at the balances, 11 don't look at the collections and disbursements? 12 A. Well, I won't say don't. Don't just look at the 13 collections and disbursements. We do not -- this system does 14 not foot until 1996, and the balance information is in the 15 system but it's not -- is tied -- perhaps it's timing or 16 whatever it is, but it's not tied closely. 17 Q. You said it didn't foot until 1996? 18 A. That's correct. 19 Q. Did I understand you correctly? 20 A. That's what these data show. 21 Q. That's what this model shows, correct? 22 A. Well, there is two ways to answer that. One of them is in 23 those years we have data, collections, disbursements, and 24 balances. The only years which we have data in foots begin in 25 1996. Jacqueline M. Sullivan, RPR Official Court Reporter 1011 1 Q. Okay. Now, is that something that you set the model up to 2 do, to make sure it footed as of 1996? 3 A. No. It does. That's real data. 4 Q. Okay. Then let me ask this: The data for 1996 is real 5 data, is what you're saying? 6 A. Yes, sir. 7 Q. And is that different than the other data that you show 8 here as reported values? Are you treating that different in 9 some fashion? 10 A. You're right in that some of the data, the data that's in 11 the boxes, okay, was reported data, but has been model adjusted. 12 The data that was not in the -- does not have a box around it 13 and is not color-coded is reported data. 14 Q. Okay. So when I look under the columns Collections and 15 Disbursements on Defendants' Exhibit 462, everything that was a 16 reported value earlier now shows as being a model adjusted value 17 all the way up until 1996, correct? 18 A. When we're talking about collections and disbursements, 19 what you said is true. The balances were not model adjusted. 20 They were as we left them after imputation. 21 Q. You didn't try to adjust those in the same way that you 22 tried to adjust collections and disbursements, correct? 23 A. That's correct. They were there to help us model 24 disbursements and collections. 25 Q. You said something during your direct examination, and I'm Jacqueline M. Sullivan, RPR Official Court Reporter 1012 1 talking from my notes, not from the transcript, but it was 2 something about that the model was so unstable that you treated 3 1996 and beyond as actual data, at least that's the gist of what 4 I heard you say. 5 A. That is not what I said. 6 Q. Okay. Is that true? 7 A. Let me separate the truths that you're saying into pieces 8 and then see what the causal links are, if there are causal 9 links. May I start? 10 Q. Please. 11 A. The first thing to remember is that the 1996 data, which 12 was very useful, has been audited it. All the data since 1996 13 has been audited and all of it is footed. It behaves in the 14 proper way, is an accounting system. The earlier data does not 15 behave in that way, which made us suspicious of it. We took the 16 1996 data as useful in understanding the earlier data, but we 17 didn't change it. 18 Q. Okay. Did you say, did you talk at all during your direct 19 examination about any instability in the model as it relates to 20 these years 1996 and beyond? 21 A. No, not about that year. 22 Q. Now, so you are taking from 1996 through 2007 because there 23 are audited financial statements, you're accepting those as 24 being correct? 25 A. I'm accepting those, yes. The word "correct" is your word. Jacqueline M. Sullivan, RPR Official Court Reporter 1013 1 Q. Okay. When you say you're accepting them, is there still a 2 part of your analysis that assumes any uncertainty with respect 3 to the data from 1996 forward? 4 A. No. 5 Q. Okay. So you are not viewing that as being uncertain in 6 the least? 7 A. It is not part of the modeling process, correct. 8 Q. I did not understand what you said. 9 A. It is not -- our focus of the modeling. 10 Q. Can you just repeat your answer? 11 A. I can't do that exactly but I'll say it differently maybe 12 and that will help you. 13 Q. Okay. 14 A. And of course you'll have the transcript too to compare 15 what I said the first time to what I said now. 16 We know the data was audited in 1996. It foots, it 17 makes sense in terms of the earlier data, okay, for use earlier 18 and collections and disbursements. And we took -- we took that 19 data as a given and not a need of any further adjustment, either 20 an imputation adjustment, because there was nothing missing, or 21 model adjustment because of the possibility that there was some 22 errors still in that data. That is not to say that there 23 couldn't be errors in that data, we do not know that, but we 24 were so behaved so well that we did not do any modelling for 25 that. Jacqueline M. Sullivan, RPR Official Court Reporter 1014 1 Q. Okay. I got off on this as I was trying to find out what 2 went on in these two different analyses for the pre-1972 period, 3 and let's go back to what I'm calling Plaintiffs' Exhibit 117. 4 Now, you told me that part of the reason these changed had to do 5 with the -- that you included 1980 through 1986 in your 6 analysis, correct, is one reason that the numbers changed? I 7 may have said 19. I should be saying 1880 to 1886. 8 A. That would help me, yes. 9 Q. I'm sorry I misspoke. Can you explain any of the other 10 reasons that the numbers changed between May 30 and June 4? 11 A. We of course are fitting a different model, okay, than we 12 were fitting earlier, all right. A model which we liked better 13 and we were comfortable with, and also I mentioned earlier that 14 we had -- we have, subsequent to the multiple imputation, we had 15 done the model adjustments for the reported values, and I'm 16 pretty sure that those were not done until after May 30th, 17 although we may have done them, but the version you got on May 18 30th did not have them. 19 Q. Okay. And when you talk about the model-adjusted numbers, 20 those are the ones on DX 462 that show up in the black boxes? 21 A. That's correct. 22 Q. Okay. And certainly the addition and subtraction of those 23 don't come anywhere, the addition of all the additional revenue 24 reflected here, are you saying that that more is shown in those 25 boxes than that that explains a $600 million difference? Jacqueline M. Sullivan, RPR Official Court Reporter 1015 1 A. Let me try to answer the question in a somewhat different 2 way. I am not claiming that the difference, numerical 3 difference between these numbers is due to A, B, C and D. What 4 I am telling you is what A, B, C and D, which is what we did, 5 occurred between these two numbers. We went away from this 6 model. We were in happy with it. And the fact that we're 7 continuing to talk about it is your prerogative, but I'm not 8 happy with this model. 9 Q. And can you list to me the primary reasons you were not 10 happy with the model of May -- that resulted in the May 30 11 numbers? 12 A. Well, I told you two of them and I'll mention them again. 13 And to order them in terms of a report I don't know if I can do 14 that right now. First thing is that we're getting some negative 15 values from the earlier value. 16 Q. Say it again. 17 A. Negative values. The balance over the whole life of the 18 trust was negative, and that didn't make sense. That did not 19 make sense. And the second reason was that we had in 1887 20 things, so as soon as we got rid of the 1887 problem that we 21 treated as a zero the model behaved in a way that made sense, 22 and then we went on to do the model-adjusted steps for the 23 reported values. I didn't realize that I was going to have to 24 be asked to talk about this. I'm happy to talk about it of 25 course and then maybe I can give you a better answer later, but Jacqueline M. Sullivan, RPR Official Court Reporter 1016 1 this is the answer I have right now. 2 Q. Okay. Well, let's do move on. Have you ever applied this 3 multiple imputation model to a situation such as this one? 4 A. The degree to which the situations are similar is in the 5 eye of the be holder. I've applied multiple imputation in lots 6 of situations. Exactly like this, never. 7 Q. Okay. Now, as I have read your article that was produced 8 today and other articles on multiple imputation, it originally 9 grew out of trying to impute values for missing data in 10 extremely large surveys, correct? 11 A. Correct. 12 Q. And those values needed to be imputed in those surveys so 13 that then-existing techniques for studying a completed data set 14 could be run on those sets, correct? 15 A. That is correct. Those techniques of course are still 16 being used. 17 Q. And that's the reason why in those surveys that you want to 18 impute or put some value in for the missing data? 19 A. The value of the imputation, the multiple imputation -- 20 important word -- was to get to uncertainty, but the analysis 21 had to be done, especially given this offer in those days, was 22 to make the data look like a complete data set because the 23 methodologies for those complete data sets already existed. 24 Q. Okay. You say that similarities or whatever are in the eye 25 of the beholder. Let me ask you, as the beholder, can you Jacqueline M. Sullivan, RPR Official Court Reporter 1017 1 describe for us the closest situation to this that you have ever 2 used multiple imputation for? 3 A. I wish I had known you were going to ask that question. I 4 would you have thought about a better answer, but most of my 5 applications have been in administrative records when I was at 6 the IRS, and in surveys. 7 Q. Okay. Would it be fair to say that this is the first time 8 that you've used multiple imputation in trying to develop 9 missing values for a trust or banking situation? 10 A. That's a bit broad, but it's certainly true that I've not 11 done this kind of analysis before. 12 Q. Now, when you look at -- you used a phrase during your 13 direct, a word that I guess is one of those words that 14 statisticians come up with. You said the "missingness" of 15 certain data. Do you remember using that word? 16 A. Yes, I did. I do use it, yes. 17 Q. Now, if you would bring up Defendants' Exhibit 461. Can 18 you describe to me what you mean by the missingness of data? 19 A. Well, there are multiple measures of that that you can 20 have. One of them is simply to look at the number of missing 21 cells relative to the total number of cells. Another one is to 22 look at the dollars in the imputed data and in the reported 23 data, how many of the dollars have been imputed. That's another 24 kind of degree of missingness, okay. 25 Q. Okay. Let me ask before we go through, you don't know what Jacqueline M. Sullivan, RPR Official Court Reporter 1018 1 the imputed information is when you've got Defendants' Exhibit 2 461, correct? 3 A. That's correct. 4 Q. And when you make an assessment, for example, of the first 5 page of Defendants' 461, how would you characterize the 6 missingness of the data on that chart? 7 A. Considerable. It's considerable. 8 Q. Is there a point where you have so much missing data that 9 multiple imputation is not a good device to use? 10 A. I'm sure there is. 11 Q. And we're at that point if this was the only page that we 12 had in Defendants' Exhibit 461? 13 A. I admit we are, yeah. 14 Q. Look at the next page. If I told you on the next page, Dr. 15 Scheuren, to assume that all of the values that are stated for 16 collections between the years 1927 through 1949 are incorrect 17 and you should disregard them, in other words, if all of that 18 was yellow, would you agree with me then that there was too much 19 missingness on that page to use for a multiple imputation 20 process? 21 A. 1927 to 1949, is that what you said? 22 Q. Yes, for the collections column. You have reported values 23 there and I've asked you to assume those are all missing and are 24 yellow. 25 A. Let me make two comments. First is to compare the first Jacqueline M. Sullivan, RPR Official Court Reporter 1019 1 page to the second page if you would let me, and then second is 2 to answer your question directly. 3 Q. Yes. Would you show -- 4 A. I don't need to see it again. There is considerably more 5 data here even if those, that second column, disbursement 6 column, were yellow, than there is on the first page. What is 7 weakened here greatly is the relationship between collections 8 and disbursements in that second -- well, it's the third column 9 on the page if the disbursement column were yellow, so my answer 10 here on this page was that it depends on what it is you are 11 trying to impute here. What's your final number? If you're 12 into, and we are, into a balance figure displacement, 13 collections minus disbursements, then I would be worried about 14 this probably if that were the case. 15 Q. Okay. To make clear, your answer when you referred to the 16 second column, you're referring to the column entitled 17 Collections, and when you refer to the third column you were 18 referring to the column entitled Disbursements, correct? 19 A. Yes. And your question was about the disbursement column, 20 I believe. 21 Q. No, it was about the collection column, that if you -- 22 A. Oh, I'm sorry. 23 Q. That if you did not -- let's make sure it's clear because I 24 think your answer is the same. But if on the second page of 25 Defendants Exhibit 461 they reported values in column two that Jacqueline M. Sullivan, RPR Official Court Reporter 1020 1 is labeled Collections were all missing between 1927 and 1949, 2 that was the premise of my question. Did you understand that? 3 A. Now I do. I misunderstood. I thought it was disbursement, 4 but I'll say it a different way as I think as you've already 5 anticipated my answer here. 6 If one or the other of these columns were all missing, 7 then what I said would be correct. 8 Q. And what you said would be correct is that then there would 9 not be sufficient information to use a multiple imputation model 10 to try to project a missing data, correct? 11 A. If I might interject a word, the word "strong model." I 12 can still model it, but I don't have enough feedback in the data 13 to have me believe the model. The data is not telling me that 14 the model can be relied on. You can always make strong 15 assumptions if you want, but I wouldn't have used that, no. 16 Q. So that when what you're saying is you could still model it 17 but you wouldn't find that to be reliable? 18 A. If you make too many assumptions you have to worry. 19 Q. Okay. Now, you've been in court on and off since the trial 20 started, correct? 21 A. Yes, I have. 22 Q. And are you aware of there being any questions or concerns 23 about the collection information that has been used in the 24 period from roughly 1922 or 3 all the way through 1949? 25 A. I'm afraid you're going to have to refresh my memory here. Jacqueline M. Sullivan, RPR Official Court Reporter 1021 1 I'm not sure what I was hearing. Some of the time I wasn't 2 paying as much attention as at other times. 3 Q. Let's pull up Defendants' Exhibit 372, and look at page 4 eight. 5 I've called out the wrong page. Go back -- well, hold 6 on just a second. Go back to page four, please. And then 7 let's -- 8 Your Honor, may I get my notebook? I'm sorry, I don't 9 have it at the podium. 10 THE COURT: Sure. 11 BY MR. DORRIS: 12 Q. Page ten, Defendants' Exhibit 372, page ten, and I'll tell 13 you what. If you blow up the top part so that Dr. Scheuren can 14 see that. And you see that this particular chart is referring 15 to the years 1934 through 1949. Are you with me? 16 A. Yes, I am. 17 Q. And then it gives a document reference to the far right 18 column, footnote two. Do you see that? 19 A. I do. 20 Q. Okay. And then if you would blow up footnote two. 21 Dr. Scheuren, we've read this many times. I would ask 22 you to read footnote two to yourself. 23 A. Yes, I've read it. 24 Q. Those indicate that the reports on which the collections 25 for those years are being based likely understate total Jacqueline M. Sullivan, RPR Official Court Reporter 1022 1 receipts. Do you see that? 2 A. I do, but I remind you that you're looking at tribal here, 3 not the total. 4 Q. Okay. 5 A. Pull back up to the top. You're looking at tribal. 6 Q. Okay. Let's go there. Go to 1934 and kind of blow up the 7 top year so we get it nice and large. 8 You say it's highlighted there. The receipts that are 9 being referred to there, sir, are receipts, those are receipts 10 that were used -- well, you tell me. What did you use in your 11 model on -- let's just look at this number. See the 10.6 12 million? 13 A. I see it. 14 Q. And that's for 1934. You're with me, the 10.6. 15 Let's pull up Defendants' Exhibit 461 and go to the 16 second page. 17 And do you see under 1934 you used as a starting, as a 18 reported value 10.6. Are you with me now? 19 A. I do see the coincidence, yes. 20 Q. That's not tribal, is it? 21 A. No, it's not. 22 Q. And do you see there where we saw a footnote there that 23 indicated that the information from which that was taken and 24 then a statement that Ms. Herman I believe testified that she 25 and Dr. Angel prepared together that says that all those revenue Jacqueline M. Sullivan, RPR Official Court Reporter 1023 1 figures are likely understated, were you aware of that before 2 right now? 3 A. I heard this business about understatement, of course. 4 This connection that you've just made between these two exhibits 5 I am hearing and seeing for the first time. 6 Q. Go back to page -- on Defendants' Exhibit 372 to the 7 first -- to page four. And blow up kind of, I need to go to the 8 other -- over to the Other Receipts column, the first seven or 9 eight lines. Do you see over there under the Other Receipts 10 column here on this front page where it is showing that you look 11 at schedule A-1 page one in order to get the numbers that are 12 there? 13 A. Apparently there is come kind of a footnote here that I'm 14 not seeing. 15 Q. Okay. And then do you -- 16 A. So my answer to your question is no, I do not see it. 17 Q. Would you point the arrow under Tribal IIM to the footnote 18 A-1-1? Do you see where the red arrow is pointing now? 19 A. I do. 20 Q. Do you see the footnote? 21 A. No, I haven't seen the footnote. Let me see the footnote. 22 Q. No, but you just see the reference to A-1 page one? 23 A. Yes, sir. 24 Q. And when we turned to A-1 page one in this exhibit, it is 25 page number eight. Are you with me? Jacqueline M. Sullivan, RPR Official Court Reporter 1024 1 A. I'm getting there. 2 Q. And then do you see for Tribal IIM, if you'll point the 3 arrow at it, the footnote, you see Tribal IIM and then you see 4 the footnote next to it that you look at page A-1.1 page one? 5 A. Um-hmm. 6 Q. And then if you go to page ten, and that's the page we were 7 just looking at with the note on it, do you see that? 8 A. Yes. 9 Q. And that has to do with A-1.1. So do you see how this 10 document fits together? 11 A. Yes, but you're moving a little fast for me. Remember my 12 vision is a little slower than that, but, yes, I see this. I 13 commend you for putting this together. Thank you. 14 Q. Let me ask it to you in a question this way since you're 15 testifying as an expert statistician. If I ask you to assume 16 that all of the collection information shown on Defendants' 17 Exhibit 461 as reported values from 1923 through 1949 are likely 18 understated, what modifications or adjustments, if any, would 19 you make to what you have done? 20 A. If that's the case then I would have wanted to have 21 adjusted them up to account for that before I did the rest of 22 the analysis. 23 Q. And that was not done, was it? 24 A. It was not. We did take into account uncertainty in the 25 data, as you heard me tell you, but we did not expect to model Jacqueline M. Sullivan, RPR Official Court Reporter 1025 1 any bias in the data. 2 Q. Okay. I don't understand the last phrase you did not 3 expect any modeling bias in the data. What does that mean? 4 A. I don't think I quite said that. 5 Q. Tell me what you said. 6 A. Suppose that every one of those values was off by $3 7 million, okay, every year. 8 Q. Off by what? 9 A. By $3 million, just hypothetically. You give me a 10 hypothetical I give you one back, okay? So off by $3 million 11 and I was able to learn that somehow. I mean, enormously expert 12 people in the room here at Angel and Michelle Herman, and if 13 they had come to me and said these data are off by $3 million I 14 would have adjusted them upwards by $3 million. I would have 15 told you that. 16 Q. And if they were not able to come up with a reasonable 17 estimate for you as to how understated those values were, what 18 would you do? 19 A. I would not have been able to have done any more than I 20 did. 21 Q. So you would have run them in the model just like you have, 22 even though they're -- 23 A. If I knew they were consistently understated I would have 24 developed an adjustment process that was asymmetric. The 25 process we used assumed errors in reporting in both directions. Jacqueline M. Sullivan, RPR Official Court Reporter 1026 1 I would have adjusted it only reporting errors in one direction. 2 I did not do that, though. 3 Q. All right. And those are adjustments that you would have 4 made before you ran the model, correct? 5 A. What we actually did, of course, is we took out the really 6 extreme outliers as we thought them outliers, and then knowing 7 they were reporting errors we went ahead and did the imputation 8 and then we went back in afterwards and adjusted the reporting 9 and imputed the data. And the reason we did that is because the 10 system is so sparse that we needed the imputed data to help us 11 to decide how to adjust the reported data and the imputed data 12 together. 13 Q. So that even with all of the reported values that you've 14 used such as they are, the data was so scarce that you actually 15 needed to run an imputation and impute values in order to then 16 make further changes to the model? 17 A. To the reported data, yes. 18 Q. To the reported data? 19 A. And the imputed data at the same time. 20 Q. Let's go back to Defendants' Exhibit 462. Do you see the 21 years 1909 through 1911 are now in black boxes as being model 22 adjusted values? Do you see that? 23 A. I of course do. 24 Q. That was some of the reported value you had before, 25 correct? Jacqueline M. Sullivan, RPR Official Court Reporter 1027 1 A. Correct. 2 Q. And explain to us why the model adjusted the values for 3 these reported amounts. 4 A. What we had done is we had fit a time series model to the 5 whole data set after we had done the imputation and identified a 6 degree to which the model did not fit, and then we took the 7 residuals from the model and we treated those as a way of 8 thinking about the remaining uncertainty in the reported values, 9 and then we used those residuals to increase the amount of our 10 measure of uncertainty. Remember that our focus is not really 11 on the model values at this level. Our focus is on the overall 12 uncertainty at the balance level. That was our goal. If we had 13 a goal of producing a best estimate for 1918, okay, collection 14 and disbursement, we would have done additional work localized 15 to that year. 16 Q. So if you were trying to come up with the best estimate for 17 19 -- let's just say 1912, what additional work would you have 18 done to try to come up with the best estimate for those years, 19 for collections and disbursements? 20 A. I don't know the answer to that right now. Do I have to 21 know the answer? I mean, you're asking hard questions. You 22 know, at least they have hard answers from my world. 23 Q. Well, an acceptable answer is to say "I don't know at this 24 time." 25 A. That's the answer. I don't know at this time. Jacqueline M. Sullivan, RPR Official Court Reporter 1028 1 Q. Okay. Well, let me ask this. When I compare the reported 2 values on Defense Exhibit 461 to the model adjusted values on 3 Defendants' 462, I actually find that the model adjusted the 4 collections for those three years up by about $100,000 or 5 $200,000 and the disbursements down. 6 Can you put the other one next to it? Thank you. I 7 don't know if you can then blow up those particular areas. 8 It's not a lot of numbers, but it's odd to me. Can 9 you see the screen at all? 10 A. I can see. What I hear -- 11 Q. It's coming. You see in 1909 -- 12 THE COURT: The story of this trial, you know, is 13 going to be the dualing laptops. These guys are both good. 14 MR. DORRIS: They are, and it makes it much more 15 efficient. 16 BY MR. DORRIS: 17 Q. And now, Dr. Scheuren, do you see, for example, what I said 18 in looking at DX 461 for the years 1909, 1910, and 1911 and 19 comparing that to those same years for collections in Defense 20 Exhibit 462, how each of those numbers has been increased by one 21 hundred or $200,000? 22 A. I do see that, correct, yes. 23 Q. And do you see similarly with the disbursements for those 24 three years from Defendants' Exhibit 461 to 462, the 25 disbursements were all decreased? Jacqueline M. Sullivan, RPR Official Court Reporter 1029 1 A. No. 2 Q. The disbursements were all increased by $100,000; is that 3 correct? 4 A. Thank you. Yes. 5 Q. All right. So why did the model make those kind of 6 adjustments, do you know? 7 A. Specifically why did it make those? No, I don't know that. 8 This is a model with lots and lots of issues in the underlying 9 data. I could do a diagnostic and pull up the ten thousand 10 observations under each of these and look at them and see what's 11 going on with them, but I haven't done that. You actually have 12 the data so you could do it. 13 Q. I can assure you I can't do it. But let me ask this: Was 14 that because of any uncertainty in the reported data? 15 A. The reason we did the change was because we wanted to model 16 the uncertainty in the reported data and our focus was not on 17 the point estimates, which is what you're looking at, but on how 18 much more uncertainty overall we had in the balances, and, you 19 know, I'm delighted that you're looking at it this way, but this 20 was not our goal. 21 Q. Okay. The long and short of it is, though, that the model 22 itself for reasons that you would have to run a diagnostic on 23 the computer to determine, changed the values of some of the 24 reported numbers, correct? 25 A. It did in all cases do that, yes. Jacqueline M. Sullivan, RPR Official Court Reporter 1030 1 Q. Until we get the 1996? 2 A. That's correct. 3 Q. It didn't change those after that time, did it? 4 A. We did not do any imputations in 1996 and later. 5 Q. Now, in terms of these values, if we take away that 6 comparison but leave the two slides up, do you see down at the 7 bottom of the page -- I'm not going to ask you to read the 8 numbers -- but the concept, that similarly for the years 9 beginning in 1923 that the model adjusted the reported values in 10 disbursements and collections from 1923 through, and if we'll 11 look at the second page of 462, all the way through 1949 except 12 for the disbursements in 1945, correct? 13 A. I'm trying to understand the yellow in 1945. 14 Q. Well, it was saying that's imputed for missing data. In 15 other words, no reported value in the year. If you look over to 16 the left of the charts you'll see 1945 and the -- 17 A. That's the answer, yes. 18 Q. Okay. And again you can't explain why the model changed 19 reported value in those years, can you? 20 A. I don't think that's what I said. 21 Q. Okay. Sitting here today without going and -- 22 A. In terms of a specific number, I cannot. In terms of a 23 process, I have already described it in general, and if you 24 allow me to go in and look at the data, which you all have, we 25 can do more than that. Jacqueline M. Sullivan, RPR Official Court Reporter 1031 1 Q. Let's look at Defendants' Exhibit 462, page three. Let's 2 blow that up where it sits by itself. There we go. Blow up 3 right at the top 1968. 4 Do you see where in 1968 now it is showing that this 5 model imputed a missing value for that year in collections of a 6 hundred million dollars? 7 A. I do, I do. 8 Q. And let's look at the prior page of this Defendants' 9 Exhibit 462, which would be page two? 10 Down at the bottom let's look at 1965 through 1967. 11 And do you see where in 19 -- well, you see 1964 it's $100.2 12 million. In '65 it's $93.4 million. In '66 it's $95.4, and in 13 1967 it's a hundred million. Do you see that? 14 A. I do. 15 Q. And if I told you that there was an audit report that 16 states in -- that for 196 -- as of 1968 that collections in the 17 IIM system were averaging $121 million, what adjustments, if 18 any, would you make? 19 A. Well, if I had the report I would of course have used it. 20 And we would have adjusted this data to conform with that if we 21 found it reliable. 22 Q. Okay. And if Dr. Angel considered it to be a reliable 23 statement of the average collections up to the 1968, for some 24 period you would accept that, correct? 25 A. Oh, I would have accepted that, yes. I would have checked Jacqueline M. Sullivan, RPR Official Court Reporter 1032 1 it, but I would have accepted it. 2 Q. And you would have checked it by doing what, sir? 3 A. Get into the actual report itself and understand it, 4 because often times are these subtle definitional issues, and 5 remember we're different, we think differently here, and there 6 are many different disciplines in this room right now, and I 7 apologize to all those who don't think the way I do, but I'm 8 glad that you're helping me explain myself better, but we're 9 thinking differently, even Dr. Angel and I, who have worked 10 together for a long time. 11 Q. With the information that Dr. Angel provided to you did you 12 ask him to provide you with the backup documentation for that 13 information? 14 A. For this particular year you're talking about, no, we did 15 not. 16 Q. Oh. I'm not saying he provided you any information about 17 this year, but for the information that he did provide to you, 18 did you ask him for documentary support for each of those 19 figures he provided to you? 20 A. Let me go back and raise another question. The answer to 21 your questions is yes, we did every year that Dr. Angel provided 22 me before 1972 we had full documentation for and confirmed. 23 This I think is a -- this is a -- what is it, 1960? What is the 24 year we're looking at here? What is the year we're looking at 25 here, '68? Jacqueline M. Sullivan, RPR Official Court Reporter 1033 1 Q. 1968 was the year. I started asking you questions about 2 it. Did you personally look at any of that documentation or did 3 you have your staff do it? 4 A. I looked at some of it, but no, I didn't do all of it at 5 all. 6 Q. How much of it did you look at? 7 A. Fifteen percent, twenty percent; something like that. 8 Q. If you pull up Defendants' Exhibit 72 and go to page four. 9 Your Honor, I know you've seen this many times, but in 10 light of the witness' testimony I'd like to show it to him. 11 I'm sorry. Let's go back to page one so we can 12 orient, Dr. Scheuren. This is an audit report from you see 13 March 1969 on it? 14 A. Not yet. You have to bring it up for me if you want me to 15 see it here. I do see March 1969. Thank you. 16 Q. Can you see March 1969 now? 17 A. Yes, sir. 18 Q. Let's go to page four, and if you'll bring up the 19 highlighted information there. All right. Let me let you look 20 at that. 21 A. I see that. 22 Q. And it says, Cash receipts are running at the rate of 121 23 million per year. Do you see that? 24 A. I do. 25 Q. Tell me how you would factor that into your analysis. Jacqueline M. Sullivan, RPR Official Court Reporter 1034 1 A. Well, I'd like to get behind that number because in many of 2 these reports there are statements that are not net necessarily 3 made by people close to the system. They're using overall 4 averages or impressions. But if it was a real number, after I 5 had done that review then I would have wanted to have that 6 average be met in the data. 7 Q. Okay. And the average that we've seen in the data for the 8 period coming up, that would be something between, say, probably 9 about ninety and a hundred million dollars, correct? 10 A. It's a little over a hundred in a couple years, but yes, 11 it's under this number considerably. 12 Q. And so that you would want to go then go in and put as a 13 reported value $121 million for a certain number of years 14 leading up to 1968? 15 A. I would have tried to find out what period this was in 16 reference to, but remember I have to change the disbursements 17 too. I can't just change the collections because if this number 18 is -- if the numbers I have about collections are too low and 19 they get raised, then I have to raise the disbursements too or I 20 have to have an explanation for why I can't raise the 21 disbursements. 22 Q. Do you remember that we saw the year 1945 where you had a 23 reported value of collections and no report on disbursements? 24 A. I do. 25 Q. We just looked at that. Jacqueline M. Sullivan, RPR Official Court Reporter 1035 1 A. I do. 2 Q. So for 1968 and the prior years, you could enter a value 3 for collections without putting a value in for disbursements, 4 correct? 5 A. I could have imputed it, yes. 6 Q. You could have? 7 A. I could have left it blank, but if I had the numbers, if I 8 had both numbers, disbursements and collections, I would have 9 gone to Ed and asked him for his advice about what to do. 10 Q. And go back to the first page of this exhibit and blow up 11 the title at the top. 12 And do you see that this was done by the Office of 13 Survey and Review for the Department of Interior? 14 A. Yes. 15 Q. And you would expect them to be knowledgeable of the IIM 16 system? 17 A. I would. 18 Q. And this is a change that we look at here today that you 19 could make to your system still, couldn't you? You could go run 20 it, go back to the office and run that this evening, right? 21 A. This evening maybe. Certainly we could run it quickly 22 after we had confirmed it with discussions. 23 Q. Another area where there was some examination about the 24 accuracy of the information that was used by Ms. Herman was in a 25 period that you talked about, and I think you've talked about it Jacqueline M. Sullivan, RPR Official Court Reporter 1036 1 as the CD&L information that was a period between 1972 and 1985, 2 correct? 3 A. Correct. 4 Q. And what I understood you to say is that as a result of 5 hearing that testimony you did some further analysis about the 6 missing information on the spreadsheets that were used to 7 support her numbers and therefore you prepared a new histogram 8 the day that we looked it; is that correct? 9 A. Let's me say it a little bit differently. I already 10 commended you and thanked you for pointing those out, and when 11 we went back in and looked at that as you pointed it out to us 12 we did not change the histogram itself. I used the histogram to 13 make an assessment. This is obviously somewhat perhaps 14 incomplete to move the upper bound, the 95 percent upper bound, 15 to a higher value in order to deal with the fact that there was 16 more uncertainty in the system than I had expected. 17 Q. And when you're talking about more uncertainty in the 18 system that you expected, you were talking specifically about 19 that 13- or 14-year period, correct? 20 A. That's correct. 21 Q. You said that you got Dr. Angel to provide you with backup 22 documentation for all the information he gave to NORC. Did you 23 have Ms. Herman likewise provide the supporting information for 24 all the data and the reported value she gave to you? 25 A. Generally, no. Jacqueline M. Sullivan, RPR Official Court Reporter 1037 1 Q. Why the difference, sir? 2 A. We had been looking at the '72 forward data for a long 3 time. It appears in one of the earlier reports. We thought we 4 knew what we were looking at, and we've been following the 5 changes in the micro data that Ms. Herman and her team has been 6 introducing over the years in the ways that she has described 7 here, so we thought we knew who we were getting. 8 Q. In light of the uncertainty in the system I think as you've 9 referred to it, that you heard regarding the CD&L period, you 10 think that the upper bounds of the competencies at this level 11 that should be used is 97.5 percent? 12 A. That is correct. 13 Q. Why wouldn't it be 99 percent, sir? 14 A. It could be 99 percent. I indicated in a conversation that 15 you heard, bringing the judge into this, that is a decision that 16 the decision-maker makes as to what that upper bound should be. 17 Q. Okay. To help in that decision, can you explain you would 18 say 97-and-a-half percent rather than 99 percent? 19 A. Well, I had looked at the data, and the amount of 20 uncertainty added to it did not seem to be very large. I have 21 not had a chance to rerun everything, so I'm basing this on just 22 simply my judgment here, and 97 percent of them had seemed to be 23 a good compromise between '95 and '99. 24 Q. Do you recall in your direct testimony saying words to the 25 effect that uncertainty has to be scored in the plaintiffs' Jacqueline M. Sullivan, RPR Official Court Reporter 1038 1 favor? 2 A. I do remember saying that, yes. 3 Q. If I ask you to apply that same principal, would that point 4 toward using a 99 percent confidence level rather than a 5 97-and-a-half percent? 6 A. I've already told you that I moved from 95 to 97-and-a-half 7 because I felt that it was appropriate in the context of what I 8 said about scoring things. I don't think it goes to 99, but 9 it's not my decision. It's the judge's decision. 10 Q. Now, let me ask this: If in fact some of the information 11 that has been used in your system reported value as deemed to be 12 so uncertain that it should not be used as a reported value, 13 what effect, if any, does that have on the upper bounds of the 14 confidence factors? 15 A. That is a question I can't answer without doing it both 16 ways, and if I do it both ways I would expect -- and this is I 17 think where you're going with this -- I would expect that 18 treating it as missing incompletely would increase the 19 uncertainty level to treating it as partially reported, which is 20 the way we're doing it with most of the data here that is 21 reported. 22 Q. And that would cause the upper bound to be a higher total 23 amount, correct? 24 A. That's correct. And we had various looks at what if we 25 were to impute all the data in the data process going from where Jacqueline M. Sullivan, RPR Official Court Reporter 1039 1 we are now into the future. There's a way to do that. We 2 haven't done that. One of the things that repeatedly you've 3 heard with other witnesses is that with me more could be done, 4 and we are happy with this analysis given the time that we were 5 allowed to look at it and do it, but with more time we would do 6 more. 7 Q. Certainly you're not happy with this analysis in light of 8 some of the things that you've seen today with understatement of 9 revenues in between the 1930s and '40s and the 21 million 10 average leading up to 1968, are you? 11 A. I am not happy with the fact that we didn't incorporate 12 some of what you've told me today into this. 13 Q. Okay. Let's pull up Plaintiffs' Exhibit 137. 14 A. Other should I say we'd have to do. 15 Q. Put it at the top so we can see the title. 16 Do you see that this is a page that appeared in 17 Statistical Methods and Medical Research in 1999 by professor of 18 statistics at Penn State by the name of Joseph Schaeffer, do you 19 see that? 20 A. Yes. 21 MR. WARSHAWSKY: Do you have a copy of it? 22 THE COURT: You're looking at it on the screen. 23 MR. WARSHAWSKY: Normally they've been providing 24 copies. 25 MR. DORRIS: I don't, but why don't you take mine? Jacqueline M. Sullivan, RPR Official Court Reporter 1040 1 MR. WARSHAWSKY: I don't want to take your marked-up 2 copy. 3 MR. DORRIS: It will be helpful to ask you to follow 4 along. 5 MR. WARSHAWSKY: Maybe. 6 MR. DORRIS: I know what I want to ask about this one. 7 BY MR. DORRIS: 8 Q. Professor Schaefer is someone that you know and works in 9 and is knowledgeable in the field of imputation, correct? 10 A. Yes. I know him quite well. I'm mentioned in his book. 11 I'm mentioned in a lot of books. 12 Q. I made it through the first two pages of this article 13 before I couldn't follow it anymore. He's written here you say 14 a primer for people on multiple imputation. Do you see that? 15 A. His model is what we're using for MPI analysis. 16 Q. I want to concentrate on this paragraph that runs over to 17 the next page. 18 If you will blow this up. 19 And he refers to multiple imputations, MI. Do you see 20 that? 21 A. MI is not the only principal method for handling missing 22 values, nor is it necessarily the best for any given problem. 23 Q. That's entirely consistent with what you've testified to? 24 A. Yes, I agree with that. The word "principal" means that 25 you know statistically the principal. Jacqueline M. Sullivan, RPR Official Court Reporter 1041 1 Q. If you want to read the in between be my guest, but I want 2 you to go to the highlighted part that begins down at the bottom 3 of the page. It says, Given sufficient time and resources one 4 could perhaps derive a better statistical procedure than MI for 5 any particular problem. In real life applications, however, 6 we're missing data are a nuisance rather than a major focus of 7 scientific inquiry. A readily available approximate solution 8 with good properties can be preferable to one that is more 9 efficient but problem-specific and complicated to implement. Do 10 you see that? 11 A. Yes, I do. 12 Q. Is multiple imputation, should it be used to impute values 13 for the missing data where they are a nuisance other than a 14 major focus of the inquiry? 15 A. Can I answer that question by interjecting something first 16 before I answer it? 17 Q. Well, if you can answer it yes or no and then explain your 18 answer would be the preferable approach. 19 A. That's fine. I don't think that using multiple imputation 20 only when you're going to get rid of nuisance problems as needed 21 is appropriate. You can use it for other methods too, but there 22 are other specialized methods that could be used. I think it 23 could in the mathematical sense dominate. They could be better 24 if you had enough time and money. 25 Q. So you would not agree to the extent that Professor Jacqueline M. Sullivan, RPR Official Court Reporter 1042 1 Schaeffer is implying that it should be used where missing data 2 is a nuisance rather than the major focus of scientific inquiry, 3 you wouldn't agree with that? 4 A. I couldn't agree with it, no. 5 Q. Now, let me ask you to look at your article. Do you have 6 that? It was passed out today. It's Defendants' Exhibit 507. 7 I suspect you have a copy with you, Dr. Scheuren. 8 A. Am I allowed to elaborate on what you just said? 9 Q. You can answer any questions that the government asks you. 10 A. Go ahead then. 11 Q. Looking at your article -- I'm going to pause just for a 12 moment till it's brought -- you need to switch over, I think. 13 Sorry to do that to you. 14 I'll tell you what I am working with now. 15 Go to the third page of this exhibit, please, and go 16 under 4.2, the second paragraph, and blow that up for me. 17 The part I'd like you to look at and I'd ask to 18 highlight is a sentence that begins about halfway down there and 19 says "I still remain skeptical of models, however, and worry 20 about people complaints believing them too much." 21 Can you highlight that statement for me, please, sir? 22 Exactly. 23 Do you see that? 24 A. I do. 25 Q. What were you saying about that? Jacqueline M. Sullivan, RPR Official Court Reporter 1043 1 A. Let me give you some background, if I might. In the world 2 of sampling that I'm in, one of the greatest statisticians, who 3 recently died, W. Edwards Demming, did not believe in models. 4 He thought that we should base our inferences on what are called 5 frequentious-based as the same from Bayesian methods. And 6 frequentious-based methods are what we usually do when we're in 7 the world I live in, in the sampling world, because of Demming, 8 and we, because of Demming, did not adequately pay attention to 9 the fact that we really needed to use models for a whole lot of 10 settings that we weren't using them for, and where it's 11 particular when there are missing and misreported observations. 12 Even Demming would have agreed that we need missing data models 13 for missingness, but to the extent to which you have only one 14 model is a problem, because you really need to look at multiple 15 models because you need to look at the fact that the models 16 themselves have concerns with them, and we've been doing that 17 here, and because of the way this is being done, we're 18 presenting a model without all the models. 19 Q. And you really have looked at one model; is that correct? 20 A. No. You've actually called my attention to something we 21 gave you earlier, which is we've looked at at least two models, 22 and you asked me about the difference between those two models. 23 We spent quite a bit of time on that. There were other models 24 we looked at too. 25 Q. Two models constructed or run in the same analytical Jacqueline M. Sullivan, RPR Official Court Reporter 1044 1 framework, correct? 2 A. The word "model" and the word "analytical framework" phrase 3 are not equivalent. 4 Q. Let me ask you to step back, and there's a paragraph on the 5 other column just above where you've highlight there that begins 6 the non-ignorable case. Do you see that sentence that begins, 7 The non-ignorable case NMAR which adds variance with intractable 8 bias is a generally have-to-live-with form of missingness? 9 A. That's correct. 10 Q. Can you state that in more layman's language? 11 A. There are three types of missingness that are described in 12 here by the people at Don Rubin in 1976. One kind is just have 13 data that is not there. There's nothing particularly bad about 14 it except that your sample is too small and you don't have as 15 much data as you thought. There's nothing you need to do about 16 it except not over-react to it. The second phase which is 17 missing at random is a case where if you know what's missing and 18 how to fix it then you can adjust the data so that it gives you 19 answers that are not biased, but if you don't have enough 20 information from the data set to deal with certain things, then 21 you cannot adjust for the missing -- the non-ignorable portion. 22 Because the non-ignorable portion is usually something that you 23 don't know, not in the data set, somewhere else, and maybe not 24 on anyone's radar screen, and that is affecting the results. 25 Sometimes you can't even tell that it's there, and in this sort Jacqueline M. Sullivan, RPR Official Court Reporter 1045 1 of philosophical discussion, which is what this is, and 2 recollection, you need to be cognizant of that. 3 Q. And the missing data that we're confronted with in this 4 matter is which of the three types? 5 A. It's all three. 6 Q. Okay. 7 A. I've been talking about the all three part for quite a 8 while. 9 Q. Let's look at the next page. There is one last quote I 10 want to ask you about in this article. Under 6.1, if you would 11 pull that up and highlight that in the first sentence, it's kind 12 of a long sentence, you see there you wrote, Typically in my 13 practice subject matter experts believe in their imputation 14 models more than is warranted, possibly in part to the chronic 15 problem we have had in many surveys of underestimating 16 imputation variance and expense taken in time and money in 17 adjusting for missingness. Do you see that? 18 A. Yes, I do. I think it applies in this case. 19 Q. How does it apply in this case? 20 A. I think Professor Cornell believes that about his model. 21 Q. Okay. 22 A. It doesn't have any measure of uncertainty. 23 Q. So what I hear you saying, because you recognize that there 24 is tremendous uncertainty here -- well, you may not like that 25 word, tremendous. Would you agree there is tremendous Jacqueline M. Sullivan, RPR Official Court Reporter 1046 1 uncertainty in the data here? 2 A. A little more quantative and less qualitative, but yes, 3 I'll let you get away with that. You're doing a good job, by 4 the way. Thank you. 5 MR. DORRIS: There's one thing we can agree on, your 6 Honor. Civility here. The questioner and the witness can 7 stipulate to one thing. I'm sorry for the interruption there. 8 BY MR. DORRIS: 9 Q. Okay. Dr. Scheuren, you used a phrase in your direct 10 examination that I didn't understand. You said that the 11 statistician need to let the data speak to them. 12 A. That's correct. 13 Q. What do you mean? 14 A. Well, it means a simple as -- you used this illustration 15 earlier as doing chatter blots with the data as relating the 16 data you have with other things you think are true, okay, or 17 with experts who may have created the data or who have knowledge 18 of what it means and all of that you factor into your 19 understanding of what the data is and allowing it to speak. 20 Q. For example, of what we went through before lunch when I 21 showed you the collection and disbursement data from 1909 to 22 1911 and we looked at what the rate was there and then looked at 23 how the rates before and after were different than that, do you 24 remember us going through that? 25 A. Oh, I certainly do. It was very well done. Jacqueline M. Sullivan, RPR Official Court Reporter 1047 1 Q. And that would be something that a statistician would let 2 that kind of data speak to him, correct? 3 A. Correct. If my objective had been to focus on approving 4 the point estimates, then I would have done a lot more work with 5 that. 6 Q. Okay. Now, let's go to Defense Exhibit 460. It will help 7 make the question clearer. Okay. This is kind of a -- by the 8 way, did you put this Power Point slide together? 9 A. I wrote these words. I wrote these words, yeah. I may 10 have given it to them as a Power Point. I can't remember. 11 Q. You say, After examining the existing data and then using 12 multiple imputation to generate estimates of missing data and to 13 assess missing data uncertainties. Do you see the second bullet 14 there that I was just reading? 15 A. Yes, yes. 16 Q. You testified during your direct examination about 17 something and I didn't follow it, but you did say you did 18 describe the process as how you go about beginning to impute the 19 data in that process and you said something to the extent that 20 you would essentially make a first guess at that and then move 21 on. Can you explain to me what you mean by making a guess at 22 those values? 23 A. Well, you have some kind of set of prior beliefs which 24 you've incorporated and you have this notion we're going to use 25 a multi vari normal distribution in background and then we're Jacqueline M. Sullivan, RPR Official Court Reporter 1048 1 going to do our best to make an estimate from the available data 2 of what the mean and the variance of this data set are. A 3 vector mean and a coprehence (ph) matrix, and we do that, and 4 then we use the results, okay, the distribution we've created, 5 to draw an observation from random numbers, using random numbers 6 from that distribution. 7 Q. Okay. So that there is work done based on certain beliefs 8 that you have that then lead to the selection of the random data 9 to put in for the imputed data, correct? 10 A. I have prior beliefs and I have the data itself, the 11 reported data, and I use those to produce an initial starting 12 point for the multiple imputation, and then I repeat that 13 process over and over again and going back and forth until the 14 process converges, and then I stop and then I do another one and 15 so forth. 16 Q. Is there any way, looking at the information that you 17 provided to us electronically, for us to see these prior beliefs 18 that you have that lead to this first step or this really second 19 step that you're talking about here? 20 A. I think it's embedded in the software that we gave you 21 access to. Seeing the way you've conceptualized it I think we 22 probably would have to do a little more to make it visible for 23 you. 24 Q. And it's not visible? 25 A. It's in the software. It's in the proc MI, the multiple Jacqueline M. Sullivan, RPR Official Court Reporter 1049 1 imputation procedure from SAS has this information in it. We're 2 calling commands from that proc, and I've already mentioned to 3 you that related to what Joe Schaeffer has done. 4 Q. Well, let me ask it to you this way: With what we've been 5 provided so far it's a little bit like you've told us 6 information that you've gone and put into a black box and now we 7 look at what the results are, correct? 8 A. Well, how about opaque? Can you say they're opaque? 9 Q. I'll settle for opaque. Okay. 10 A. That's fair, that's why I have to look at the data, and 11 look at the software in detail. 12 Q. What do you mean by "opaque"? 13 A. We didn't walk you through the process from front to back, 14 you know, connecting the words that you heard me say with the 15 software that you got Friday, and we perhaps should do that. 16 Q. And you've not brought anything into court today that would 17 permit you to do that for us, have you? 18 A. No. As a result of your question about Professor Cornell's 19 data we spent the lunch hour providing that data to you. 20 Q. Well, I've had some interesting lunches during this trial 21 too, so I apologize for taking up your lunch hour, but I do 22 understand from government counsel that that data about his 23 analysis of Dr. Cornell's information can be provided to us, and 24 what I'm going to ask your Honor is that I go as far as I can 25 go, cover everything except that, any questions that may come Jacqueline M. Sullivan, RPR Official Court Reporter 1050 1 from that, that once that's provided we could then ask any 2 follow-up questions, if any, that we have of Dr. Scheuren on 3 that. 4 THE COURT: We'll talk more about that. Go as far as 5 you can and then we'll talk about it. 6 MR. DORRIS: Okay. 7 BY MR. DORRIS: 8 Q. Now, let me ask you this question about the relationship 9 between revenues and disbursements. You explained to me I think 10 before lunch that you would look at that and there would be some 11 type of coefficient or colinear relationship between those two, 12 correct? 13 A. We would look for that, and in fact we found it, yes. 14 Q. And let me ask it to you this way: The revenues are to a 15 large extent driven by market forces, correct? 16 A. Can we use "collection" instead of "revenue"? Whatever you 17 want to do. 18 Q. I'll be glad to use collection. 19 A. Okay. 20 Q. What do you mean by "collection"? 21 A. Well, because there's a lot of things in the IIM system 22 that aren't really revenues, okay? For example, bid deposits, 23 okay, it's not revenue. It's just a deposit. 24 Q. So where you've put "collections" on your chart you've just 25 tried to have those include things that are not the total Jacqueline M. Sullivan, RPR Official Court Reporter 1051 1 revenue but only collections; is that right? 2 A. Collection is the total -- is the thing we're looking at 3 here. What's coming into the system is what we're calling a 4 collection. 5 Q. For example, you gave bid deposits. Are those in your 6 collection number or not? 7 A. They're in the collection number. 8 Q. I'll use collections with you. Now, collections are driven 9 to some extent by market forces, correct? 10 A. Of course. 11 Q. The price of all -- the price of a quart of timber, grazing 12 rights and the like? 13 A. All of that's true. 14 Q. Disbursements are not driven as much by market forces, are 15 they? 16 A. There are different periods in this time period. You know, 17 once -- you know, at some point until you remember what year it 18 is, if you have more than $5.00 in oil receipts you got a $5.00- 19 check and disbursements of, you know, disbursements of other 20 kinds of revenue as well, so once you receive a certain 21 threshold you got the money. In the old times that wasn't 22 always done. 23 Q. Well, let me ask you if this. 24 A. So they're linked, by the way, in modern times because of 25 this rule. Jacqueline M. Sullivan, RPR Official Court Reporter 1052 1 Q. When you say "modern times," what time period begins 2 "modern times" for that answer? 3 A. I know the answer to that but I can't give it to you right 4 now. Sorry. I just don't know. It's been a long time since 5 this change has occurred. 6 Q. Okay. But before that period of time they are not linked 7 as a result of that policy; is that correct? 8 A. That's correct. 9 Q. And so that while your data established a colinear 10 relationship between collections and disbursements, actually 11 market forces affect those two numbers in very different 12 fashions, don't they? 13 A. You just ignored what I said, didn't you? 14 Q. I may not have understood what you told me. 15 A. Part of this time, part of this historical period, that is 16 correct, but although you don't establish that the effect of 17 disbursements, you just said something about revenue. Revenue 18 and disbursements are linked in modern times. In earlier 19 periods they were not necessarily linked. But if you look at 20 the data during the Great Depression in our data set you will 21 see that disbursements exceed collections year after year, and 22 that's perfectly understandable. The money was coming out of 23 the trust in order to feed people. 24 Q. And that's that same period where we've seen even what Dr. 25 Angel and Ms. Herman note, that the revenues, their word was Jacqueline M. Sullivan, RPR Official Court Reporter 1053 1 "revenues," were understated during that same period, correct? 2 A. I'm not sure when the beginning. I remember the 1949 end 3 point, but I don't remember what the early point is. 4 Q. How can there be a colinear relationship between balance 5 information and collections and disbursements when even 6 according to you the balances do not foot with the collection 7 and disbursement amounts? 8 A. Well, there are two things going on in which you've said. 9 First of all, a correlation can exist even though there is not a 10 direct functional relationship between something because there's 11 uncertainty. The extent of the correlation is a measure of 12 that, so of course there can be a correlation between these 13 values, even if they don't foot. 14 Q. Now, there are times in your numbers, for example, where 15 there are known information. 16 Let's pull up Defendants' Exhibit 462 to 19, the 17 second page, and blow up the first four lines or so. 18 Let me just ask you this: You see we're going along 19 and there is a missing value for the balance in 1929. Do you 20 see that? 21 A. Yes, I do. 22 Q. And so you're working with reported values for collections 23 and disbursements in the years before and after where that 24 missing data is. Do you see that? 25 A. Um-hmm. I've analyzed this data. Jacqueline M. Sullivan, RPR Official Court Reporter 1054 1 Q. And you see that it says the balance as of the end of the 2 year, fiscal year 1928, is $74.7 million, correct? 3 A. I do. 4 Q. And then the next year there is about $9.5 million more 5 collected than disbursed, correct? 6 A. Correct. 7 Q. And then your imputation model has put in a balance that, 8 if I did the math right, would be at about $85 million, it puts 9 in a value of $33.3 million there, correct? 10 A. Yes, I know that. I've analyzed this data, as I already 11 told you. I noticed this when I saw it. 12 Q. Okay. So even when the model then comes and imputes 13 balances, it imputes balances that don't foot, correct? 14 A. That's correct. 15 Q. Is that because of the tremendous uncertainty in the data? 16 A. We looked at this data and we actually we had this 17 procedure of looking for outlies, and one of the things that we 18 did with the data that you have here is that we looked at the 19 outlies here too, and this is the worst. There are other out- 20 lies but they're mostly small. This is a spectacularly bad 21 outlier. 22 Q. The one that I just give you? 23 A. That's correct. I'm not surprised given who you are that 24 you would have noticed it, and we noticed it too and we went 25 back and looked at that. Our focus here is not on the balance. Jacqueline M. Sullivan, RPR Official Court Reporter 1055 1 Our focus is here on the collections and disbursements, okay, 2 and so we didn't like it but we left it. 3 Q. You didn't like it but you didn't do anything about it, 4 correct? 5 A. Correct. 6 MR. DORRIS: Your Honor, I don't have any further 7 questions subject to the one note I mentioned about the analysis 8 that the government has agreed to provide -- or the data they've 9 agreed to provide to us. 10 THE COURT: Let's talk for a minute about that. Mr. 11 Warshawsky, do you have something to say about it? 12 MR. WARSHAWSKY: Yes, your Honor. 13 THE WITNESS: Can I say thank you? 14 MR. DORRIS: Sure. 15 THE COURT: And you may also step down. Thank you, 16 sir. 17 MR. WARSHAWSKY: Your Honor, we have what the 18 plaintiffs have asked for, and it's provided to them. It 19 consists of two files. One is actually the SAS application I 20 believe that they already had. The other is an Excel file. 21 This was an attempt to reconstruct -- 22 THE COURT: Dr. Scheuren, actually, would you come 23 back up? I have a couple of questions I'd like to ask you. 24 THE WITNESS: Oh, certainly sir. 25 THE COURT: Go ahead. Jacqueline M. Sullivan, RPR Official Court Reporter 1056 1 MR. WARSHAWSKY: It's an attempt to reconstruct Dr. 2 Cornell's model. We've never been provided any of the data or 3 information regarding Dr. Cornell's model. We first learned of 4 it, it was actually Wednesday, last Wednesday when he testified. 5 We had the good fortune that I would say that his model 6 apparently didn't change since the March 2008 brief that 7 plaintiffs filed. He made no adjustments even though we had 8 provided information in our brief, but although Thursday we 9 were, told both sides had been told to provide this information, 10 we still haven't received anything about Dr. Cornell's model, so 11 that's what we've got, is an attempt to reconstruction Dr. 12 Cornell's efforts and NORC's conclusions and now plaintiffs have 13 it. I'm certainly prepared to do the redirect on everything but 14 this and we can have Dr. Scheuren come back tomorrow if that's 15 the Court's wishes. 16 THE COURT: You're turning over the SAS application 17 and the spreadsheet? 18 MR. WARSHAWSKY: It's an Excel document, and Dr. 19 Scheuren could probably tell you about it, but basically my 20 understanding is it's the inputs that go into the application. 21 THE COURT: I thought I heard Mr. Dorris asking more 22 than that. I think he wants to look inside the opaque box. 23 MR. WARSHAWSKY: Well, your Honor, I'm glad you asked 24 about the opaque box. The opaque box is a reference to the SAS 25 application. I advised Mr. Smith before we had any discussion Jacqueline M. Sullivan, RPR Official Court Reporter 1057 1 with the Court, I should say before we had the change, I advised 2 him Thursday that the application was a multiple imputation 3 application, and in fact when we delivered it on Friday I told 4 him that we had somebody from NORC waiting by if they needed any 5 assistance walking through it. That was in the afternoon. I 6 heard nothing as of 9:45 that evening. Nothing. Saturday I 7 spoke with Mr. Smith. He called simply to inquire about Dr. 8 Angel's information, so they've never asked us for the 9 assistance that we offered, and it's frankly a little odd at 10 this point to hear plaintiffs complaining that they haven't been 11 walked through the SAS application because the offer was there 12 as early as Friday afternoon. 13 THE COURT: All right. Do you want to do some 14 redirect now? 15 MR. WARSHAWSKY: I can do the redirect, your Honor, on 16 everything, obviously reserving -- 17 THE COURT: Actually I want to ask two or three 18 questions and then you can do your redirect, so I released you 19 prematurely, Doctor. 20 I need to understand better than I do what this ten 21 thousand iterations business is all about. I don't quite 22 understand what you mean by that. Do you mean ten thousand 23 pieces of data, do you mean you run the model ten thousand 24 times, and if you run it ten thousand times how does the 999 25 differ from the thousand and so forth? Jacqueline M. Sullivan, RPR Official Court Reporter 1058 1 THE WITNESS: We have a knowledge that we have from 2 the data to begin with and we use that to start the modeling 3 process. We bring it in. We then use random numbers to 4 actually do the simulations so each simulation is different 5 because of the randomness that we're adding, but they're 6 constrained by what we know about the model, so after we do the 7 first iteration we construct the posterior from the data. And 8 we compare the results, the parameters that we're interested in, 9 which was usually the means and variance, covariances in this 10 world, we compare them to what or what our starting values were, 11 and if they're not close, and close is really quite close, if 12 they're not close we average the first iteration with our 13 estimate, our starting value, and then we do it again and we 14 keep doing that until the numbers converge, so we get estimates 15 of the means and totals, means variances that are the same from 16 one iteration to the next, and this goes on a long time. Of 17 course it's microseconds because it's in the computer, and that 18 is the first iteration, that is the first imputation that we do, 19 and then we start begin, okay, a new starting point and do the 20 process over and over again until we've done ten thousand of 21 those. That's how we did it. 22 THE COURT: If I were to sit down at a keyboard with 23 that software would it show me what you mean? 24 THE WITNESS: Yes, yes. You have to have SAS on your 25 computer and things like that, but yes. Jacqueline M. Sullivan, RPR Official Court Reporter 1059 1 THE COURT: The other thing I want to talk about is 2 the use of the Osage average share as one of the five variables 3 because you've got nothing about a year and an Osage variable. 4 Now, is that variable useful because of the quantity of it, or 5 is it useful because of the comparative size to this year's to 6 next year's to the year after that, or does it have anything to 7 do with oil price versus timer price? 8 THE WITNESS: I can't speak to oil price versus timber 9 price. It does give us some information about what's going on 10 with the economy generally, and of course it says specifically 11 regarding oil, and it goes all the way back into this period, 12 and it is directly relevant to the Indian trust system. We 13 could have added other variables. 14 THE COURT: Like what? 15 THE WITNESS: Price indices, GDP, you know, general 16 economic variables, but again we're talking about very rural and 17 regional areas where these general kinds of variables might not 18 get us down from enough or have any more predictive value than 19 the Osage did. 20 THE COURT: You said at one point, I think, you could 21 have run this without the Osage numbers. 22 THE WITNESS: We have done that, yes. 23 THE COURT: What's the result? How sensitive is this 24 model to using the Osage? 25 THE WITNESS: I can't give you the answer to that Jacqueline M. Sullivan, RPR Official Court Reporter 1060 1 right now, but it is sensitive in the early period, before 1909. 2 It is sensitive. It's not going to make much difference. 3 THE COURT: If you ran it with the Osage numbers you'd 4 get what, higher balances or lower calculated balances? 5 THE WITNESS: I'm not sure what I would do if I 6 were -- I don't know the answer to that question, your Honor. 7 If I were really pressed and said you can't use Osage because 8 it's this, that or the other thing, then I would probably go 9 look for some other variables which I didn't think was 10 particular relevant, but I would check and maybe find them to be 11 relevant. 12 One thing that's been said a lot is we're operating 13 quickly here. If we had more time, you heard me make a comment 14 just a little while ago about an outlier that was identified by 15 plaintiff, the Plaintiffs' lawyers, which we identified 16 ourselves but after we delivered this. And of course I would 17 have fixed that, okay, but I don't think it would make much 18 difference so I didn't bring it up, but we've looked at all the 19 outliers. If we had more time we'd do better, but I think the 20 basic results that we have -- 21 THE COURT: You're comfortable with them? 22 THE WITNESS: With the terms of the overall 23 uncertainty. I mean, we're looking at something -- there's a 24 lot of uncertainty here and the uncertainty cuts against the 25 government, but the kind of uncertainty that is on the table Jacqueline M. Sullivan, RPR Official Court Reporter 1061 1 here is rather different in size than the numbers that you've 2 seen from the plaintiffs by a factor of ten. 3 THE COURT: Yes. Okay. Thank you. 4 Do you want to do some redirect, Mr. Warshawsky? 5 THE WITNESS: Thank you, your Honor. 6 MR. WARSHAWSKY: Thank you, your Honor. I just have a 7 couple of redirect. 8 REDIRECT EXAMINATION 9 BY MR. WARSHAWSKY: 10 Q. Dr. Scheuren, I'd like to ask you about a couple of things 11 that Mr. Dorris inquired about with respect to information you 12 received from Morgan Angel. 13 A. Yes. 14 Q. Now, you may recall on cross you were being asked about 15 some of the collection figures from 1922 to 1949. 16 A. Yes. 17 Q. Do you remember having any conversations with anybody from 18 Morgan Angel about disbursements during those periods? 19 A. No. 20 Q. If the disbursements -- let me ask it this way. If the 21 disbursements were similarly understated, there is a discussion 22 about collections being understated, how would that impact your 23 calculated balance estimates? 24 A. It wouldn't have changed the balances potentially at all. 25 The things we saw going forward, especially in the period Jacqueline M. Sullivan, RPR Official Court Reporter 1062 1 before, the early period, is that even though it doesn't have to 2 move together, as I've stipulated, it tended to move together 3 up or down, except during the great economic stress during the 4 Depression. 5 Q. And just to be clear, when you're talking about things 6 moving together, you're talking about the collections and 7 disbursement? 8 A. Yes. If collection was up disbursement would also go up 9 and vice versa. 10 Q. And your focus is the calculated balances? 11 A. That's what we're after, yes. We're not trying to make 12 best estimates for each of these cells. 13 Q. Now, if you could pull up Defendants' Exhibit 72, please, 14 page four. Okay. And if you could highlight the section that 15 Mr. Dorris asked about here. 16 Do you remember the discussion about cash receipts are 17 running at a rate of $121 million per year? 18 A. Yes, I do. 19 Q. Do you know if Dr. Angel actually used that figure in his 20 analysis? 21 A. I don't know. I don't think he did, but I don't know. 22 Q. You weren't present this morning when Dr. Angel explained 23 on his redirect why he didn't use that, were you? 24 A. No, I was not. 25 Q. So you don't know the reasons that he rejected that figure? Jacqueline M. Sullivan, RPR Official Court Reporter 1063 1 A. No, I don't. 2 I want to make this point. I asked for data from Dr. 3 Angel that was connected directly to a historical, you know, 4 historical document, okay. And that's what we got and his 5 judgment is what I relied on. 6 Q. Let's pull up Plaintiffs' Exhibit 137. I'm not sure, do we 7 need to get that from you? Yes. 8 Dr. Scheuren, Mr. Dorris asked you about a paragraph 9 that continued from page one to page two. If it assists you I 10 can provide you with Mr. Dorris' copy. 11 A. Let me have the copy. Just looking through the lines it's 12 a little strange. It's a missing data problem. 13 Q. Okay. All right. And on the screen we do have those first 14 two pages up. Now, do you remember your testimony about that 15 paragraph? 16 A. Yes, I do. I wanted to elaborate and I didn't get to do 17 that, but maybe I can do it now. Maybe you'll ask me a question 18 and I can do it. 19 Q. Here's your chance. Why don't you agree with that 20 statement, Dr. Scheuren? 21 A. Well, I think we're having a situation where there is a lot 22 of different approaches to missing data. And Joe is trying to 23 convince people that at least in some cases multiple imputation 24 is the method of choice but he doesn't want to be absolute about 25 it and say it's always the method of choice. That doesn't make Jacqueline M. Sullivan, RPR Official Court Reporter 1064 1 friends and influence people so he's careful and he's modest in 2 his statement. I actually believe that is much more of an 3 advocate for multiple imputation than that, but he's trying to 4 deal with people who -- and there's some controversy among 5 multiple imputation. If you looked a little bit at the material 6 that I had in the paper that we looked at this morning, all you 7 Google when you Google -- I'm very sorry, Jackie -- you Google 8 this issue, the first thing that comes up is a paper by a good 9 friend of mine, Bob Fay, that has some problems with multiple 10 imputation, so there's some underbrush here. However, that kind 11 of underbrush actually leads to a particular kind of problem 12 which is the opposite of the problem here. If there is a 13 problem with the multiple imputation in this particular setting, 14 it doesn't affect our result except that we may have slightly 15 overstated the uncertainty as a result of it. Slightly 16 overstated. 17 Q. And what is the significance if you overstate the 18 uncertainty in this case? 19 A. That the amount of money that is at issue here is larger, 20 but we've already done that and I think I'm weighing all these 21 things, and what I heard the other day from Michelle Herman and 22 her response from the Plaintiffs' lawyers, I weigh that in with 23 the fact that we may slightly have overstated, but I don't think 24 that's very important in this case and I think that a modest 25 adjustment to a 97.5 upward bound is the right answer here in Jacqueline M. Sullivan, RPR Official Court Reporter 1065 1 rough justice. 2 Q. And to the extent you overstated, does that inure to the 3 benefit of the plaintiff or the defendant? 4 A. It benefits the plaintiffs. It benefits the plaintiffs. 5 MR. WARSHAWSKY: Your Honor, I have no more redirect 6 subject to whatever is covered. 7 THE COURT: Doctor, I think we're going to release you 8 today. I think we're going to have to say that you're subject 9 to recall, but when and under what circumstances remain to be 10 seen. 11 And we're going to take a ten-minute break. 12 MR. WARSHAWSKY: One request, your Honor. I know Mr. 13 Scheuren is going out the country I think next week. 14 THE WITNESS: Yes. 15 MR. WARSHAWSKY: So we request that he be called this 16 week. 17 THE WITNESS: Thank you, Judge. 18 COURTROOM DEPUTY: Court stands in recess. 19 (Recess taken at about 3:17 p.m.) 20 COURTROOM DEPUTY: Please come to order and be seated. 21 THE COURT: Okay. What's next? 22 MR. WARSHAWSKY: Your Honor, just to clarify, as I 23 indicated, we would like to get Dr. Cornell's backup data as 24 well and his analysis, and hopefully that's coming soon. I 25 haven't heard plaintiffs clarify that one. Jacqueline M. Sullivan, RPR Official Court Reporter 1066 1 Also, if we could get an indication as to when Dr. 2 Scheuren is to be recalled so we can make sure he's here. 3 THE COURT: Does anybody want to respond to any of 4 that? Mr. Dorris? 5 MR. DORRIS: If Mr. Warshawsky had asked me about it 6 at the break I would have been glad to address that with him. I 7 don't know what backup from Dr. Cornell he wants. We gave him 8 all of the information with the documents supporting it on March 9 19 and there hasn't been further analysis. Will there be? 10 We've indicated to Mr. Warshawsky we think there will be and 11 that we'll provide that to him as soon as it's done and that's 12 what we're committed to doing, but that has not -- that process 13 is not done and we don't have anything further to provide to him 14 at this point. 15 THE COURT: Okay. What about Scheuren, when do you 16 want to bring Scheuren back? 17 MR. DORRIS: Your Honor, they provided information, 18 and I would think given the witnesses I'm responsible for 19 tomorrow that Thursday would probably be better. Today is 20 Tuesday. Thursday would probably be best for Dr. Scheuren. 21 That gives me tomorrow to look at it. 22 THE COURT: Let's ask Dr. Scheuren to be available on 23 Thursday. 24 MR. WARSHAWSKY: Very good, your Honor. 25 THE COURT: Mr. Siemietkowski? Jacqueline M. Sullivan, RPR Official Court Reporter 1067 1 MR. SIEMIETKOWSKI: Your Honor, the defense calls 2 Terence Kehoe. 3 THE COURT: All right. 4 COURTROOM DEPUTY: Please raise your right hand. Do 5 you solemnly swear that the testimony you give to the Court in 6 the case now on trial will be the truth, the whole truth and 7 nothing but the truth so help you God? 8 DR. KEHOE: I do. 9 COURTROOM DEPUTY: Thank you. Please be seated. 10 MR. SIEMIETKOWSKI: Your Honor, we intend to have Dr. 11 Kehoe testify regarding two areas. Primarily Dr. Kehoe will 12 testify regarding IIM banking policy, specifically where he has 13 been throughout the history of the IIM system. Secondarily and 14 more briefly, we expect Dr. Kehoe to testify in terms of the 15 Plaintiffs' calculation of alleged benefit to the government. 16 Now, your Honor, Dr. Kehoe has never testified as an 17 expert witness before. In fact, he has never testified at all. 18 We would like to qualify him as an expert like Dr. Angel in the 19 history of federal Indian relations and in the federal Indian 20 records. To do that, however, rather than simply show the Court 21 a C.V., we'd like to do that, but also lay the foundation with 22 Dr. Kehoe. 23 THE COURT: Fine. Go ahead. 24 MR. SIEMIETKOWSKI: And I ask that DX 498 be put on 25 the screen, please. Jacqueline M. Sullivan, RPR Official Court Reporter 1068 1 TERENCE KEHOE, Ph.D, WITNESS FOR THE GOVERNMENT, SWORN 2 DIRECT EXAMINATION 3 BY MR. SIEMIETKOWSKI: 4 Q. Dr. Kehoe, would you please state your full name? 5 A. Terrence Patrick Kehoe. 6 Q. Where do you work? 7 A. I work at Morgan Angel & Associates in Washington, D.C. 8 Q. What's your profession? 9 A. I'm a historian. 10 Q. Dr. Kehoe, would you please explain your educational 11 background? 12 A. I have a bachelor of science degree in business 13 administration from Bowling Green State University and a 14 master's and a Ph.D in American history from Ohio State 15 University. 16 Q. In the course of your formal studies have you had a chance 17 to learn the conduct of historical records research? 18 A. Yes. In graduate school I had extensive experience 19 researching public policy and how it changed over time working 20 in various state and federal archives primarily in government 21 records of various kinds. 22 Q. What about your employment, Dr. Kehoe, have you ever taught 23 at all at the university level? 24 A. Yes. I was a visiting professor at Wake Forest University 25 for like two years, and among the courses that I taught there Jacqueline M. Sullivan, RPR Official Court Reporter 1069 1 was American environmental history, history of the American 2 West, the United States since 1945. 3 Q. Did any of those courses taught at Lake Forest University 4 touch about Indian matters? 5 A. Certainly the course on environmental history did, and most 6 extensively the course on the American West. That was a big 7 component in that course. 8 Q. Dr. Kehoe, how long have you worked at Morgan Angel? 9 A. I joined the firm in December of 1999. 10 Q. Who has been your supervisor there since December of 1999? 11 A. Dr. Edward Angel. 12 Q. Could you provide the Court an estimate, please, of what 13 percentage of your work at Morgan Angel has been focused on 14 Indian matters? 15 A. About 75 percent. 16 Q. And what about the other 25 percent? 17 A. Primarily devoted to research and writing as it relates to 18 environmental litigation. 19 Q. Within Indian matters, what types of issues have you 20 focused on? 21 A. Well, I've done some research and writing concerning 22 natural resources management on reservations. I would say that 23 the bulk of my work has concerned the management of both tribal 24 trust funds and individual Indian monies. 25 Q. While employed at Morgan Angel who have your clients been? Jacqueline M. Sullivan, RPR Official Court Reporter 1070 1 A. The work that I have done involving American Indian policy 2 has been for the Office of Historical Trust Accounting and the 3 Department of the Interior, the Department of Justice, and the 4 Department of the Treasury. 5 Q. Have you written my papers for these clients? 6 A. Yes, I have. I've written a number of major papers, 7 research papers, that have run between about 20 and 75 typed 8 pages. 9 Q. Are some of those papers reflected on the screen that's 10 shown now? 11 A. Yes, they are. And as you can see, I've written somewhat 12 about natural resource issues, but again, the primary focus has 13 been on the government's administration of Indian trust funds. 14 As you can see here, the Treasury as well in the administration 15 of travel trust funds, investment of travel trust funds. I've 16 written on IIM banking policy. And if you can scroll to the 17 next page on my resume. 18 MR. SIEMIETKOWSKI: Technical difficulties. They're 19 not getting in. Let's hold up a second, Dr. Kehoe, until we get 20 the screen working on both sides. 21 BY MR. SIEMIETKOWSKI: 22 Q. You wanted to refer to the papers that you've written? 23 A. As noted here, in addition to those research papers I've 24 also submitted 29 reservation histories to the Office of 25 Historical Trust Accounting. These are brief studies abou