From: David Rudiak <DRudiak@earthlink.net>
Date: Mon, 3 Feb 2003 20:44:54 -0800
Fwd Date: Tue, 04 Feb 2003 08:57:15 -0500
Subject: Re: Validating The Ramey Memo - Rudiak


 >From: Jim Houran <JHouran@siumed.edu>
 >To: ufoupdates@virtuallystrange.net
 >Date: Fri, 31 Jan 2003 11:51:40 -0600
 >Subject: Re: Validating The Ramey Memo

 >>From: Thomas Carey <TCarey1947@aol.com>
 >>To: ufoupdates@virtuallystrange.net
 >>Date: Fri, 31 Jan 2003 10:14:42 -0500
 >>Subject: Re: Validating The Ramey Memo

 >>>From: Jim Houran <JHouran@siumed.edu>
 >>>To: UFO UpDates - Toronto <ufoupdates@virtuallystrange.net>
 >>>Date: Tue, 28 Jan 2003 14:25:53 -0600
 >>>Subject: Re: Validating The Ramey Memo

 >>>I cannot speak for Kevin Randle, but I will leave this now non-
 >>>constructive debate on this final note. It seems that Rudiak
 >>>neither understands the rationale for my arguments on bias or
 >>>the influence of preconceived notions, nor the implications of
 >>>the various statistical findings in my JSE paper with Randle.
 >>>Suffice it to say, we must agree to disagree. I strongly suggest
 >>>to Rudiak, however, that he submit his study of the Ramey Memo
 >>>to a peer-reviewed, scientific journal, as I have done. I will
 >>>continue my work on the problem in the way I feel it should be
 >>>handled, and I will disseminate the results (be they pro or
 >>>con) in said forums.

 >>In case you missed it folks, what Jim Houran just executed
 >>is known in debating and debunking circles as the "discreet
 >>withdrawal". Congratulations, David.

 >Reality check, Tom. As I have said repeatedly, I urge anyone
 >that has issues with my study or proposals to submit a formal
 >Letter to the Editor of the JSE, as well as submit their own
 >work to a peer-reviewed, scientifc journal, and allow others to
 >independently attempt to replicate and validate that work. That
 >is what we academics call the scientific method.

Here's a real reality check. The "scientific method" demands
that conclusions follow logically from data. In the case of the
Houran/Randle paper, the scientific method has been turned on
its head. The primary conclusion ("very significant" context
"priming" effect) has no data to support it. When asked for the
data by one of the referees of JSE, it turns out there was no
data because the research assistant had tossed it in the trash.

At that point, the paper should have been automatically rejected
by the peer reviewers for lack of proper supporting data. But
it wasn't, another serious failing of the "scientific method"
and the process of peer review.

The "scientific method" also demands that data reduction be done
properly. It turns out that in at least one instance that I
just discovered, this wasn't done either, and should have been a
big red flag to at least one author of this paper or at least
one JSE peer reviewer. In the "Results" section, the average
words deciphered per person for each of the 3 testing conditions
was discussed: the Roswell context condition with a mean of 4.6
words, the atomic testing condition with 4.8 words, and the
context-free control condion with 1.6 words.

What struck me as odd on my first read-through was that the
standard deviations for all three groups were identical at 0.23
words. The odds of that happening by chance are pretty slim,
probably much less than 1%.  Furthermore, these SD's seemed
much too small. We're talking about 2/3's of the readers being
within about "quarter of a word" of the mean in number of words
read, and 95% being within about "half a word" of the mean.

When I finally looked at it more closely yesterday, it turned
out to be more than odd -- it was impossible, unless they were
scoring fractional words or syllables. E.g., in the Atomic
testing group, one could obtain a mean of 4.8 words by 47 of the
58 subjects reading 5 words and the remaining 11 reading 4
words. But the standard deviation is 0.40, not 0.23, even in
this best-case scenario. The minimum possible SD's are even
larger for the Roswell and control groups, where the means are
further removed from being whole numbers.

This isn't opinion on my part. This is just math, correct math,
another key component of the scientific method.

Thus on top of a conclusion with no supporting data, we now have
some rather obviously bogus standard deviations. Where did
these come from? Are these just innocent typos? Did somebody
not know how to calculate SD's? Or did somebody just make them
up after the paper was submitted? Without the raw data, it
would be impossible to calculate them if later requested by a
referee. So maybe somebody just pulled them of a hat but didn't
have the sense to make them mathematically possible much less
plausible.

All I know is that this is another instance of where JSE's peer
review process failed miserably. It doesn't take that much
smarts to realize that something was probably wrong with those
standard deviations. It certainly took me only a few minutes to
verify that they were impossible.

There are other oddities in the numbers, such as the Atomic
testing group reading so-called "exlusive" words with nearly
three times the frequency as the Roswell group despite spending
20% less time on average (200 words vs. 75). That makes no
sense, nor do other numbers in the test results, all of which
should be raising red flags that maybe the experiment wasn't
carried out properly. But there is no way to check any of this
because the raw data got tossed in the dumpster. Incredible!

I guess what present-day "academics" call the "scientific
method" has changed somewhat since I was in the academic
community.

 >Some of the comments I have read make me, an outsider to
 >ufology, think that selected ufologists and researchers have
 >never heard of it, or have no practical knowledge re: research
 >methods and study design. I don't hide from criticism or debate,
 >nor I do refrain from giving it where I see it should be
 >applied, as anyone who knows me and my work record can attest.
 >Indeed, there is nothing discreet about me.

James Houran's psychological experiment has completely bogus
standard deviations. There is no data to support the main
conclusion. Allegedly the research assistant threw it out. Or
maybe Houran's dog ate it. How can one reach a conclusion
without the proper data and write a paper about it? And why
would the peer reviewers OK such a paper for publication? The
whole point of peer review is to screen out such obviously
deficient research.

Under the circumstances, James Houran's statements about how he
understands the scientific experimental process while his
critics don't are supercilious and hypocritical, if not
laughable. Instead, pretending to be above it all is his rather
transparent way of avoiding embarrassing questions.

 >My impression is that my work on the Ramey problem scares people
 >who have clear vested interests in particular interpretations of
 >the Ramey Memo.

My impression is that James Houran is scared to answer questions
about his work on the "Ramey problem." Some of his reported data
is clearly invalid (the standard deviations), perhaps even
fabricated, and his primary conclusion has no supporting data.
He stated in his paper that their "expectations" were that they
would find a "very significant" "priming" or biasing of readings
based on context. Lo and behold, that's what they found, or so
they claim. When a referee asked to see the relevant data, they
claimed it had been thrown away by the research assistant.

Supposedly it's this unsupported expected conclusion from data
that does not exist that has those of us with "vested interests"
so "scared." Houran has the emotion wrong. It is not fear that
I feel. It is anger and outrage that this study with it's sham
results could possibly be considered "scientific" or that
anybody would take it seriously.

That a paper with so many serious and obvious deficiencies got
through the peer review process at JSE is also highly
disturbing. There is no way to describe the peer review on this
paper other than it being shoddy and inept. Maybe some "peer"
back-scratching was involved. JSE needs to take a very close at
its peer review process if it wants to develop credibility as a
journal of alternative science.

 >Tom, the person you should really congratulate is Kevin Randle,
 >who has had remarkable fortitude in resisting participation in
 >this thread. That is what we call in certain circles "class".

Kevin Randle has obviously put in an enormous amount of time
into the Roswell case. He has also always been very helpful to
me when I've asked him questions concerning Roswell. So I am
very sorry to see him associated with this train wreck of a
"scientific" study. Nonetheless, he is the coauthor of the paper
and shares responsibility, even if maybe he didn't handle the
psychological experiment portion with the imaginary standard
deviations and missing supporting data of the primary
conclusion.

 >Good luck to you, Tom, and all other researchers on your work on
 >the Ramey Memo. I look forward to seeing it published in
 >scientific journals.

 >Sincerely,

 >Jim Houran

I look forward to Jim Houran anwering questions for a change
instead of stalling and dodging. I asked him for the numbers
supporting his primary conclusion. No answer. Instead I get his
evasive response that if I have problems with the paper I should
write a letter to the JSE. Why, do they have the missing data?
Why won't the primary author answer a simple question?

To the list of quesions, I would now like to add where he got
those obviously bogus standard deviations for words deciphered?
However, I seriously doubt we will get any answers from this
not-so-sincere sincere academician. Instead he pretends to be
taking the high road and representing the scientific process,
which has actually been very badly corrupted in this study.


David Rudiak