• Welcome to BirdForum, the internet's largest birding community with thousands of members from all over the world. The forums are dedicated to wild birds, birding, binoculars and equipment and all that goes with it.

    Please register for an account to take part in the discussions in the forum, post your pictures in the gallery and more.
ZEISS DTI thermal imaging cameras. For more discoveries at night, and during the day.

Scientific papers, can they be trusted? (1 Viewer)

alcedo.atthis

Well-known member
A Note Regarding the Science & Publication of Research Studies

In order to critically address the published works of any writer, it is important for the reader to have an understanding of the purpose of scientific research publication, the process of scientific study review, and the role peer review plays in the publication of scientific research.

According to An Introduction to Critical Analysis of Publications in Experimental Biomedical Sciences (Rangachari, 1994-2001),“Scientists publish research reports for a variety of reasons. Ideally, a research report is a free communication by a scientist or a group of scientists informing their peers about a set of novel findings that either provide answers to puzzling problems or raise issues that are of academic or practical interest. At the opposite extreme lie reports that serve merely to add to the curriculum vitae of the investigators and have little or nothing important to say. Most often, the situation lies between these two extremes.” This author goes on to provide a summary of the scientific review process and a checklist for the critical evaluation of a scientific paper. The checklist provided by P.K. Rangachari is as follows:

CHECKLIST

as provided by P.K. Rangachari in “An Introduction to Critical Analysis of Publications in Experimental Biomedical Sciences.” http://www.science.mcmaster.ca/biopharm/critanal.htm10.31.03

Introduction

  1. Did the authors indicate why the study was undertaken?
  2. Was the background information provided adequate to understand the aims of the study?
Methods

  1. Were the methods described in sufficient detail for others to repeat or extend the study?
  2. If standard methods were used, were adequate references given?
  3. If methods were modified, were the modifications described carefully?
  4. Have the authors indicated the reasons why particular procedures were used?
  5. Have the authors indicated clearly the potential problems with the methods used?
  6. Have the authors indicated the limitations of the methods used?
  7. Have the sources of the drugs been given?
  8. Have the authors specified the statistical procedures used?
  9. Are the statistical methods used appropriate?
Results

  1. Were the experiments done appropriate with respect to objectives of the study?
  2. Do the results obtained make sense?
  3. Do the legends to the figures describe clearly the data obtained?
  4. Are the data presented in tabular form clear?
  5. Are the legends to the tables clear?
  6. Has appropriate statistical analysis been performed on the data?
Discussion

  1. Were the objectives of the study met?
  2. Do the authors discuss their results in relation to available information?
  3. Do the authors indulge in needless speculation?
  4. If the results obtained were statistically significant, were they also biologically significant?
  5. If the objectives were not met, do the authors have any explanation?
  6. Do the authors adequately interpret their data?
  7. Do the authors discuss the limitations of the methods used?
  8. Do the authors discuss only data presented or do they refer consistently to unpublished work?
References

  1. Do the authors cite appropriate papers for comments made?
  2. Do the authors cite their own publications needlessly?
Abstract

  1. Is the abstract intelligible?
  2. Does the abstract accurately describe the objectives and results obtained?
  3. Does the abstract include data not presented in the paper?
  4. Does the abstract include material that cannot be substantiated?


To review scientific work, the first question should be: Is the study free of bias? According to “High School Students’ Critical Evaluation of Scientific Resources on the World Wide Web,” (Nathan Bos, University of Michigan-Ann Arbor Accepted for publication in the Journal of Scientific Education Technology. http://www-personal.si.umich.edu/~serp/work/critical_evaluation.pdf 10.31.03), when analyzing any study, it is important to begin by knowing who funded or sponsored the study. Why? To determine if there may be any potential biases to the study as a result of the funding. Organizations funding research that are dedicated to a certain “constituent” or outcome are likely to fund a study and a group that in the past has shown their findings to be in line with the goal of the funding agency. Frequently, a funding agency reserves the final right of refusal of publication of the study results, which can also result in biased research.

The next important question is: Was the study published in a scientific journal? This is critical for validating the results of the study. This is not to say that a published study has no flaws or that its conclusions are inherently correct. But, according to “Systematic Critique – the art of scientific reading” (PDF file) (©2002 Biomedical Scientist: February 2002 http://www.ibmsscience.org/reading/systematic_critique.pdf10.31.03), when a study is submitted for publication to a peer-reviewed journal, the entirety of the study is subject to scrutiny. That includes the original hypothesis (for instance, what effect cats have on bird populations), the materials and methods used to study the hypothesis, and the results and the conclusions of the study. Each of these areas is carefully examined in a peer-review process. The authors are required to address each concern raised by the reviewers, either by changing or modifying the text to comply with the comments of the reviewers (who are examining both the methodology and the conclusions of the study and may not agree with the research conclusions as presented by the study authors); or by persuasively arguing why their original statements/conclusions are correct. The editors of the journal ultimately have the power to decide if the authors have adequately addressed the concerns of the reviewers or not. If they have, the article is published (usually with numerous modifications, it’s very rare a paper gets accepted in its first form). If they have not, the article is not published. If the reviewers believe adequate controls or proper methodology were not used in the study, the study will not be published. So when a research study is not published in a peer-reviewed journal, it has not gone through the rigorous screening process by people in the same field. Essentially, within the scientific community there is a reluctance to rely heavily on unpublished studies, because their validity or scientific rigor cannot be adequately addressed.

Also important to note in any review: Do the authors of any given work reference their own material? Are proper reference citations used? Dr. P.K. Rangachari warns that needlessly quoting your own studies to support your findings is generally unacceptable for obvious reasons. We note one very important caveat to this: if you are the only scientist/researcher conducting research in the particular field. If you have been breaking new ground and have the only published studies in your field of research, there is no choice but to reference your own work. But citing your own work to support conclusions when other studies in the same field of work have been published is simply unacceptable. Also, citing “other studies” that support your findings is not acceptable either. Any credible work will properly cite all reference material.

Finally, was the methodology of the study sound? Are projections made on the basis of single-point studies or averages of multiple studies? One of the tactics frequently used by bird conservationist and other wildlife activists is to extrapolate the implication of a study on small numbers of animals to large populations of animals. This practice is deceptive, inaccurate, and statistically unacceptable scientifically. This is an extremely important point.

For further reading on the subject of critical analysis of scientific research, please visit:

An Introduction to Critical Analysis of Publications in Experimental Biomedical Sciences” ©1994-2001 P.K. Rangachari. If you have trouble with the above link, please copy and paste this web address directly into your browser: http://www.science.mcmaster.ca/biopharm/critanal.htm10.31.03.

Systematic Critique – the art of scientific reading” (PDF file) ©2002 Biomedical Scientist: February 2002 for further reading on the critique of scientific research reviews. If you have trouble with the above link, please copy and paste this web address directly into your browser: http://www.ibmsscience.org/reading/systematic_critique.pdf 10.31.03.

“High School Students’ Critical Evaluation of Scientific Resources on the World Wide Web,” Accepted for publication in the Journal of Scientific Education Technology Nathan Bos, University of Michigan-Ann Arbor. If you have trouble with the above link, please copy and paste this web address directly into your browser: http://www-personal.si.umich.edu/~serp/work/critical_evaluation.pdf 10.31.03.

Regards

Malky
 
Well, having read your post in its entireity (very valuable for me, being a young scientist in the process of authoring my first publication!!), I have to say YES, we can 'trust' publications in peer-reviewed scientific journals. The system is not perfect, and scientists are after all only humans, but 99% of papers that get published will be honest interpretations of well-designed investigations that have been subjected to rigorous and unbiased scrutiny prior to publication. That's not to say that the papers are not then open to reasoned criticism, especially when they are dealing with controversial or extremely difficult subject matter (which, I have to say, constitutes a tiny proportion of research), but basically, in my opinion, most scientists are out to present the public (as far as they are interested!!) in 'the truth'.
James
 
I need to use four and a punctuation mark.
You're having a laugh!
The pedantic amongst you can argue wether You're is one word or two.
 
Last edited:
As a working scientist, not in the life sciences though, I would give a hesitant yes to refereed publications. Hesitant because cases of outright fraud, although few, have been found in many disciplines. In general one should keep in mind that "extraordinary claims demand extraordinary evidence...", thus the more important, revolutionary etc, the results claimed the better scrutinized the report should be.

Dalcio
 
No one shouldn't accept the conclusions of scientific papers without question, even in refereed journals (imo!).

Scientists, like everyone else, are susceptible to 'fads'. When an attractive new idea appears, a 'bandwagon' atmosphere may develop in the scientific community and data that does not fit the new concept may not be given sufficient consideration. What may then happen is that one hypothesis is permitted to override and overrule all other hypotheses, regardless of actual evidence.
 
It's always good to question scientific research, but in general I think peer refereed studies are fairly trustworthy. The big question is, if you don't use scientific studies as a basis for conservation decisions what do you use? Gut-feeling and anecdotal evidence?

Richard
 
On the whole I'm with Tim on this one, though I think it depends on the science. As a chemist with around 50 peer-reviewed publications I'm hardly disinterested. However I suspect chemistry is a field one can have more confidence in as it is more of an "exact" science.
Biological sciences are necessarily less exact but still, in the vast majority of cases, entirely trustworthy. As Richard has said, what are the alternatives?

Rob
 
There is a good system in place in science called "ambition" - basically, if methods and conclusions are not sound, they will certainly be found out by re-examination at a later date. One particular paper taken in isolation is essentially just the isuue from a particular direction and most are perfectly valid. Occasionally, an issue can be examined from a slightly different perspective and the results come out a little different (although not always invalidating the original conclusion). Mostly, the new idea confirms the original and it is only in a very small number of cases that the original idea is completely overturned or a counter argument is produced. There are some issues in palaeontology that are open to interpretation rather than a definitive conclusion and long-standing disputes are common but that is less the case in pure biological topics such as predator - prey relationships and population dynamics that we were discussing last week.

Ian
 
In my opinion, yes, you can trust.

But if you are non-scientist, your main problem is to distinguish peer-reviewed journals from any world news beginning with "scientists discovered that...".

If I hear about interesting "scientific discovery" in the news, I generally quickly take keywords and google it or pubmed it to the original publication. So you read the facts without any stupid misunderstanding and interpretation which journalists can add to it.

If you read something cracking in nature (like Flores man discovery recently), usually it is in Science or Nature and they have good popular introductions which you can read. They are aimed for scientists who are not specialists in the field. For medicine, trust Lancet. About extinct animals, everything is discussed in depth on archives of "Dinosaur mailing list". About bird ecology, behaviour - there are some papers.

But I am very suspicious about any stories about dangers to nature coming from conservation bodies - even Birdlife or WWF. They normally take part in shouting contests with politicians which follow the rule: "if you describe everything twice worse than it is, people will do half what is necessary". Sorry for Ian here. :) If they talk about specific count of species, or specific threat it is OK. When they talk about broad things like climate change, GMOs etc. - I don't trust them.
 
jurek said:
But I am very suspicious about any stories about dangers to nature coming from conservation bodies - even Birdlife or WWF. They normally take part in shouting contests with politicians which follow the rule: "if you describe everything twice worse than it is, people will do half what is necessary". Sorry for Ian here. :) If they talk about specific count of species, or specific threat it is OK. When they talk about broad things like climate change, GMOs etc. - I don't trust them.

Hi Jurek,

Actually, I understand what you mean here and sometimes, it is a case of conscience. The RSPB has been caught out by reporting issues in too general terms but that is not a problem because it is often something that has been overlooked in the wording. The word "could" is often overlooked by the reader (and occasionally, omitted by the author mostly by accident) so it is better to overstate the issue. However, the RSPB (and others) is bound by the rules of the charities commission and we can and should be corrected if erroneous information has been released (it has happened). The larger issue comes out of commenting on one bit of scientific work and ignoring another, which is too easy to do with climate change but less easy to do with population information. The trouble with projecting into the future is that we just don't know - or do we? Perhaps when we look at the past, we do indeed have enough experience to draw upon. Biology was once thought to be the lesser science because it could not easily be pinned down to rules. Yet, the more research is done, the more we realise there are rules albeit much more complex than in physics or chemistry. That is why I made the point that re-examination of research often validates the original concept, which is something that is happening more and more in behavioural and population biology.

Ian
 
I think scientifically produced papers can in general be beleived and can as been said challenged .The only problem I have is when these papers are produced by people with a pre ordained agenda , by this I mean when the research is carried out by or on behalf of either a big organisation or mega rich company , as in the past this tends in my opinion to skew the data in favour of the interested party or maybe Im just getting cynical in my old age.
 
The Tom said:
I think scientifically produced papers can in general be beleived and can as been said challenged .The only problem I have is when these papers are produced by people with a pre ordained agenda , by this I mean when the research is carried out by or on behalf of either a big organisation or mega rich company , as in the past this tends in my opinion to skew the data in favour of the interested party or maybe Im just getting cynical in my old age.

Can't think of any big conservation organisations that do that!! ;)

My personal gripe is when minority groups quote credible scientific papers out of context to suit their own agendas. So similar to your point, just a different perspective.
 
I think its all been said.. no one is infallible... but its well known that there is nothing a scientist like to do more than prove another one wrong....
 
Looking back to Malky's original question, he asked if scientific papers can be trusted. Surely the answer to this is going to depend on the nature of the subject under discussion, isn't it? And, as 'edenwatcher' has pointed out, whether that subject can be described as an "exact" science or not will also have a bearing on the reader's degree of confidence in the results quoted.

When it comes to scientific papers dealing with matters connected with wild birds, all too often the results contain one or more of a number of words which, I feel, demonstrate an element of doubt, a degree of uncertainty or even a lack of knowledge on the part of the author. By way of explanation, a quick trawl through some of the extracts taken from a variety of scientific papers used on another thread revealed that the following words and word combinations were cropping up noticeably:-

probably; may (may also, may lead, may suggest, may be); could (could have, could be); speculate; generally.

Would it be fair to suggest that words such as these have no place in any scientific paper which is intended to be a yardstick by which other people's efforts are judged?

Anthony

Edit - Posted before I read Ian's posting #13.
 
Last edited:
Anthony Morton said:
Looking back to Malky's original question, he asked if scientific papers can be trusted. Surely the answer to this is going to depend on the nature of the subject under discussion, isn't it? And, as 'edenwatcher' has pointed out, whether that subject can be described as an "exact" science or not will also have a bearing on the reader's degree of confidence in the results quoted.

When it comes to scientific papers dealing with matters connected with wild birds, all too often the results contain one or more of a number of words which, I feel, demonstrate an element of doubt, a degree of uncertainty or even a lack of knowledge on the part of the author. By way of explanation, a quick trawl through some of the extracts taken from a variety of scientific papers used on another thread revealed that the following words and word combinations were cropping up noticeably:-

probably; may (may also, may lead, may suggest, may be); could (could have, could be); speculate; generally.

Would it be fair to suggest that words such as these have no place in any scientific paper which is intended to be a yardstick by which other people's efforts are judged?

Anthony

Presumably the inclusion of such words/combinations is to provide the authors with a get-out if a given conclusion is challenged at some point in the future. Just like politicians, they're avoiding being too precise, as it's easier to be taken up on a definitive statement. Broadly speaking I'd have hoped that the whole point of a scientific paper was to air a new opinion substantiated by relevant data; I guess some papers aim to make a specific point beyond all question, and others seek to further a debate or advance knowledge one step further... maybe the words you identify as indicative of uncertainty might simply be a tacit acknowledgement that there will still be room for more research about a subject?

Plenty of uncertainty in my musings, maybe even indicating a lack of knowledge... ;)
 
Anthony Morton said:
When it comes to scientific papers dealing with matters connected with wild birds, all too often the results contain one or more of a number of words which, I feel, demonstrate an element of doubt, a degree of uncertainty or even a lack of knowledge on the part of the author. By way of explanation, a quick trawl through some of the extracts taken from a variety of scientific papers used on another thread revealed that the following words and word combinations were cropping up noticeably:-

probably; may (may also, may lead, may suggest, may be); could (could have, could be); speculate; generally.

Would it be fair to suggest that words such as these have no place in any scientific paper which is intended to be a yardstick by which other people's efforts are judged?

Anthony
Completely agree. Sadly (or should I say, criminally), this type of language isn't limited to the world of wild birds. It is rife in other controversial environmental issues. What is particularly disturbing is when speculative language is used by scientists who are underwritten by government and/or military agencies that have virtually unlimited funds available to them so that, if the will were there and hidden agendas not paramount, the reason for the speculation could be eliminated. By "criminally" I mean that such "evidence" is used to gain public/legal (Congressional, in US parlance) approval for going ahead with expensive and extremely hazardous projects and using bogus "science" as justification.
 
Katy Penland said:
Completely agree. Sadly (or should I say, criminally), this type of language isn't limited to the world of wild birds. It is rife in other controversial environmental issues. What is particularly disturbing is when speculative language is used by scientists who are underwritten by government and/or military agencies that have virtually unlimited funds available to them so that, if the will were there and hidden agendas not paramount, the reason for the speculation could be eliminated. By "criminally" I mean that such "evidence" is used to gain public/legal (Congressional, in US parlance) approval for going ahead with expensive and extremely hazardous projects and using bogus "science" as justification.

Hi Katy,

Yes, I agree but the trouble usually starts when 'The Other Side' latch on to it and replace all the little 'doubt' words with 'WILL' - that's the time to head for the hills! ;)

Anthony
 
Last edited:
Warning! This thread is more than 19 years ago old.
It's likely that no further discussion is required, in which case we recommend starting a new thread. If however you feel your response is required you can still do so.

Users who are viewing this thread

Back
Top