Making Sense of Research

by Thomas Incledon

What Do You Believe?

One study indicates that a supplement works and another study claims that the same supplement resulted in "no significant differences," or in other words, it didn’t work. A few weeks later, you read in the newspaper that the lead investigator concludes, "More research needs to be done in this area." With the widespread access to the web and other information resources, emphasis has been placed on finding out all the facts in order to make an informed decision. Part of this process can be running simple searches on Medline and other databases. Now here you are with the studies you were looking for, yet for some reason, things seem even more confusing. This article should help clear up some of that confusion by arming you with tactics and strategies to make sense of the information from scientific studies.

Pick up any magazine, look at the label on a product from the health food industry, or read an advertisement on the same, and you are bound to find little numbers referencing statements throughout the article, label, or ad. This is a step in the right direction, as claims should be referenced. This makes it easier for the consumer to follow up on the claim by looking for the published paper and reading the study used to support the claim. In lay terms, a scientific study is simply a systematic, ordered approach to examining something. From this simple definition, it seems like anyone can do a study, and this is probably true in most cases. But how would you know if the study I did in Florida was performed to the same standards that your study was when you may live in California or in another country? To deal with this and other issues, scientists submit a write-up of their work to a journal to be "peer-reviewed." During this process, other scientists with a strong background in the topic under investigation review the submitted information. They check the research study design, the techniques used to measure changes, and the conclusions drawn. To maintain fair and equal standards, journals produce guidelines for the submission of papers. The guidelines state the conditions for accepting the information, the order in which the information needs to be presented, and the style of referencing. After a paper is submitted to a journal, the peer-review process can take some time. Papers can be accepted with minor changes recommended, accepted with major changes recommended, or they can be rejected. A rejected paper can be rewritten and then submitted to another journal, and the process begins again.

Once a study is published, anyone can read it. The information is no longer private, because it is in the public domain. Many companies will do a study "in house," meaning they performed all the treatments and measurements. They then make this information available only in ads, never submitting it to credible journals for review. When asked why they do not submit the articles for peer review, they may cite proprietary interests. This is a very silly answer because in most fields, everyone knows everyone else, and they already know what each other is doing anyway. Regardless, there are many ways that quality research can be done and still protect the proprietary interests of the company. Another reason why a company doesn’t publish a study is that they realize the information wouldn’t be publishable in a peer-reviewed journal.

The key point about a study is that it is usually published in a peer-reviewed journal. There are legitimate studies done by various organizations but the data may never be intended for publication and instead may be used for limited or internal use only. Newsletters, advertisements, photocopies without journal titles, Special Ad-Reports, non-journal web sites, and magazine articles are not acceptable sources for studies. These sources represent what the general public reads though, so this is what companies target as a means of increasing product sales.

Suppose you wanted to test one of the new supplements out on the market. You take a few capsules and tell everyone it works because you lifted more weight in the gym. You’re sold on the product. In research, things aren’t that simple. If you know what you are taking, that can bias you towards having certain results. So to make sure it is actually the supplement that is working, investigators don’t tell you what you are taking. This is called single blind. If the investigator knows what the subject is taking, even though the subject does not, the investigator may unintentionally bias the results. To prevent this from happening, investigators make sure that neither the subject nor investigator know who is taking the supplement. This is called a double-blind study. Another consideration is that the very act of taking a pill can have an effect. This is called the placebo effect, and it can be negative or positive. By planning for this effect, we say the study was placebo controlled, meaning some people received a placebo (an inert pill) and others received a supplement. Normally, a study would involve two groups, one receiving the treatment (or supplement) and the other receiving no treatment (or a placebo). Because of the many differences between people, it makes it difficult for scientists to explain if what is happening is due to the treatments or a random occurrence. To minimize errors in this area, subjects can be divided into two groups where the first group receives the supplement and the second group receives the placebo. After a defined period of time, subjects are switched or "crossed-over." To minimize carry-over from one period to the next, a washout period is used so that the supplement results do not interfere with the placebo results, or vice versa.

From the above, you can see that a sound study design to test a supplement involves a double blind, placebo-controlled, crossover design, with an adequate washout period. If the duration of the effects of a supplement is not known, it is better to use two separate groups for comparison and then continue to monitor the groups after the supplement and placebo are discontinued. Other study designs include adding more groups to compare the effects of different amounts of treatment or different treatments, longitudinal studies where groups are monitored periodically over a long period of time, and open container trials where everyone knows what is going on. All of these designs have advantages and disadvantages.

Most people would agree that if numerous studies from different labs were published in peer reviewed journals and found similar results, that this is sufficient evidence. The more difficult dilemma is trying to interpret evidence when there is only a single published study, when the data is from cells, when the data is from animals, or the study deals with a specific population. These additional sources can have value. They help to provide the conceptual framework for interpreting all of the available research information as a whole. Before doing clinical trials with humans, it’s important to know that the supplement, drug, etc is safe. Usually this process begins with cell studies. Issues often overlooked are the concentrations of the agent used to test the cells. For example, if the concentration of glutamine in a culture of skeletal muscle cells was many times higher than what one would see in the blood, it does not mean that at lower concentrations we would get the same results. It does, however, provide us with an understanding of the possible mechanisms by which glutamine could effect muscle cells. What if the agent treating the muscle cells had a structure that permitted very poor absorption across the GI tract? The fact that it affected cells is significant, but the fact that it cannot cross the GI barrier means that there is limited application. Another issue often overlooked is the passage of the cells. A primary cell line is fresh out of the tissue. This cell line can then be frozen. In the future, samples could be taken and studied. As the cells decrease in supply, allowing a sample of cells to culture can generate more cells. Each time a new batch is generated for the purpose of creating a new supply of cells to take samples from, it is called a passage. Multiple passage cells are not the same as early passage or primary cells. This means the effects observed may or may not be applicable. Other concerns when reviewing cell data include the absence of physiologically significant factors such as hormones, growth factors, proteins, antioxidants, vitamins, etc. In an animal or plant, cells are bombarded by numerous factors, yet in culture many of these factors are not present. The results we observe after a treatment may have something to do with the absence of one or more factors. Experience and education teach us what information is useful and what is not.

After cell studies have established that an agent is not toxic to cells, animal studies may be done. The purpose of these experiments is to understand the physiological effects of the agent as well as to determine safe and lethal dosages. While animals are similar to humans in many ways, there are often subtle differences that make direct application to humans difficult. It is incorrect to say that animal data never applies, but it is also incorrect to assume it will apply. An understanding of the differences between the animal model tested and humans can make interpretation much easier.

After animal studies have established that a product is safe, human trials may begin. Often the trials involve specific populations (ie AIDS patients, diabetics, etc). Since many diseases are the result of a disturbance in normal function of the body, it is not a good idea to assume that improvements in sick population will apply to a healthy population. Additionally, age, gender, and ethnic differences also exist. This may limit the application of results from one group to another.

A simple strategy for making sense of the information can be used. This strategy is based on a presentation of "Direct vs Indirect Evidence" by Jose Antonio, PhD. Direct evidence is a study on humans that can be applied as is. An example would be a study that shows a given weight loss product causes weight loss in obese males. No side effects were found, so one could see that at least for the time period studied, the product worked and was safe to use in obese men. Whether or not this applies to women, different age groups, leaner men, etc. remains to be proven. If one applies this evidence from obese men to other populations, this is extrapolating. This is often done and, in most cases, differences between the populations are not that great and future research supports applying data from one group to another. Understanding the differences between populations minimizes the misapplication of data from one group to another.

Indirect evidence is supporting research from cell studies, animal studies, and different populations other than the actual population the data will be applied to. This is the area that many supplement companies and speculators love. The reason is that they are allowed to use their creativity to apply information from unrelated samples to humans. For example, if a supplement shows it counteracts the effects of cortisol in isolated rat muscle cells, will this apply to humans? Obviously, the company selling the supplement wants you to think this, but we really need to know how this takes place in the rat muscle cells. Then it needs to be compared with what we know about human muscle cells. Then we investigate the possibility for human blood levels to reach the concentration studied in a safe and practical fashion. If all of these areas hold up, it may actually apply. On the other hand, if the rat uses a different mechanism than the human, it may not be sound to apply the information. Regardless if it can be applied or not, we still need data on humans to validate that the effects do occur in humans as well.

As individuals, we have our own little lines drawn as far as what is acceptable information and what is not. Many times health care providers and scientists are criticized because they do not give interpretations of the research, or in other words they refuse to comment. This is a safety precaution, not a weakness. These groups are responsible for passing out balanced and accurate information. They are accountable for this. They cannot make statements after every paper that is published. They wait until enough evidence is available and then issue a position or policy statement. This is slow and often not popular with the lay public. So lay people turn to other areas for information. Many times people with very suspect backgrounds are in a position to offer advice. The rule of thumb is if they sell the product they are talking about, disregard most of what they say. Financial ties to products have influenced many people to bias their interpretation. By treating information from these sources as suspect, you are less likely to be taken advantaged of by misinformation. If the information is accurate, then other sources will eventually substantiate it.

The next few sections will overview each area of a scientific paper. In general, the first thing you see (other than the title and names of the researchers) is an abstract. This is simply a summary of the entire paper. Only key points are mentioned. The next part of the actual paper is the introduction. Here the conceptual framework is outlined that supports what the researchers expect to find. Their expectation is called a hypothesis. The introduction is usually very concise without any detailed explanations of mechanisms or how things occurred. The details are often saved for the Discussion section of the paper.

The most difficult part of any study for most people is the methods. These are simply the techniques and tools used to perform the measurements in the study. Finding problems in this section is difficult. Research experience in the area being reviewed is certainly the best education because you learn about the limitations of various methods. For people from a non-research setting or from a different background, there are other options. One is to e-mail an e-mail news group, list, bulletin board, or web site in that area about the techniques or methods. You may get a variety of replies, so find out the background of the people replying. Make sure they have the right credentials to offer an explanation. Another option is to run searches on Medline looking for review papers on the method or topic. Then you can obtain a copy of the paper and read what scientists have to say are the concerns regarding the use of the methods in question.

Sometimes, one does not need to have a formal background in research to interpret the appropriateness of a method. Years ago, several papers were published where push-ups were used as a measure of strength. Strength is defined as how much you can lift one time for a given movement pattern. Since push-ups are done multiple times, they are not a valid measure for determining upper body strength. Even if you didn’t have a formal background in research, you could see the method was not appropriate.

Another thing to consider in the methods section is verification of the product. Over the years, many studies have been done on dietary supplements and the products were never analyzed. Consider whey protein (or any protein) as one example. There are numerous components of whey. These microfractions have biological activity. Without knowing microfraction components and the amounts given to the subjects, we don’t know what was causing the effect. The proteins could have been denatured. A simple chemical analysis would reveal the total protein content, but it would not reveal the denaturation or loss of bioactivity of the proteins.

When results are reported as raw numbers, these are called absolute data. When reported as percentages of improvement, they are considered relative data. Many companies love to show relative data in nice color graphs in their ads. It makes the changes appear more significant. Ideally, the original paper would report both, so that it is easy for the reader to interpret the changes. The results are reported usually as the mean plus or minus the standard deviation (SD) or the mean plus or minus the standard error (SE). The mean represents the average of all the results. This is interpreted as some results were lower and some were higher. There may have been people that had no response to the treatment, while others improved or decreased quite a bit. To give us an idea of the variance of responses to the treatment, we look at the mean and then add or subtract the SD or SE. If every subject appears to have had a positive response to the treatment, then you could conclude the treatment works. However, how do you know if something had an effect when the results don’t appear so obvious?

Statistics help us to determine if something had an effect in the study. A poor study design cannot be corrected by statistics. There are all kinds of statistical techniques and theoretical models that guide researchers as to how they should treat the various components of their data. Explaining these techniques will surely put you to sleep, so let’s consider some things that may be slightly more important. The first thing to consider is quality versus quantity. Sometimes the effects of a treatment appear very small, yet they are significant. Let’s say you took a supplement and it made you 1% faster. Is this important? At the elite level of sports, 1% can be the difference between placing first or second. For the average person, 1% means very little. When interpreting this information then, one has to consider to whom it applies to. As another example, suppose you read a study that indicates a fraction of the original amount administered of a supplement crosses the GI barrier. Statistically this is shown to prove significant. But is this meaningful from a practical sense? The answer is that it depends on the type of substance and how it is metabolized and/or cleared from the body. If it is water soluble and easily removed by the kidneys, the supplement is unlikely to have much of an effect since the concentration is too low relative to the clearance time. Suppose, however, that it is an enzyme or some type of large protein. The concentration may not be as important because the clearance time is so slow.

A common mistake made by many people is to look at a study and see how many subjects there were and look at the level of significance (p value). If you only do this, you may be making a hasty decision. There is no way you can tell if the study used an adequate number of subjects just by looking at the sample size (n). In order to determine if there were enough subjects in a study, you need to know the effect size, the statistical power, and the level of significance. The effect size is used to describe the mean of the difference between the treatment and control (placebo) groups. Effect size and power interact. By knowing these values before a study is started, one has a good chance of finding an effect. This gives us an indication of the strength of the data. A large effect size with a high power means that fewer subjects are needed to see a significant effect. This is somewhat tricky, as these numbers are related. The level of significance (p value) is simply the probability of these results occurring randomly. Most often researchers set the acceptable p value at p = .05. When they run their statistical analysis, if they get a p value less than or equal to .05, it supports the notion that the treatment had an effect. Significance does not tell you the data will apply to everyone or that it will be noticeable. Taking supplement X may increase your bench press by .5%. If you are benching 300 pounds, an increase of 1.5 pounds is unlikely to be perceived significant, since in most gyms the next increase would be 5 pounds to 305 pounds.

This is the section of a paper where the researchers explain their findings and how these results compare with previously reported studies. Ideally, the limitations of the study and how they can be avoided in the future are discussed. This helps readers to interpret the information and avoid making the same mistakes. Since this section of a paper is the researcher(s) interpretations of the results, one has to be careful when reading this. Scientists can easily assume that a product has an effect based upon the statistical significance of their results, yet their study design may not have proven anything from a practical point of view. A common example is a study on an isolated component of a plant. Assume scientists show that giving 50 mg of vitamin C to people decreases free radical production during exercise. Other people read this and assume that they should start taking vitamin C. However, the study did not look at whether or not vitamin C is needed if one is eating foods high in vitamin C already. Chances are that by adding one or two food items, you would get the same or better protection.

At the very end of a paper, an acknowledgement is usually made that thanks the funding organization. Consider that you are getting lots of money from a company to do research in your lab. You study one of their products and find that it has no effect. If you run the study, sales go down for the company on the product and they may stop funding you. We like to think that all scientists are able to rise above these pressures and do the ethical thing, but don’t bet on it. The long term funding needs of the lab will most likely win out.

Hopefully you were able to get some ideas of how to read and interpret scientific papers. The first one is the hardest to make sense of. With time, the technical jargon becomes understandable and before you know it, your head will be filled with all kinds of Trivial Pursuit-type answers. If you have any questions or comments, please don’t hesitate to contact me. Below are some links to various web sites that may be useful.

Medline - http://igm.nlm.nih.gov/index.html
http://www.ncbi.nlm.nih.gov/PubMed/

International Bibliographic Information on Dietary Supplements - http://odp.od.nih.gov/ods/databases/ibids.html

Patent information - http://www.uspto.gov/

National Center for Complementary and Alternative Medicine - http://nccam.nih.gov/

Sports Detective - http://www.sportquest.com/

High Wire Press for free full text articles - http://highwire.stanford.edu/