How Science is Misused and Misunderstood
Note: This is an article that originally appeared on the Pose Method website. It’s currently not available there. I’ll update with links when and if that changes.
Introduction
It’s very common when debating subjects of a scientific nature, for people to quote a study as “proof” of their point or opinion. They may even quote several studies if they have a more sophisticated understanding of the science. The conversation usually ends with the person who referenced the studies walking away smugly, believing that he or she has proven his or her point and the debate is over. While it’s great that people are referencing scientific studies, they often do so incorrectly and inappropriately. Unfortunately, I’m not just referring to scientific laypeople but often to actual scientists, doctors, and engineers. The very people who should know better.
The reality is that not all studies are created equal. Some studies provide great insight, while others provide little value. Designing a good study usually requires a great deal of practical knowledge about the subject, allowing the researchers to avoid collecting data without real meaning or practical use. Unfortunately, it is almost a cliché that many of those doing research often lack the practical experience needed to ask the relevant questions needed to avoid these problems.
How many studies have been conducted on forefoot vs. heel-striking? There have been many, and the data have been very ambiguous. Why? Footstrike is one variable among many in regard to running technique, and it’s not very meaningful without a great deal more context. Studies on stride length are another example. Again, a single variable is not very meaningful by itself.
Generally, these studies add little to what is already known, and yet they keep coming, with many people quoting these studies, believing that they offer valuable insight. The unfortunate result has been that the often heated discussions about running technique have generally not progressed much beyond arguments over foot strike and stride length, lacking any other context and missing the larger and more important concepts.
Common Mistakes in the Interpretation of Studies
When laypeople (and sometimes actual scientists) attempt to interpret studies, they often make certain assumptions that they shouldn’t. Here are some important things to keep in mind.
All studies are not created equal. Studies vary wildly in quality, and no single study is perfect. Before quoting a study, it is important to understand the strengths and weaknesses of its design and the questions those strengths and weaknesses raise. Many studies are deeply flawed, often undermining the author’s conclusions. Because of this, no study should be taken simply at face value. Unfortunately, more often than not, people do no more than read the conclusion and assume that everything else about the study is in order.
Studies rarely, if ever, “prove” anything. Outside of mathematics, “proof” is actually quite rare. All studies must be evaluated in the context of all other related studies, and the value of the data must be weighed based on the quality of the study’s design and methodology. Simply selecting studies that support one’s opinion or “cherry-picking” can be very misleading. Also, if there are only a limited number of relevant studies, one should not draw hard conclusions. It is quite common for further research to expose flaws in earlier research.
Another issue is that many people often assume that the current scientific consensus is definitive. Science is a process, and that process is slow and error-prone. One should always consider the possibility that new research can completely shift the context of preceding research.
I could go on; the design and interpretation of scientific studies are subjects that could fill many books, but the basic point is that designing meaningful and relevant studies is not a straightforward or obvious process.
Limitations with Reductionist Methodology
In school, everyone learns the basics of how science is performed. It goes something like this: Researchers are supposed to strictly control all the variables, leaving all but one constant, methodically and diligently checking every possible combination of the variables in order to fully understand how they interact. Unfortunately, in practice, this is not practical or even possible, particularly in the biological and social sciences and, more specifically, in studies involving human subjects. This reality has important implications when interpreting the meaning of scientific research.
What I described above is called reductionism. This approach assumes that studying the individual variables makes it possible to understand how they all work together. Often, though, we don’t even know what all of the significant variables are. This opens up the concept of ‘Confounding Variables’. If a variable is unknown, it is unlikely to be controlled, and the resulting data will likely be ambiguous. There are statistical methods to help analyze and make sense of such data despite the presence of confounding variables. However, statistical studies can rarely be used to uncover causal relationships. That is to say, they do not directly determine cause and effect. Usually, they lead researchers to likely possibilities for further research.
Researchers lacking much practical knowledge of a subject have no choice but to use the reductionist approach. Basically, they are forced to thrash around, hoping to uncover meaningful data. In new areas of study, there may not be any other choice. However, after data has accumulated and the pieces of the puzzle start to fall into place, there are alternative methods that often lead to deeper and more meaningful insights.
Alignment of Variables in Systems
Reductionist methodology has an important place as the most fundamental form of scientific investigation to be sure. However, in practice, it is slow, cumbersome, and impractical. Firstly, it is seldom possible to control every relevant variable within a study because people have so many differences, such as genetics, diets, dimensions, life experiences, etc. It is simply impossible, or even ethical, to attempt to control everything. Secondly, even if it was possible to control for all of the important variables, it is not always possible to identify them all. The result is that researchers are forced to use statistical tools and methods, which, as I already stated, generally don’t uncover the underlying mechanisms of cause and effect.
So what happens if clear differences in results can only be identified when the variables align in specific ways? This is common in many systems, where ideal efficiency or effectiveness is only achieved when all variables are “just right”. In theory, it is possible to uncover this “alignment of variables” using a purely reductionist methodology, but in practice, it’s neither practical nor likely. For the sake of expediency, researchers are forced to make educated guesses to eliminate as many combinations of variables as possible. However, to do this, they must have some underlying concept or model of the system and how the variables relate to each other. The branch of study called “Systems Science” (also called Cybernetics) addresses this. Systems Science recognizes that the relationships between the variables can often be as important, or even more important, than variations in specific variables. This method of analysis based on relationships is called a “synthesis”. The reductionist approach ultimately must use synthesis to place the variables in some meaningful context. If the context is too limited, the bigger picture is missed, and the results can be misleading. This is also one reason so many studies contradict each other. They lack proper context when analyzed.
Interestingly, evaluating a runner as a system places many arguments about running and running technique in a context that can help explain some perplexing observations. One key element of a system is resiliency. Despite less-than-ideal conditions, systems must often continue working well, or at least well enough. The reason for this is the need for adaptability. If a system is too specialized and only works well under ideal conditions, then it is of limited value. This would explain why so many runners can run well despite having less-than-ideal technique. However, it is important to understand that although they may be running well, they are not achieving their full potential.
Another way to frame this discussion is as “Conceptual Science” versus “Descriptive Science”. The concept of conceptual science is akin to systems science, and the concept of descriptive science is more similar to the reductionist approach. However, one frames the discussion; these concepts are not mutually exclusive. They are interdependent methods when studying complex subjects with many relevant variables. One method is more concerned with the specific variables, and the other is more concerned with their relationships. The problem with a lot of research is that rather than studying the relationships between the variables, many researchers only focus on specific variables, failing to advance our understanding of the subject significantly.
Credits
I would like to credit Ivan Rivera Bours, whose blog runninginsystems.com introduced me to the idea of applying systems science to running. I would also like to thank Ivan for his invaluable assistance and feedback while writing this article.