Is it Really in the Water? A Critical Reexamination of Toxic Metals in Reef Tanks
Part I
By Richard Harker
During my recent talk at the International Marine Aquarium Conference, I outlined the evolution of modern reef keeping explaining that the hobby had evolved through a series of stages to reach its current status. I suggested that the haphazard trial and error that had helped the hobby reach its current level of understanding would carry us no further. To continue to evolve and grow, the hobby had to enter a new phase where scientific methods replaced the "voodooism" that has characterized too much of what had guided the hobby to this point.
A recent series of articles on metals in artificial seawater and reef tanks would seem to be an example of what I was advocating. (Shimek 2002a 200b) Although that is true to some degree, the series also illustrates the potential pitfalls and dangers that we face as we enter this new phase of reef keeping. The response to the articles also demonstrates that many reef hobbyists are naive regarding scientific methods and are ill prepared to interpret the validity and usefulness of seemingly significant research.
If the hobby is to benefit from scientific methods, hobbyists need to develop a greater understanding of the principles and methods of science as well as an ability to distinguish between good and bad science, The goal of this and subsequent articles examining the reef tank metals articles is to help develop a sufficient understanding within reef keeping that hobbyists can critically judge articles that appear in hobbyist literature.
What is science?
In its most basic form, science is simply systematic study of phenomena. The Oxford Dictionary defines science a little more elaborately as, "a connected body of demonstrated truths or with observed facts systematically classified and more or less comprehended by general laws, and which includes reliable methods for the discovery of new truths." There are many moving parts to the definition and each matters in deciding whether something is science. It has also been said that the history of science consists of a series of conjectures and refutations. That's because in science, most conjectures are wrong and few are right. In other words, scientists are wrong more often than they are right. So one might more accurately describe scientists as seekers of falsehoods. (This is one key difference between the psyche of the scientist and that of the reef keeper. Scientists accept error as inevitable and the uncovering of errors of others as essential elements of scientific progress. Many reef keepers seem uncomfortable with this notion.)
Scientific knowledge has grown through the centuries as scientists have built on the discoveries (and errors) of previous work. Observations led to hypotheses, that in turn led to tests and experiments. Some experiments supported hypotheses while others disproved them, but each step of the way scientists learned a little more about our world. Scientific methods refer to the use of tested and accepted methods to study phenomena. The use of accepted scientific methods lends credence to one's findings and help other scientists understand the results. It also helps others replicate the studies to confirm their findings.
The metals studies are useful in illustrating both the use and abuse of scientific methods. In the following sections, I'll examine each stage of the studies and explain what is right and wrong about the author's methods and conclusions.
Scientific work involves a series of steps, all of which determine the accuracy and usefulness of the work. A misstep at any point in the process can undermine the entire process, so each step must be closely examined. Scientific papers generally begin with an introductory section explaining the central question under study. This section will generally review previous studies and relevant literature. It will explain why the study is important, and outline the hypotheses to be tested. In the metals articles the author asserted that metal concentrations in the average reef tank were significantly higher than on natural reefs and that scientific studies had conclusively demonstrated that at the levels found in reef tanks metals were toxic to marine organisms, He proposed that metal accumulation in reef tanks might explain why some tanks deteriorate over time (old tank syndrome).
When critically examining research, one should first question the premises of the author. Are the arguments of the author logical? Given what we know, are his assertions reasonable? If they are, will the approach he proposes address the issues he raises?
Methods-and why you can't always trust the government
Scientific papers always have a methods section devoted to outlining the methods used in the study. One way to judge a study is to examine the methods used and see if the methods are a reasonable means to test the author's hypotheses. To prove that the average reef tank has high levels of metals, one has to measure metal levels in a reasonable number of tanks representing a cross section of hobbyists. Ideally, a sample of participants would be drawn reflecting all the possible variables that might affect metal levels. The age of each system, the experience of each reef keeper, the different sources of make-up water, and so on should be considered in designing a sample. In the case of this study, the author solicited the help of hobbyists who would be willing to submit tank water for evaluation and pay for the analysis. Ultimately, only 23 hobbyists submitted samples. In statistical terms, this is a self-selected sample. Rather than sample a cross section of tanks, the author simply accepted whoever had the money and inclination to participant. As a rule, self-selected samples tend to be unrepresentative of large populations. With only 23 self-selected participants, one should be quite cautious about assuming that any analyses of these tanks can be extrapolated to the hobby as a whole.
The methods section should also outline how the levels of metals will be determined. In the metals articles, the author listed the method as, "Inductively Coupled Plasma Emission Spectrometry or ICP Scan, EPA method 200.7." A long impressive name like this lends an air of authenticity to the work. Most hobbyists have probably never heard of the technique and are in no position to judge the appropriateness of the method. In a scientific paper, the methods section will often present an explanation of why a certain technique was chosen. It will also explain any limitations in the method. The series author, unfortunately, did not address limitations in the method chosen. A review of the scientific literature on the subject of measuring metals in seawater finds significant problems with the ICP method. (Crompton 1989) Saltwater is a complex soup of chemicals in widely varying concentrations. For technical reasons addressed in a recent column by Randy Holmes-Farley, an ICP scan has great difficulty differentiating metals, particularly toxic metals. (Holmes-Farley 2003) Scientists do use ICP scans to study seawater, but the metals are first concentrated using resins or other methods.
So why has the EPA approved the use of ICP scans? Because they are fast and inexpensive. The Federal government's interest is in finding a method that can provide cost effective data for monitoring sites and enforcement of environmental laws. Other methods are more sensitive, but more time consuming and costly. Consequently, an EPA endorsement of any methodology is not evidence that it is the most accurate or useful for scientific studies.
Every experiment and study has multiple potential errors, and it is important to consider how the author deals with potential error. The samples were collected by individual hobbyists and then shipped to the author who in turn shipped the samples to the lab. This means that 23 different people collected the 23 samples. No attempt was made to filter particulates out of the tested water, and it isn't clear how careful the 23 hobbyists were in collecting their samples. Under these circumstances, the risk for contamination is great. The metals of interest are in extremely small concentrations, and it would be easy for a hobbyist to inadvertently introduce foreign substances into his sample. For example, even if rinsed repeatedly, using the same cup one uses to add supplements or feed the tank would inevitably contaminate the sample water. In a study of professional marine scientists, it was found that even professionals produced wildly varying results when analyzing metals in seawater, probably because of contamination. Because contamination is so easy when testing for metals, very elaborate procedures have been developed to make sure that contamination is minimized at each stage of analysis. The author makes no mention of handling techniques, so it is unlikely that the hobbyists involved used accepted procedures for handling the water.
(CONT)
Part I
By Richard Harker
During my recent talk at the International Marine Aquarium Conference, I outlined the evolution of modern reef keeping explaining that the hobby had evolved through a series of stages to reach its current status. I suggested that the haphazard trial and error that had helped the hobby reach its current level of understanding would carry us no further. To continue to evolve and grow, the hobby had to enter a new phase where scientific methods replaced the "voodooism" that has characterized too much of what had guided the hobby to this point.
A recent series of articles on metals in artificial seawater and reef tanks would seem to be an example of what I was advocating. (Shimek 2002a 200b) Although that is true to some degree, the series also illustrates the potential pitfalls and dangers that we face as we enter this new phase of reef keeping. The response to the articles also demonstrates that many reef hobbyists are naive regarding scientific methods and are ill prepared to interpret the validity and usefulness of seemingly significant research.
If the hobby is to benefit from scientific methods, hobbyists need to develop a greater understanding of the principles and methods of science as well as an ability to distinguish between good and bad science, The goal of this and subsequent articles examining the reef tank metals articles is to help develop a sufficient understanding within reef keeping that hobbyists can critically judge articles that appear in hobbyist literature.
What is science?
In its most basic form, science is simply systematic study of phenomena. The Oxford Dictionary defines science a little more elaborately as, "a connected body of demonstrated truths or with observed facts systematically classified and more or less comprehended by general laws, and which includes reliable methods for the discovery of new truths." There are many moving parts to the definition and each matters in deciding whether something is science. It has also been said that the history of science consists of a series of conjectures and refutations. That's because in science, most conjectures are wrong and few are right. In other words, scientists are wrong more often than they are right. So one might more accurately describe scientists as seekers of falsehoods. (This is one key difference between the psyche of the scientist and that of the reef keeper. Scientists accept error as inevitable and the uncovering of errors of others as essential elements of scientific progress. Many reef keepers seem uncomfortable with this notion.)
Scientific knowledge has grown through the centuries as scientists have built on the discoveries (and errors) of previous work. Observations led to hypotheses, that in turn led to tests and experiments. Some experiments supported hypotheses while others disproved them, but each step of the way scientists learned a little more about our world. Scientific methods refer to the use of tested and accepted methods to study phenomena. The use of accepted scientific methods lends credence to one's findings and help other scientists understand the results. It also helps others replicate the studies to confirm their findings.
The metals studies are useful in illustrating both the use and abuse of scientific methods. In the following sections, I'll examine each stage of the studies and explain what is right and wrong about the author's methods and conclusions.
Scientific work involves a series of steps, all of which determine the accuracy and usefulness of the work. A misstep at any point in the process can undermine the entire process, so each step must be closely examined. Scientific papers generally begin with an introductory section explaining the central question under study. This section will generally review previous studies and relevant literature. It will explain why the study is important, and outline the hypotheses to be tested. In the metals articles the author asserted that metal concentrations in the average reef tank were significantly higher than on natural reefs and that scientific studies had conclusively demonstrated that at the levels found in reef tanks metals were toxic to marine organisms, He proposed that metal accumulation in reef tanks might explain why some tanks deteriorate over time (old tank syndrome).
When critically examining research, one should first question the premises of the author. Are the arguments of the author logical? Given what we know, are his assertions reasonable? If they are, will the approach he proposes address the issues he raises?
Methods-and why you can't always trust the government
Scientific papers always have a methods section devoted to outlining the methods used in the study. One way to judge a study is to examine the methods used and see if the methods are a reasonable means to test the author's hypotheses. To prove that the average reef tank has high levels of metals, one has to measure metal levels in a reasonable number of tanks representing a cross section of hobbyists. Ideally, a sample of participants would be drawn reflecting all the possible variables that might affect metal levels. The age of each system, the experience of each reef keeper, the different sources of make-up water, and so on should be considered in designing a sample. In the case of this study, the author solicited the help of hobbyists who would be willing to submit tank water for evaluation and pay for the analysis. Ultimately, only 23 hobbyists submitted samples. In statistical terms, this is a self-selected sample. Rather than sample a cross section of tanks, the author simply accepted whoever had the money and inclination to participant. As a rule, self-selected samples tend to be unrepresentative of large populations. With only 23 self-selected participants, one should be quite cautious about assuming that any analyses of these tanks can be extrapolated to the hobby as a whole.
The methods section should also outline how the levels of metals will be determined. In the metals articles, the author listed the method as, "Inductively Coupled Plasma Emission Spectrometry or ICP Scan, EPA method 200.7." A long impressive name like this lends an air of authenticity to the work. Most hobbyists have probably never heard of the technique and are in no position to judge the appropriateness of the method. In a scientific paper, the methods section will often present an explanation of why a certain technique was chosen. It will also explain any limitations in the method. The series author, unfortunately, did not address limitations in the method chosen. A review of the scientific literature on the subject of measuring metals in seawater finds significant problems with the ICP method. (Crompton 1989) Saltwater is a complex soup of chemicals in widely varying concentrations. For technical reasons addressed in a recent column by Randy Holmes-Farley, an ICP scan has great difficulty differentiating metals, particularly toxic metals. (Holmes-Farley 2003) Scientists do use ICP scans to study seawater, but the metals are first concentrated using resins or other methods.
So why has the EPA approved the use of ICP scans? Because they are fast and inexpensive. The Federal government's interest is in finding a method that can provide cost effective data for monitoring sites and enforcement of environmental laws. Other methods are more sensitive, but more time consuming and costly. Consequently, an EPA endorsement of any methodology is not evidence that it is the most accurate or useful for scientific studies.
Every experiment and study has multiple potential errors, and it is important to consider how the author deals with potential error. The samples were collected by individual hobbyists and then shipped to the author who in turn shipped the samples to the lab. This means that 23 different people collected the 23 samples. No attempt was made to filter particulates out of the tested water, and it isn't clear how careful the 23 hobbyists were in collecting their samples. Under these circumstances, the risk for contamination is great. The metals of interest are in extremely small concentrations, and it would be easy for a hobbyist to inadvertently introduce foreign substances into his sample. For example, even if rinsed repeatedly, using the same cup one uses to add supplements or feed the tank would inevitably contaminate the sample water. In a study of professional marine scientists, it was found that even professionals produced wildly varying results when analyzing metals in seawater, probably because of contamination. Because contamination is so easy when testing for metals, very elaborate procedures have been developed to make sure that contamination is minimized at each stage of analysis. The author makes no mention of handling techniques, so it is unlikely that the hobbyists involved used accepted procedures for handling the water.
(CONT)