15 Reasons You Should Think Twice Before Using Percent Removal to Assess BMP Performance

More than a decade ago, members of the American Society of Civil Engineers (ASCE) Environmental and Water Resources Institute’s (EWRI’s) Urban Water Resources Research Council (UWRRC) identified a need to gather sufficient technical design and performance information and data to improve urban stormwater best management practice (BMP) selection and design so that stormwater-quality problems could be cost-effectively addressed. As a result, the International Stormwater BMP Database project was initiated (www.bmpdatabase.org) and is now supported through a collaborative effort of the Water Environment Research Foundation (WERF), American Public Works Association (APWA), Federal Highway Administration (FHWA), US Environmental Protection Agency (USEPA), and ASCE/EWRI/UWRRC.

Photo 1. Stormwater managers and others need reliable data on BMP performance.

One of the first tasks completed by the project team, which includes Wright Water Engineers Inc. and Geosyntec Consultants Inc. (the original project team for the UWRRC also included Ben Urbonas, P.E., D.WRE, of the Denver Urban Drainage and Flood Control District), was to review various approaches used to assess BMP performance. One of these approaches was percent removal. The project team determined that this approach often led to erroneous conclusions regarding BMP performance and advocated alternative approaches to assessing performance; however, percent removal continues to be used by many entities across the country to assess BMP performance or to “qualify” BMPs as providing acceptable performance.

Our project team is frequently asked why we do not use percent removal to assess BMP performance for the approximately 300 BMP studies (as of September 2007) in the database. This article summarizes some key shortcomings of percent removal as a tool to assess BMP performance. While the project team recognizes that percent removal is an easy-to-understand concept that is attractive to many entities and that numerous references provide percent removal information, the following shortcomings are significant and require that an alternative measure (or measures) of BMP performance be used.

1. Percent removal is primarily a function of influent quality. In almost all cases, higher influent pollutant concentrations into functioning BMPs result in reporting of higher pollutant removals than those with cleaner influent (see Figure 1). In other words, use of percent removal may be more reflective of how “dirty” the influent water is than how well the BMP is actually performing. Therefore (and ironically), to maximize percent removal, the catchment upstream should be dirty (which does not encourage use of good source controls or a treatment train design approach).
2. Significant variations in percent removal may occur for BMPs providing consistently good effluent quality. Stated differently, the variability in percent removal is almost always much broader than the uncertainty of effluent pollutant concentrations. These variations in percent removal have little relationship to the effluent quality achieved (see Figure 2).
3.  BMPs with high percent removal (e.g., greater than 80% removal of total suspended solids, or TSS) may have unacceptably high concentrations of pollutants in effluent (e.g., greater than 100 milligrams per liter of TSS), which can lead to a false determination that BMPs are performing well or are “acceptable,” when in fact they are not.
4. Various relationships between influent and effluent concentrations have been demonstrated for a variety of BMPs and designs. The relationships are often complex and are not well represented by a single ratio of inflow-to-outflow concentrations. In addition, many BMPs that are functioning well appear to reach an irreducible concentration. Any measure of BMP performance should be universally interpretable regardless of influent concentration, BMP function, design, number of samples collected, etc.
5. Methods for calculating percent removal are inconsistent (e.g., event by event, mean of event percent removals, inflow median to outflow median, inflow load to outflow load, slope of regression of loads, slope of regression of concentrations). Very different percent removals can be reported from the same data set.
6. Frequently, in many methods, percent removal is dominated by outliers or high-concentration events in a series that have high leverage on an average. The standard reporting of percent removal carries none of the statistical support needed to assess uncertainty in the reported value.
7. Many BMPs that have been monitored do not have enough data to reject the null hypothesis that the influent and effluent concentrations are even different from one another (i.e., we cannot tell if the BMP reduces anything), yet these numbers are published as indicative of performance. Some studies have reported small percent increases in performance erroneously, when in fact the influent and effluent concentrations are not statistically different from one another.
8. When percent removals are applied in modeling efforts, the resulting estimated effluent concentrations can be very misleading—particularly when the effluent quality predicted has not been observed in data sets for the practice being modeled.
9. Many volume-based BMPs have long-term performance that is not evident if a paired inflow-outflow percent removal approach is taken (i.e., material from one event is discharged in another).
10. In terms of meeting receiving-water standards, BMP discharges can comply with receiving-water numeric targets while simultaneously not showing favorable percent removals.
11. Range of expected effluent-quality concentrations is a much better planning and design tool than percent removal estimates. For example, an engineer can use effluent concentrations as a tool to estimate the range of pollutant loading that could be expected at a new development. This is particularly important in sensitive watersheds where it is important to have confidence that BMPs will be adequately protective.
12. The requirement to use percent removals to assess BMP performance can bias monitoring designs. In effect, incentive is provided to monitor BMPs at relatively dirty locations or areas with poor source controls in place so that the BMP performance “looks better.” The project team has seen this intentionally done.
13. Percent removal does not provide a meaningful mechanism to address the well-established concept of irreducible pollutant concentrations expressed by Thomas R. Schueler in Center for Watershed Protection publications (See “Article 65: Irreducible Concentrations Discharged From Stormwater Practices” in ThePractice of Watershed Protection).
14. Percent removals do not adequately reflect the effect of volume reductions. In some percent removal calculation methods, volume reductions are partially taken into account, but not in others. Even when load reductions are used, this approach misses the benefit of the reduced frequency of discharges.
15. Percent removal methods also sometimes miss the measurement of how much runoff is and is not treated. There are example studies in which the percent removal has been reported based on the influent and low-flow effluent (e.g., the flow stream that has received treatment) from a BMP; however, the majority of flow was bypassing the BMP due to clogging. BMP sizing relative to incoming runoff is important in performance metrics.

Photo 2. BMP performance data are needed for receiving water impact assessments.

Advertisement

For these reasons, among others, the project team does not present percent removal estimates with the BMP analysis it conducts. Instead, the team recommends using an approach that focuses on the following:

• How much the BMP reduces runoff volumes
• How much runoff is treated (versus bypassed)
• Whether the BMP can demonstrate a statistical difference in effluent quality compared to influent quality
• What distribution of effluent quality is achieved
• How well the BMP reduces peak runoff rates, especially for smaller, frequent storms (which helps reduce hydromodification)

For more information on applying these approaches to assessing BMP performance, visit www.bmpdatabase.org. The project Web site also provides statistical analysis of BMP performance relying on these alternative, recommended measures of performance.

Author's Bio: Jonathan Jones, P.E., is chief executive officer, with Wright Water Engineers Inc. in Denver, CO.

Author's Bio: Jane Clary, is a senior water resources scientist, with Wright Water Engineers Inc. in Denver, CO.

Author's Bio: Eric Strecker, P.E., principal, is with Geosyntec Consultants in Portland, OR, and Acton, MA.

Author's Bio: Marcus Quigley, P.E., associate, is with Geosyntec Consultants in Portland, OR, and Acton, MA.