Choices with Consequences #1: Outlier Treatment in Continuous Indicators

Key Takeaways

• Outliers, extremely high or low values of a continuous variable are common in survey data. They can be atypical but valid observations (e.g., a productive household reporting a large quantity of maize harvested) or the result of data collection or processing errors.
• Appropriately handling outliers can reduce their distortion of summary statistics or analytical results, and transparency around methods can support reproducibility.
• The outcome of an outlier treatment method depends on the type of measurement error that generated outlier observations, the overall sample size, the subgroups of interest, and the degree of variation across those subgroups.
• We developed a simple tool to explore the consequences of different methods, using household survey data from the LSMS-ISA.

The outlier problem in survey datasets

Outliers in survey datasets can represent valid observations when they reflect truly
atypical cases within the sample or population. However, many outliers arise from
errors. Three main types of errors can generate outliers:

1. 1. Response error: when respondents misstate their answer. For example, a household might report having harvested 1,000 grams of maize when they intended to report 1,000 kg.
   2. Recording error: when the enumerator incorrectly enters a response that still passes the data entry validation checks. For example, an enumeratorentering 10,000 kg of maize harvested while the respondent states 1,000 kg
   3. Derived values from plausible reporting: In some cases, the combination of otherwise plausible values can produce atypical outcomes. For example, a farmer reports harvesting 1,500 kg of maize on a 0.1-hectare plot, two individually plausible values that result in an atypically high estimated yield of 15,000 kg maize/ha – more than maize yields in most high-input agricultural systems and far above typical estimates for small-scale producers (SSPs).

If left unaddressed, outliers can meaningfully distort summary statistics, variable
distributions, and analytical results. In practice, treating outliers involves two sequential decisions: selecting a method to detect outliers and choosing how to handle those that are identified. Each of these methods has its own strengths and limitations. It is therefore important to understand the consequences of different outlier-handling approaches and to transparently document the chosen method when reporting statistics or conducting analyses.

Approaches to detecting outliers

There are several methods for detecting outliers, ranging from simple techniques that rely solely on the distribution of a given indicator to more complex model-based approaches that incorporate information from additional variables. Due to their simplicity, common use, and ease of implementation, in this post, we discuss three non-model-based approaches: i) percentiles; ii) median absolute deviation-based (MAD); and iii) transformation-based approaches. Table 1 describes each method and characterizes its strengths and limitations.

Table 1: Three approaches for detecting outliers. Tradeoffs exist in terms of what each method assumes about the underlying distribution (and therefore how the outliers are classified), threshold selection (typically set by convention), and level of sensitivity.

Method	Definition	Threshold Selection	Distribution Assumption	Sensitivity and outlier considerations
Percentile	Classify all values above/below a specified percentile (threshold or cutoff) as outliers	None	Based on estimated outlier occurrence rates	Affects a predictable number (proportion) of observations that depends on the percentile threshold and whether trimming is at one or both tails.
Median Absolute Deviation	Classifies values whose absolute deviation exceeds a specified multiple of the MAD as outliers. MAD is the median of the absolute differences between each observation and the sample median.	Symmetric and typically normal	Commonly uses a multiple of MAD (e.g., 3-3.5*MAD)	Sensitivity and outliers depend on the clustering of observations close to the median and normality of the distribution. Enforces symmetry on the resulting distribution.
Transformation	Applies a transformation function and then classifies values to select outliers from the transformed dataset based on a z-score threshold. Examples include simple log or piecewise functions like Yeo-Johnson.	Non-normal, usually lognormal	Rule of thumb based on standard deviation/z-score (typically +/- 3 SD)	Depends on how closely the transformed distribution matches a normal one. Better for asymmetrically distributed data, may flag fewer outliers than the other methods.

Approaches to handling outliers

Once outliers are identified, the analyst must decide whether to ignore them, delete them without replacement, or delete them and replace them with more plausible values. Dropping outliers from the dataset, a process termed trimming, involves an approach for detecting outliers (Table 1). Trimming without replacement reduces the sample size and could potentially bias the distribution if valid observations are removed. In small datasets, there is also a risk of reduced statistical power.

Alternatively, the analyst can replace the outliers with plausible values to maintain the original sample size (Table 2). Winsorization is a popular method that combines percentile-based detection of outliers (e.g. beyond the 99th percentile) with replacement at the threshold. It assumes that the “correct” value of the measurement being replaced is most likely near the tail. While this method reduces the influence of individual observations over the summary statistics, it can also lead to an increased concentration of observations at the threshold values and may have little practical effect if the threshold value is close to the outlier values. Median replacement is also an option that is often combined with the MAD outlier detection approach. Median replacement assumes that the proper value is closer to the center of the distribution. While it also reduces the overall variance of the data, it can potentially bias summary statistics such as the mean and standard deviation.

All three methods may fail to handle small (error-based) outliers in data with many valid zeros, such as when farmers lose their entire harvest.

Table 2: Comparison of replacement or trimming choices.

Method	Definition	Notes
Trimming without replacement	Removes outliers entirely	Removes extreme distortions Reduces sample size Risk of selection bias if true high/low values are removed Removes tail variation
Threshold Replacement	Replaces the outliers with the values of the thresholds; typically used with percentile-based outlier detection (winsorization)	Removes distortions while preserving the number of large or small values Maintains all observations Potential distortion due to an inaccurate number of large values
Median Replacement	Replaces extreme values with the median	Removes distortions and maintains all observations In some scenarios, replacement value is more likely to be close to the “correct value” than using the extremes. Final distribution is more likely to meet assumptions of normality Potential to bias observations toward the median, and under-represent the true range

How to choose?

To support analysts comparing alternative outlier-treatment methods, we developed an interactive web-based tool implemented in R Shiny using several LSMS-ISA datasets, the Ethiopia ESS, Malawi IHS, Nigeria GHS, Tanzania NPS, and Uganda NPS. The tool allows users to select a dataset, variable, outlier-treatment method, and thresholds, and examine how these choices affect both the variable’s distribution and key summary statistics. An example using the tool is discussed below.

Example: Comparing the effect of outlier treatment methods on maize yields

To illustrate how different outlier treatment methods are applied, we use maize yield in Nigeria’s GHS wave 5 as an example. Yield is measured as the ratio of quantity harvested to area planted. It is a challenging indicator from an outlier management perspective because otherwise reasonable-seeming measurements of the numerator and denominator can produce extreme values when combined.

The raw data contain 1,109 observations. Maize quantity harvested ranges from 0 to 10,000 kg and area planted from 0.05-9.29 hectares^[1], while yield ranges from 0.81 to 43,060 kg/ha. For quantity harvested, all methods reduce the top end of the distribution while leaving the bottom untouched due to the high prevalence of valid zeros (i.e. crop failure, roughly 4% of observations). Observations where the crop was not ready for harvest when the enumerators visited are removed.

Method for detecting outliers	Thresholds		Number of observations identified as outliers			Difference in mean yield estimate between the raw data and the outlier-corrected data
			Quantity Harvested (kg)	Area Planted (ha)	Yield (kg/ha)	Replacement	Trimming Only
Percentile	Lower first and upper 99th percentile		12	25	10	-2.65%	-5.44%
MAD	+/- 3.5*MAD		126	169	140	-17.9%	-12.2%
Yeo-Johnson Transformation	+/- 3 SD		4	0	3	-0.48%	-4.11%

The large differences between methods results from the substantial right skew of the data, i.e., there tended to be more very small observations than very large ones, and most observations were close to the mean. For example, in area cultivated, the minimum (0.05) was closer to the mean (0.5) than the maximum (9.6), but the range of log-transformed data is more symmetrical (-2.30 to 2.99). Thus, large observations were more “expected” in the log transformed distribution than they were in the original distribution. This difference further illustrates how researcher expectations about the distribution of observations can shape data processing decisions. An assumption of lognormality, reasonable in a situation where a researcher anticipates a few observations where landholdings are significantly larger than the mean, suggests that most of the observations are within the expected range, whereas assuming that the sample should be more normally distributed (which occurs when employing MAD) produces a much greater number of outliers.