In 2017, the state of California passed Assembly Bill 746 requiring all public schools in the state to test their water fixtures for lead by July 2019. A previous law had offered financial assistance for testing, but not all schools took advantage of it. The test results from the previous program and early returns on the mandatory program are available in a public database providing some coverage of the state. This newly available dataset offers up recorded data of lead levels at the downstream end of water supply systems all throughout the state. Lead Alert uses these test results as labels for a machine learning model, allowing us to predict where contamination may be entering the system from the conveyance system itself.

Previous studies have shown that characteristics related to housing and construction, demographics, and property values are indicators of lead contamination for individual buildings or properties. These studies were mostly focused on a single city - Flint, Michigan. Lead Alert tries to increase the footprint of prediction to a much larger area, starting with the state of California and potentially proceeding to the United States. To do this, Lead Alert uses data from the American Community Survey as feature data to predict risk of lead contamination at the census tract level, a standard geometry defined by the United States Census Bureau to roughly represent a neighborhood. Census tracts have large amounts of aggregated data readily available.

Lead Alert also attempts to use water quality testing data as features in prediction. This data includes upstream water quality tests for chemicals that are key elements in the lead contamination process. Key elements include lead, orthophosphates, and chlorides.

To work with the data, Lead Alert employed data leveling and ensemble learning techniques. Data leveling techniques like synthetic minority over-sampling technique (SMOTE) create synthetic instances of the minority class (in this case high lead test results) based on the features of actual instances of the minority class. This avoids overfitting. Boosting-based ensemble modeling methods tend to perform well on imbalanced datasets as they create many weak learners that in aggregate create a strong learner capable of making accurate predictions. XGBoost (Extreme Gradient Boosting) is an efficient implementation of a boosting method whereby the loss function of previous learners is used when building additional learners, increasing both speed and accuracy.

Refer to the Github repository for the complete collection of modeling code and artifacts. The technical report captures further detail on the effort.

Two sets of features were used for the modeling: the full feature set and a feature subset. The full feature set contained 757 features, which included the chemical threshold percentages but did not include the lead result field. The feature subset contained a total of 44 features, also including the chemical threshold percentages but no lead result data. The selected fields were observed as having higher performance when comparing ensemble models. These fields include data related to age of construction, population age, and the year residents moved into their current residence.

The resulting data set was randomly split into training and test data with the test set being a third of the total size. The training data was then split again into training and validation with validation being a third of the total training size. The validation split further reduces the training size, although the intention is to compare validation and test results for model stability.

To work with the data, Lead Alert employed both data leveling and ensemble learning techniques. Given the imbalanced nature of the data, preprocessing options were considered to help highlight the characteristics of the minority class that would otherwise be treated as noise. Data leveling techniques like synthetic minority over-sampling technique (SMOTE) create synthetic instances of the minority class (in this case high lead test results) based on the features of actual instances of the minority class. This avoids overfitting that would result from strictly oversampling. The relatively small size of the aggregated lead contamination data lent itself to oversampling of the minority class. Random oversampling and SMOTE were employed on the training data using the Imbalanced-Learn library, along with no preprocessing for comparison.

Boosting based ensemble modeling methods tend to perform well on imbalanced datasets as they create many weak learners that in aggregate create a strong learner capable of making accurate predictions. XGBoost (Extreme Gradient Boosting) is an efficient implementation of a boosting method whereby the loss function of previous learners is used when building additional learners, increasing both speed and accuracy.

The XGBoostClassifier was trained using a 10-fold cross validation scored by accuracy. Once the best parameters were calculated, a XGBoost classifier was trained using the training data on the best parameters. Ten-fold cross validation and early stopping using the error metric were employed to help prevent overfitting. The boosted round number from the optimized XGBoost classifier as well as the best parameters were captured for defining final model to be used in evaluation.

With the final model ready for evaluation, we predicted using the validation and test sets. The prediction outcome was a set of probabilities. We evaluated several probability thresholds but have chosen a 0.75 prediction probability cutoff for indicating lead contamination. A 0.50 cutoff is more common, but we wanted to remain conservative given the nature of the problem space. We wanted locations that we predicted as potentially at risk to be very likely at risk, not just more likely than not. The true positive and false positive rates may be used to calculate the optimal threshold for the training data, although this did not prove beneficial in testing.

If a future study were to occur using building level feature data and the school testing database as label data, we would recommend following many of our steps around data leveling and modeling. The data will be imbalanced, requiring data leveling. Also, if feature data is at the building level, spatial clustering methods should be evaluated. Not every building is a school, and schools will have different properties than other building types, but this is still the most complete data that can be used to train a model.