Assess the accuracy of a perviousness classification

The City of Louisville, Kentucky, classified aerial imagery to identify areas of impervious surfaces, which will help city officials determine the storm water management fees charged to residents. But how can the city evaluate the quality of that classification? In this tutorial, you'll perform a quantitative accuracy assessment on the classification by statistically comparing it to the original image. You'll generate accuracy assessment points throughout the image, ground truth them, compute a confusion matrix, and obtain a classification accuracy percentage. The city administrators of Louisville need a reliable impervious surface classification to support their storm water management efforts.

The original classification workflow can be found in the tutorial Calculate impervious surfaces from spectral imagery. However, this current tutorial can be completed independently from it.

This tutorial was last tested on May 23, 2023 using ArcGIS Pro 3.1. If you're using a different version of ArcGIS Pro, you may encounter different functionality and results.

Requirements

• ArcGIS Pro (see options for software access)
• ArcGIS Spatial Analyst extension or ArcGIS Image Analyst

Assess the accuracy of a perviousness classification

Perform a visual assessment

To get started, you'll open the project, explore the layers it contains, and informally assess the classification accuracy by inspecting it visually. First, you'll download and open the project containing the perviousness classification layer and the reference imagery.

Note:

While you could use your own results from the tutorial Calculate impervious surfaces from spectral imagery, using freshly downloaded data will ensure that what you see on the screen matches the steps described in this tutorial.

1. Download the Imperviousness_Classified .zip file that contains your project and its data.
2. Locate the downloaded file on your computer.

   Note:

   Depending on your web browser, you may have been prompted to choose the file's location before you began the download. Most browsers download to your computer's Downloads folder by default.

3. Right-click the file, Imperviousness_Classified.zip, and extract it to a location you can easily find, such as your Documents folder.
4. Start ArcGIS Pro. If prompted, sign in using your licensed ArcGIS organizational account.

   Note:

   If you don't have access to ArcGIS Pro or an ArcGIS organizational account, see options for software access.

5. Under Open, at the bottom of the pane, click Open another project.

   

6. In the Open Project window, browse to the extracted folder, Imperviousness_Classified. Click Imperviousness_Classified.aprx to select it, and click OK.

   

   The project opens.

   

   Next, you'll explore the layers that the project contains. The top layer, Louisville_Impervious, is turned on. It is the result of the previous classification process, and shows impervious surfaces in gray and pervious surfaces in green. The basemap, World Topographic Map, is also turned on and provides the context for the project's layers.

7. In the Contents pane, uncheck the Louisville_Impervious check box to turn the layer off. Check the Louisville_Neighborhood.tif check box to turn the layer on.

   

   Louisville_Neighborhood.tif is a 6-inch resolution, 4-band aerial photograph of the area, shown in the Natural Colors band combination (red, green, and blue). This means that it looks similar to what the human eye would see. You'll use it as a reference when performing the accuracy assessment.

8. In the Contents pane, turn off Louisville_Neighborhood.tif and turn on Louisville_Neighborhood_Color_Infrared.

   This is the same imagery, displayed in the Color Infrared band combination (infrared, red, and green). It shows the vegetation highlighted in red. You'll also use it as a reference when performing the accuracy assessment. Next, you'll assess the classification accuracy informally by comparing the layers visually.

9. In the Contents pane, turn off Louisville_Neighborhood_Color_Infrared, and turn on both Louisville_Impervious and Louisville_Neighborhood.tif. Click Louisville_Impervious to select it.
10. On the ribbon, click the Raster Layer tab, and in the Compare group, click Swipe.

    

11. On the map, drag the pointer from top to bottom, to reveal the Louisville_Neighborhood.tif imagery under the Louisville_Impervious classification layer. Examine and compare the two layers.

    

    How accurate does the classification seem?

12. Optionally, you can also swipe between Louisville_Impervious and Louisville_Neighborhood_Color_Infrared to compare the classification to the color infrared imagery.
13. On the ribbon, click the Map tab and in the Navigate group, click Explore to deactivate the Swipe tool.

    

14. On the Quick Access Toolbar, click the Save button to save your project.

    Tip:

    You can also press Ctrl+S to save the project.

Create accuracy assessment points

Informally comparing the classification layer to the original imagery is somewhat useful, but it doesn't provide a formal measurement of the classification's accuracy. With storm water bills being determined from your analysis, you need a more rigorous evaluation process. To perform a formal assessment, you'll create randomly generated accuracy assessment points throughout the image. You'll then compare the classification value (pervious or impervious) at the location of each point to the actual land cover type shown in the original imagery.

Note:

The process of having a person verify the actual land cover visually at a given location is called ground truthing. The most rigorous accuracy assessment would involve someone going on site to verify the land cover in person, but in many cases, traveling to the actual location would be infeasible or too costly. So, looking at imagery is usually considered the next best choice for the ground truthing process.

1. On the ribbon, click the Analysis tab and in the Geoprocessing group, click Tools.

   

   The Geoprocessing pane appears.

2. In the Geoprocessing pane, search for Create accuracy assessment points, and press Enter.

   

3. In the results list, click Create Accuracy Assessment Points (for either Image Analyst Tools or Spatial Analyst Tools) to open the tool.

   This tool generates random points throughout an image and gives the points an attribute based on the classified value of the image at the point's location. The points will also have a field for the ground truth of the original image, which you'll manually fill in for each point.

4. In the Create Accuracy Assessment Points tool, enter the following parameters:
   • For Input Raster or Feature Class Data, choose the Louisville_Impervious layer.
   • For Output Accuracy Assessment Points, click the Browse button. In the window that appears, browse to Project, Databases, and double-click Neighborhood_Data.gdb. For Name, type My_Accuracy_Points and click Save.
   • For Target Field, confirm that Classified is selected.
   • For Number of Random Points, type 100.
   • For Sampling Strategy, choose Equalized stratified random.

   

   The Target Field parameter determines whether the attribute table of the points describes the classification value or the ground truth value. Your input image is the classified raster, so the points should contain the classified values.

   The Number of Random Points parameter determines how many points are created. For a small image with only two classes, a relatively small number of points is acceptable.

   The Sampling Strategy parameter determines how points are randomly distributed across the image. The points can be distributed proportionally to the area of each class (Stratified random), equally between each class (Equalized stratified random), or absolutely randomly (Random). Because your primary interest is in the accuracy of impervious surfaces, which has the smaller area of the two classes, you'll equally distribute the points between each class to better represent impervious surfaces in the assessment.

   Note:

   The chosen parameters mean that there will be 50 points selected on the impervious surfaces, and 50 points on the pervious surfaces.

5. Click Run.

   A new layer with 100 accuracy points is added to the map.

   

   You'll now look at the attributes for those points.

6. In the Contents pane, right-click the My_Accuracy_Points layer and choose Attribute Table.

   

   The attribute table appears.

   

   The attribute table contains information for each point location. Other than the usual ObjectID and Shape fields, the points have two attributes: Classified and GrndTruth (or Ground Truth). The Classified field has values that are either 20 or 40. These numbers represent the classes determined by the classification process, as they appear in the Louisville_Impervious layer: 20 is impervious; 40 is pervious. For the GrndTruth field, however, every value is -1 by default, signaling that the value is still unknown, and the point needs to be ground truthed. You'll inspect the imagery for each point, and edit the GrndTruth attributes to either 20 or 40, depending on the type of land cover you find.

7. In the Contents pane, uncheck all layers except My_Accuracy_Points and Louisville_Neighborhood.tif.

   Note:

   Optionally, you can change the color of the points to see them better. In the Contents pane, if necessary, right-click the symbol under My_Accuracy_Points and choose a color of your liking.

   Next, you'll hide the Classified column to make sure that your judgment during the ground truthing process is not influenced by the existing classification values.

8. In the attribute table, right-click the Classified column header and choose Hide Field.

   

9. In the attribute table, click row header 1 to select the feature. Right-click the row header and choose Zoom To.

   

   The map zooms to the selected point. (Your point will be in a different location than the point in the example image.)

   Note:

   If necessary, you can drag the top of the attribute table pane to resize it so you can see both the map area and the attribute table.

   

   In this example, the point appears to be on either grass or bare earth. Either way, the surface is pervious. You would change the GrndTruth attribute for this point to 40 for pervious. If your first point appears to be on an impervious surface such as roads or roofs, you'll change the GrndTruth attribute to 20 for impervious.

   Note:

   Depending on your map extent and the location of the point, you may not have zoomed close enough to the point to determine what kind of ground cover it is on. Feel free to zoom closer to better determine the point's ground truth.

   As a reminder, in this imagery, pervious surfaces include vegetation, trees, bare earth, and water. Impervious surfaces include roads, driveways, and building roofs.

10. In the attribute table, in the GrndTruth column, double-click the value for the selected feature to edit it. Replace the default value with either 40 or 20, depending on your findings, and press Enter.

    

11. Select the next point in the attribute table. Right-click the point and choose Pan To.

    The map pans to the corresponding point, keeping the same zooming level.

12. Depending on the location of the point, change the GrndTruth value to either 20 or 40.

    Note:

    It may be difficult to tell the ground truth for some of the points due to ambiguous features on the map. If needed, you can turn on the Louisville_Neighborhood_Color_Infrared layer temporarily, to see whether it gives you a clearer view.

13. Repeat the process for the first 10 points.
14. On the ribbon, click the Edit tab, in the Manage Edits group, click Save to save all the edits you made in the attribute table. When prompted to confirm, click Yes.

    

    You'll now pause your ground truthing efforts to unhide the Classified column and compare the values in the Classified and GrndTruth columns.
15. Click the options button and choose Show All Fields.

    

    After the Classified column reappears, examine the values in both columns and compare them. Do they all match so far?

    

    If desired, you can continue entering ground truth values for all 100 accuracy points in the My_Accuracy_Points layer. However, to save time in this tutorial, you instead can use an accuracy assessment point feature class in which the ground truth values were populated for you. If you choose to use the ready-made accuracy point layer, you'll now add it to the map.

    Note:

    If you chose to ground truth the 100 points yourself, hide the Classified attribute again, and complete the process now. On the Edit tab, click Save to save all the edits. Close the attribute table, and skip to step 18.

16. On the ribbon, click the View tab and in the Windows group, click Catalog Pane.

    

17. In the Catalog pane, under Project, expand Databases, and expand Neighborhood_Data.gdb. Right-click Accuracy_Points and choose Add To Current Map.

    

    The new ready-made Accuracy_Points layer appears in the Contents pane and on the map. You don't need the first layer, My_Accuracy_Points, any longer, so you'll remove it.

18. In the Contents pane, right click My_Accuracy_Points, and choose Remove and in the Removing a layer pop-up, click Discard.
19. In the Contents pane, right-click the Louisville_Neighborhood.tif layer and choose Zoom To Layer.

    The map returns to the full extent of the imagery.

20. Save the project.

Compute a confusion matrix

After creating accuracy assessment points and populating their attributes with ground truth data, you'll now use the points to create a confusion matrix. A confusion matrix is a table that compares the Classified and GrndTruth attributes of accuracy assessment points, and determines the percentage of accuracy between them. If the areas that were classified as impervious actually represent impervious areas in the original imagery, the confusion matrix will have a high percentage and indicate that the classification has a high accuracy.

1. If necessary, on the ribbon, on the Analysis tab, in the Geoprocessing group, click Tools to reopen the Geoprocessing pane.
2. If necessary, in the Geoprocessing pane, click the Back button.

   

3. In the Geoprocessing pane, search for and open the Compute Confusion Matrix tool (for either Image Analyst Tools or Spatial Analyst Tools).

   

   The tool has only two parameters: an input and an output.

4. In the Compute Confusion Matrix tool, enter the following:
   • For Input Accuracy Assessment Points, choose Accuracy_Points.
   • For Output Confusion Matrix, click Browse. For Name, type Confusion_Matrix and click Save.

   Note:

   If you chose to enter the 100 points yourself, for Input Accuracy Assessment Points, choose My_Accuracy_Points.

   

5. Click Run.

   The tool runs and the confusion matrix is added to the Contents pane. Because the confusion matrix is a table with no spatial data, it does not appear on the map.

6. In the Contents pane, under Standalone Tables, right-click Confusion_Matrix and choose Open.

   

   The Confusion_Matrix table appears.

   

   Note:

   If you entered the ground truth values yourself in the My_Accuracy_Points layer, the confusion matrix may show slightly different results.

   The values in the ClassValue column serve as row headers in the table. C_20 and C_40 correspond to the two classes in the classified raster: 20 for impervious surfaces and 40 for pervious surfaces. The C_20 and C_40 columns represent points with a ground truth of 20 or 40, while the C_20 and C_40 rows represent points that were classified as 20 or 40. For instance, when using the example points, 47 points that had a ground truth of 20 were also classified as 20, while one point with a ground truth of 20 was misclassified as 40. Out of a total of 100 points, four were misclassified (three were misclassified as impervious, and one was misclassified as pervious).

   U_Accuracy stands for user's accuracy. It represents the fraction of pixels classified correctly per total classifications. P_Accuracy stands for producer's accuracy and represents the fraction of pixels classified correctly per total ground truths. For instance, 50 pixels were classified as impervious, of which 47 were classified correctly, leading to a user's accuracy of 0.94 (or 94 percent). Meanwhile, 48 pixels had a ground truth of impervious, of which 47 were classified correctly, leading to a producer's accuracy of approximately 0.98 (or 98 percent).

   The final attribute is Kappa. Based on the total user's and producer's accuracies, it gives an overall assessment of the classification's accuracy. In the example above, the Kappa is 0.92, or 92 percent. While not perfect, an overall accuracy of 92 percent is fairly reliable.

   Note:

   If the Kappa value were below 85 to 90 percent, your classification would probably not be accurate enough and would need to be revisited to be improved. There are two parts of the workflow that may contribute to classification error. The first is segmentation. If your segmentation parameters generalize the original image too heavily or not enough, features may be misclassified. You could try tweaking the segmentation parameters for a better segmentation. Alternatively, the majority of errors may have been caused by your training samples. Having too few training samples, or training samples that cover too wide a variety of spectral signatures, may also lead to classification error. Adding either more samples or more classes may increase the accuracy.

7. Save the project.

You can now report to the city administrators that your classification of impervious surfaces is 92 percent accurate. This can be considered reliable enough to start using the data to compute storm water fees.

You can find more tutorials in the tutorial gallery.

Acknowledgements

Imagery from U.S. National Agriculture Imagery Program (NAIP).