Create training samples
Inventorying and assessing the health of each palm tree on the Kolovai, Tonga, plantation would take a lot of time and a large workforce. To simplify the process, you'll use a deep learning model in ArcGIS Pro to identify trees, then calculate their health based on a measure of vegetation greenness. The first step is to find imagery that shows Kolovai, Tonga, and has a fine enough spatial and spectral resolution to identify trees. Once you have the imagery, you'll create training samples and convert them to a format that can be used by a deep learning model. For the model to recognize what it's tasked with finding, you need to define images of palm trees so that it can identify similar pixels.
Download the data
Accurate and high-resolution imagery is essential when extracting features. The model will only be able to identify the palm trees if the pixel size is small enough to distinguish palm canopies. Additionally, to calculate tree health, you'll need an image with spectral bands that will enable you to generate a vegetation health index. You'll find and download the imagery for this study from OpenAerialMap, an open-source repository of high-resolution, multispectral imagery.
- Go to the OpenAerialMap website.
- Click Start Exploring.
In the interactive map view, you can zoom, pan, and search for imagery available anywhere on the planet. The map is broken up into grids. When you point to a grid box, a number appears. This number indicates the number of available images for that box.
- In the search box, type Kolovai. In the list of results, click Kolovai.
This is a town on the main island of Tongatapu with a coconut plantation.
- If necessary, zoom out until you see the label for Kolovai on the map. Click the grid box directly over Kolovai and click Kolovai UAV4R Subset (OSM-Fit) by Cristiano Giovando.
- Click the download button to download the raw .tif file. Save the image to a location of your choice.
Because of the file size, download may take a few minutes.
Take a look at the data
To begin the classification process, you'll create an ArcGIS Pro project with the imagery you downloaded and save a few bookmarks to use while creating training samples.
- Start ArcGIS Pro. If prompted, sign in using your licensed ArcGIS account.
If you don't have ArcGIS Pro or an ArcGIS account, you can sign up for an ArcGIS free trial.
- Under New, click Map.
The Map template creates a project with a 2D map.
- In the Create a New Project window, name the project CoconutHealth. Save the project to the location of your choice and click OK.
The project opens and displays the Topographic basemap.
- On the Map tab, in the Layer group, click Add Data.
The Add Data window appears.
- In the Add Data window, under Computer, browse to the Kolovai image you downloaded from OpenAerialMap. Select the .tif file and click OK.
The Kolovai image is added to your map. The layer is listed in the Contents pane by its unique identifier, which isn't meaningful. It's best practice to rename the layer to something you understand.
- In the Contents pane, click the imagery layer two times and type Kolovai Palms. Press Enter.
- Pan and zoom around the map to get an idea of what the palm farm looks like.
A large number of coconut palm trees are in this image. Counting them individually, on the field or by visually inspecting the image, would take days. To enable a deep learning model to do this work for you, you'll create a small sample of palm trees to use for training your model. First, you'll create a custom map display, so you can quickly zoom in on different areas of the image.
- At the bottom of the map window, click the map scale arrow and choose Customize.
The Scale Properties window appears.
- In the Scale Properties window, make sure the Standard Scales tab is selected. In the Scale box, type 1:500.
- Click Add and click OK.
The custom scale option has been added to the list of map scales in your project. You'll use this scale each time you create a bookmark.
- On the ribbon, on the Map tab, in the Inquiry group, click Locate.
The Locate pane appears.
- Into the Locate search box, paste the following coordinates and press Enter: 175.3458501°W 21.0901350°S.
The letter A appears on your map to mark the location of the coordinates. You'll bookmark this location using the custom scale so you can refer to it in the next section.
- Click the map scale list and choose 1:500.
You are looking at a zoomed-in display of the coordinate location in your map.
- On the ribbon, on the Map tab, in the Navigate group, click Bookmarks. In the menu, click New Bookmark.
The Create Bookmark window appears.
- In the Create Bookmark window, type Northwest palms and click OK.
- Create bookmarks for the following coordinates at a scale of 1:500:
Coordinates Bookmark name
Central east palms
Central west palms
- Close the Locate pane and save the project.
Create training schema
Creating good training samples is essential when training a deep learning model, or any image classification model. It is also often the most time-consuming step in the process. To provide your deep learning model with the information it needs to extract all the palm trees in the image, you'll create features for a number of palm trees to teach the model what the size, shape, and spectral signature of coconut palms may be. These training samples are created and managed through the Label Objects for Deep Learning tool.
- On the ribbon, click the Imagery tab.
ArcGIS Pro works on a contextual basis, so certain tools and tabs will only be available to you if the associated data is selected from the Contents pane. To activate the imagery analysis tools, a raster layer must be selected.
- In the Contents pane, make sure Kolovai Palms is selected.
Tools in the Image Classification, Mensuration, and Tools groups are now available to you. A new contextual tab Raster Layer with Appearance and Data tabs are activated.
- In the Image Classification group, click Classification Tools and choose Label Objects for Deep Learning.
The Image Classification pane appears with a blank new schema. You'll create a schema with only one class because you're only interested in extracting coconut palm trees from the imagery.
- Right-click New Schema and choose Edit Properties. For Name, type Coconut Palms. For Description, add a short explanation and click Save.
The schema is renamed in the Image Classification pane. You can now add samples to it.
- Right-click Coconut Palms and choose Add New Class.
- In the Add New Class pane, set the following parameters:
- For Name, type Palm.
- For Value, type 1.
- For Color, choose Mars Red.
- Click OK.
The Palm class is added to the Coconut Palms schema in the Image Classification pane. You'll create features with the Palm class to train the deep learning model in each bookmark you created.
Create training samples
To make sure you're capturing a representative sample of trees in the area, you'll digitize features throughout the image. These features are read into the deep learning model in a specific format called image chips. Image chips are small blocks of imagery cut from the source image. Once you've created a sufficient number of features in the Image Classification pane, you'll export them as image chips with metadata.
- On the ribbon, click the Map tab. In the Navigate group, click the Bookmarks drop-down button and choose the Northwest palms bookmark.
- In the Image Classification pane, select the Palm class and click the Circle tool.
You'll use this tool to draw circles around each palm tree in your current display. Circles are drawn from the center of the feature outward, measuring the radius of the feature.
- On the map, click the center of a palm tree and draw a circle around a single tree.
A new palm record is added in the Labeled Objects group of the Image Classification pane. When training a deep learning model, each image chip must have all the palm trees within it labeled as a palm. The image chips will be much smaller than your current map display, but you'll create a palm record for every tree you can to ensure there are many image chips with all the palm trees marked.
- Draw circles around each tree in the map display.
When you're finished, you'll have approximately 100 samples recorded in the Training Samples Manager pane.
- Create training samples for every palm tree on each bookmark.
The more samples to train the model with, the better the model performs classification.
Digitizing training samples can be a time-consuming process, but it pays off to have a large number of samples. The more samples you provide the model with as training data, the more accurate results it returns.
- When you're done creating samples, in the Image Classification pane, click Save.
- In the Save current training samples window, under Project, click Folders and double-click the default project folder, CoconutHealth.
- Name the feature class PalmTraining and click Save.
The last step before training the model is exporting your training samples to the correct format, as image chips.
- In the Image Classification pane, click the Export Training Data tab.
The list of training samples is replaced with the Export Training Data parameters.
- On the Export Training Data tab, enter the following parameters:
- For Output Folder, browse to the CoconutHealth project folder and create a new folder named ImageChips.
- For Image Format, choose JPEG format.
- For Tile Size X and Tile Size Y, type 448.
- For Meta Data Format, choose PASCAL Visual Object Classes.
- Click Run.
The tool runs. It may take several minutes to finish.
- In the Catalog pane, expand Folders. Right-click CoconutHealth and choose Refresh.
The folder refreshes to show the ImageChips folder.
- Expand ImageChips.
The folder is now populated with image chip samples and metadata.
- Save the project.
In this lesson, you downloaded and added open-source imagery to a project, created training samples using the Training Samples Manager pane, and exported them to a format compatible with a deep learning model for training. Next, you'll identify all the trees on the plantation.
Detect palm trees with a deep learning model
Clone the default conda environment
First, you created training samples of coconut palm trees and exported them as image chips. These training samples can be used to train a model using the Detect Objects using Deep Learning tool in ArcGIS Pro 2.5, which relies on deep learning frameworks such as TensorFlow, Keras, or CNTK. To install these deep learning libraries, you'll clone the default Python environment using the Python Command Prompt. You can learn more about the ArcGIS Pro Deep Learning toolset and the process of installing the deep learning frameworks in the documentation.
- If necessary, save your project and close ArcGIS Pro.
- On your desktop, search for and run the Python Command Prompt as an administrator.
The Python Command Prompt was downloaded when you installed ArcGIS Pro, so it automatically runs the propy.bat initialization file. This file, which runs in place of python.exe, recognizes your application's active conda environment and allows you to run standalone scripts using that environment.
For this project, you want to create a new environment named deeplearning. The default environment in ArcGIS Pro is read only, so you'll clone it to make changes.
- Run the following command to create a new conda environment by cloning the default ArcGIS Pro environment:
conda create -n deeplearning –-clone arcgispro-py3
The cloning process may take a few minutes. Cloned environments are stored in the envs folder at %LOCALAPPDATA%\Esri\conda\envs\. The system variable %LOCALAPPDATA% is a substitute for the C:\Users\YourUserFolderName\AppData\Local\.
- After the new environment is created, run the following commands to change your directory to that folder:
cd C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning
If your ArcGIS Pro installation is not located in the Program Files folder, you must use your installation's path instead. There are several default folders that python environments can be cloned to. These include \AppData\Local\ESRI\conda\envs, and \ArcGIS\Pro\bin\Python\envs, and may vary depending on the previous active environment.
- Run the following command to activate the new environment and update the file path to match where you stored the cloned environment:
This may take a few minutes. When the activation process is finished, the active environment name appears in parentheses before the folder location. Activating the new environment specifies that all changes you make occur only in the selected environment. When doing multiple projects that need different packages or different versions of packages, this is important. Now, you'll install the packages that the arcgis.learn module needs to run.
- Run the following commands one at a time to install the deep learning package dependencies:
conda install tensorflow-gpu=1.14.0 conda install keras-gpu=2.2.4 conda install scikit-image=0.15.0 conda install Pillow=6.1.0 conda install fastai=1.0.54 conda install pytorch=1.1.0 conda install libtiff=4.0.10 --no-deps
Some packages will require you to confirm installation. When prompted, type y and press Enter.
- Use the command proswap to set the ArcGIS Pro Project Environment to your deeplearning environment.
You now have a conda environment set up with all the libraries needed for the deep learning tools to run.
- Close the command prompt, then reopen your CoconutHealth project.
It is necessary to refresh the project to allow the new environment settings to update. Before starting the process of training a deep learning model, you'll use the Python Package Manager to confirm that the deeplearning environment is active in ArcGIS Pro.
- On the ribbon, click the Project tab and choose Python.
The Python Package Manager opens. Unless you've previously used different environments within ArcGIS Pro, the default environment, arcgispro-py3, is active. Environments created and activated in the command prompt are only persisted in that instance of the prompt unless you also set them as active in ArcGIS Pro.
- For Project Environment, confirm that the deeplearning environment is active.
Now, you'll use the Train Deep Learning Model tool to create an Esri model definition (.emd) file that is formatted to be read by ArcGIS geoprocessing tools like Detect Objects Using Deep Learning.
Train a deep learning model
The Train Deep Learning Model geoprocessing tool uses the image chips you labeled to determine what combinations of pixels in a given image represent palm trees. The training process produces an Esri model definition (.emd) file that can be used by other deep learning tools within ArcGIS. You'll populate a .emd file, then use the Detect Objects Using Deep Learning tool to identify palm trees in the image.
- On the ribbon, click the the Analysis tab. In the Geoprocessing group, click Tools.
- In the Geoprocessing pane, search for and open the Train Deep Learning Model tool.
- For Input Training Data, browse to the ImageChips folder you saved in the CoconutHealth project folder.
The ImageChips folder contains two folders, two text files, a .json and an .emd file that were created from the Export Training Data for Deep Learning tool. The esri_model_definition.emd file is a template that will be filled in by the data scientist who trained the model, with information such as the deep learning framework, the file path to the trained model, class names, model type, and image specifications of the image used for training. The .emd file is the bridge between the trained model and ArcGIS Pro.
- For Output Model, create a folder in the CoconutHealth project folder called PalmDetection_25_SSD.
This naming convention contains information on the parameters you'll use to train the model. 25 is the number of epochs you'll use, and SSD stands for Single Shot Detector, the model type you'll use.
- For Max Epochs, type 25.
An epoch is a full cycle through the training dataset. During each epoch, the training dataset you stored in the ImageChips folder will be passed forward and backward through the neural network one time. The default is 20.
- Expand Model Parameters and make sure Model Type is set to Single Shot Detector.
The Model Type will determine the deep learning algorithm and neural network that you will use to train your model. In this case, you're using the Single Shot Detector method because it's optimized for object detection.
- Accept the rest of the parameters.
Model arguments, the parameter values used to train the model, vary based on the model type you choose, and can be customized. For more information about choosing model arguments, see the Train Deep Learning Model documentation.
Depending on the processing power of your computer, this tool can take some time to run. To proceed with this lesson immediately, download the pre-trained PalmDetection_25_SSD file and continue with the next section.
- Click Run.
Palm tree detection
The bulk of the work in extracting features from imagery is preparing the data, creating training samples, and training the model. Now that these steps have been completed, you'll use a trained model to detect palm trees throughout your imagery. Object detection is a process that typically requires multiple tests to achieve the best results. There are several parameters that you can alter in order to allow your model to perform best. To test these parameters quickly, you'll try detecting trees in a small section of the image. Once you're satisfied with the results, you'll extend the detection tools to the full image.
- In the Geoprocessing pane, search for and open the Detect Objects Using Deep Learning tool.
This tool calls a third-party deep learning Python API and uses the specified Python raster function to process the image.
- For the Detect Objects Using Deep Learning tool, enter the following parameters:
- For Input raster, choose Kolovai Palms.
- For Output Detected Objects, type CoconutTrees.
- For Model Definition, navigate to PalmDetection_25_SSD.emd (downloaded with the lesson data located in C:\DeepLearning\Data) or the .emd file that was created with the Train Deep Learning Model tool.
- Padding: 0
- Threshold: 0.5
- nms_overlap: 0.6
- For Batch Size, type 1.
- Check the box for Non Maximum Suppression.
- For Max Overlap Ratio, type 0.6.
Additional arguments will appear in the tool window because of the information in the trained model. The data scientist that supplied you with the model should provide suggestions for each argument. More information about the arguments is provided if you want to experiment with different values. Otherwise, keep the default settings.
The threshold argument is the confidence threshold—how much confidence is acceptable to label an object a palm tree? This number can be tweaked to achieve desired accuracy.
When performing convolution of imagery in convolutional neural network modeling, you are essentially shrinking the data, and the pixels at the edge of the image are used much less during the analysis, compared to inner pixels. By default, the padding parameter is 0, but a padding parameter of 1 means an additional boundary of pixels is added to the outside edges of the image, all with a value of 0. This reduces the loss of information from the valid edge pixels and shrinking. You can change the parameter to 1 or 2 to see the effects.
Max Overlap Ratio (also exposed in the Python arguments as nms_overlap) controls how much each feature is allowed to intersect. The parameter appears twice because of arguments specified in the .emd file, but isn't always added. A lower number for this argument would specify that the objects could not overlap to be considered individual features. The batch_size parameter defines the number of samples that will be used to train the network in each iteration of training. For example, if you have 1,000 training samples (image chips) and a batch size of 100, the first 100 training samples will train the neural network. On the next iteration, the next 100 samples will be used, and so on. Depending on the memory your machine has available, you can increase this parameter, though training should only be done in perfect square batches. For example, you could use a batch of 4, 9, 16, and on.
The tool could take some time to run for the full image, so if you want to experiment with other parameters, you'll change the processing extent to a smaller area.
- Zoom in to a scale of 1:500 somewhere in the image.
- Click the Environments tab on the Detect Objects Using Deep Learning tool. Change the Extent to Current Display Extent.
- Click Run.
- Try changing the parameters to see if your model improves. Once you're satisfied with the results on a smaller scale, change the processing extent back to Default to process the entire image.
- Click Run.
The tool will take several minutes to run, depending on your hardware and whether you are running on CPU, GPU, or RAM.
- Save the project.
The deep learning tools in ArcGIS Pro depend on a trained model from a data scientist and the inference functions that come with the Python package for third-party deep learning modeling software. In the next lesson, you'll use raster functions to obtain an estimate of vegetation health for each tree in your study area.
Estimate vegetation health
In the previous lesson, you used a deep learning model to extract coconut palm trees from an image. In this lesson, you'll use the same image to estimate vegetation health by calculating a vegetation health index.
To assess vegetation health, you'll calculate the Visible Atmospherically Resistant Index (VARI), which was developed as an indirect measure of leaf area index (LAI) and vegetation fraction (VF) using only reflectance values from the visible wavelength:
(Rg - Rr) / (Rg + Rr - R(Rg - Rb))
where Rr, Rg, and Rb are reflectance values for the red, green, and blue bands, respectively (Gitelson et al. 2002).
Typically, you would use reflectance values in both the visible and the near infrared (NIR) wavelength bands to estimate vegetation health, as with the normalized difference vegetation index (NDVI). The Kolovai Palms raster data you downloaded from OpenAerialMap is a multiband image with three bands, all in the visible electromagnetic spectrum, so you'll use the VARI instead.
The VARI measurement requires the input of the three bands within the Kolovai Palms raster. To calculate VARI, you'll use the Band Arithmetic raster function. Raster functions are quicker than geoprocessing tools because they don't create a new raster dataset. Instead, they perform real-time analysis on pixels as you pan and zoom.
- If necessary, open your CoconutHealth project in ArcGIS Pro.
- On the ribbon, click the Imagery tab. In the Analysis group, click Raster Functions.
- In the Raster Functions pane, search for and select the Band Arithmetic raster function.
- In the Band Arithmetic Properties function, set the following parameters and click Create new layer:
- For Raster, choose the Kolovai Palms raster layer.
- For Method, choose VARI. The function requires you to provide the band index that corresponds to the input bands for the formula. The Input underneath the Band Indexes parameter shows Red Green Blue, so you'll provide the band index numbers that correspond with Red, Green, and Blue bands, in that order. Make sure to put a single space in between each band.
- For Band Indexes, type 1 2 3.
The VARI layer is added to the Contents pane as Band Arithmetic_Kolovai Palms. By zooming and panning around the area, you can see features such as the coastline, roads, buildings, and fields.
- In the Contents pane, make sure the Band Arithmetic_Kolovai Palms layer is selected.
- On the ribbon, click the Appearance contextual tab.
- In the Rendering group, select the Stretch Type drop-down menu and choose Standard Deviation.
- In the Contents pane, click Band Arithmetic_Kolovai Palms two times and rename it VARI.
Extract VARI to Coconut Palms
Having a raster layer showing VARI is helpful, but not necessarily actionable. To figure out which trees need attention, you want to know the average VARI for each individual tree. To find the VARI value for each tree, you'll extract the underlying average VARI and symbolize them to show which trees are healthy and which need maintenance.
First, you'll convert the polygon features into circles to represent the palm trees.
- In the Geoprocessing pane, search for and open the Feature To Point tool. Enter the following parameters and click Run:
- For Input Features, select the CoconutTrees layer.
- For Output Feature Class, type CoconutTrees_Points.
You have a point feature class in the centroid of each detected polygon. If you zoom in to various locations and use the Measure tool, you'll see that the palm trees have an average radius of roughly 3 meters.
- In the Geoprocessing pane, search for and open the Buffer tool.
- Specify the following parameters and click Run:
- For Input Features, choose CoconutTrees_Points.
- For Output Feature Class, choose PalmTreesBuffer.
- For Distance, choose 3 Meters (make sure the value is set to Linear Unit).
You have a polygon feature class depicting the location and general shape of each Coconut Palm treetop.
- In the Contents pane, turn off the VARI, CoconutTrees, and CoconutTrees_Points layers.
Next, you'll extract the average VARI value for each polygon.
- In the Geoprocessing pane, search for and open the Zonal Statistics as Table tool.
- In the Zonal Statistics as Table tool, enter the following parameters and click Run:
- For Input raster or feature zone data, choose PalmTreesBuffer.
- For Zone field, choose ORIG_FID.
- For Input value raster, choose VARI.
- For Output table, type MeanVARI_per_Palm.
- Check Ignore NoData in calculations.
- For Statistics type, choose Mean.
Setting the Zone field to ORIG_FID will ensure that you get statistics for each tree separately. This attribute is the unique ID from the original CoconutTrees layer.
The output table is added to the bottom of your Contents pane. If you open it, you'll see the original FID value and a column called MEAN containing the average VARI value. You'll join this table to the PalmTreesBuffer layer to get a feature class with both the Confidence score and the average VARI for each detected palm tree.
- In the Geoprocessing pane, search for and open the Join Field tool.
- In the Join Field tool, enter the following parameters and click Run:
- For Input Table, choose PalmTreesBuffer.
- For Input Join Field, choose ORIG_FID.
- For Join Table, choose MeanVARI_per_Palm.
- For Output Join Field, choose ORIG_FID.
- For Join Fields, choose MEAN.
The PalmTreesBuffer layer now has a field called MEAN joined to it. You'll rename it and symbolize it for a better understanding of the data.
- In the Contents pane, click PalmTreesBuffer two times and rename it PalmTreesVARI.
- On the ribbon, on the Appearance tab, in the Drawing group, click Symbology.
- For Primary symbology, choose Graduated Colors.
- For Field, choose MEAN.
- If necessary, for Method, choose Natural Breaks (Jenks) and set Classes to 4.
- For Color scheme, click the drop-down menu and check Show all and Show names. Scroll and select the Red-Yellow-Green (4 Classes) scheme.
- Under Classes, click each label and rename the classes from top to bottom as follows: Needs Inspection, Declining Health, Moderate, and Healthy.
You now have a map with a feature class showing the location, health, and confidence of the model for each palm tree in the image.
- Save the project.
Extra credit: Assigning field tasks and monitoring project progress
One of the biggest benefits of using ArcGIS Pro for feature extraction and imagery analysis is that it can be integrated with the entire ArcGIS platform. In the last lesson, you used the deep learning tools in ArcGIS Pro to identify coconut palm trees from imagery. The palm trees can be stored as features in a feature class that's amenable for use in a GIS. To extend the workflow, you can publish your results to the cloud, configure a web application template for quality assurance, assign tree inspection tasks to workers in the field, and monitor the progress of the project using a dashboard.
Publish to ArcGIS Online
To use configurable apps to work with your data, you need to publish the palm trees as a feature service in ArcGIS Online or ArcGIS Enterprise. In ArcGIS Pro, right-click the PalmTreesVARI layer in the Contents pane and select Sharing, then Share as Web Layer. It will publish to your ArcGIS Online account.
Using App templates to review deep learning accuracy
Deep learning tools provide results with accuracy that is proportional to the accuracy of the training samples and the quality of the trained model. In other words, the results are not always perfect. You can assess the quality of the model results by checking through the trees where the Confidence score, stored in the deep learning result, is lower than a given value. Instead of zooming to each record using an attribute filter in ArcGIS Pro, the Image Visit configurable web app template allows you to quickly review the accuracy of your results in a web application.
Using Workforce for ArcGIS to perform field verification
Workforce for ArcGIS is a mobile app solution that uses the location of features to coordinate your field workforce. You can use the Workforce app to assign tasks to members of your organization so that all the trees with a VARI score that is listed as Needs Inspection can be assigned to someone in the field, checked, and marked with a suggested treatment.
Using Operations Dashboard to monitor project progress
Finally, you can monitor the progress of the assignments dispatched in your Workforce for ArcGIS project using Operations Dashboard for ArcGIS. Operations Dashboard for ArcGIS is a configurable web app that provides visualization and analytics for a real-time operational view of people, services, and tasks.
In this lesson, you obtained open-source drone imagery and created training samples of palm trees in the image. Those image chips were provided to a data scientist as image chips and used by a trained deep learning model to extract over 11,000 palm trees in the image.
You learned about deep learning and image analysis, as well as configurable apps across the ArcGIS platform. You can use this workflow for any number of tasks, if you have the imagery and knowledge of deep learning models. For example, you can use these tools to assess structural damage resulting from natural disasters, count vehicles in an urban area, or find man-made structures near geological danger zones.
You can find more lessons in the Learn ArcGIS Lesson Gallery.