Train a model to identify street signs

While machine learning tasks in the office may replace some human activities, in the field they often complement or enhance field work activities instead of replace them. In this tutorial, you'll build and verify a model that can be used to identify street signs based on images manually captured in ArcGIS Survey123.

First, you'll acquire a representative collection of training images captured using Survey123. Next, you'll use the images to train a model with ArcGIS Notebooks. Once the necessary model files are created, you'll use them in Survey123 to test the model's predictions and classify survey photos.

This tutorial was last tested on January 11, 2024, using ArcGIS Survey123 3.19.114. If you're using a different version of ArcGIS Survey123, you may encounter different functionality and results.

Requirements

• Publisher or Administrator role in an ArcGIS organization: ArcGIS Online (see options for software access) or ArcGIS Enterprise (learn more)
• ArcGIS Survey123
• ArcGIS Notebooks with Advanced Notebooks (Premium Content) privileges.
• ArcGIS API for Python

Capture training images

Survey123 smart attributes can use an image classification model to extract values based on the objects the model detects in an image.

To create an image classification model, you need a large collection of images that identify each class of feature that you need your model to recognize. You'll identify two classes: stop signs and speed signs.

For road signs, you should have approximately 500 images for each class. Multiple photos of the same sign (with different lighting, angle, and distance to the object) are acceptable. Varied lighting, angle, and distance to the object photos enhance the model, resulting in better accuracy and higher confidence and acceptance of prediction.

Note:

Creating models that result in highest accuracy and confidence requires multiple iterations as well as testing and experimentation. This tutorial provides a workflow and a framework to demonstrate the steps required to create models for image classification in the field. Many more iterations and test images may be required to produce a model fit for use in a production environment.

Create and publish a survey

First, you'll create a survey using Survey123 Connect that captures an image and the sign type and allocates each record a unique ID.

1. Download the SignImageCollection .xlsx file.
2. Start ArcGIS Survey123 Connect.

   Note:

   If you don't have ArcGIS Survey123 Connect, you can download it from the Microsoft Store or the Survey123 product page. Run the installer and follow its steps to install the program.

3. On the ribbon, click the menu button and click Sign in.

   

4. Sign in using your ArcGIS credentials.

   If you don't have an organizational account, see options for software access.

5. Drag the .xlsx file that you downloaded onto the gallery of surveys in Survey123 Connect.

   

6. When the import is completed, click OK.

   

7. Review the survey questions.

   The Sign image question includes a camera button, which allows users to attach an image using their mobile devices. You'll use this survey to collect images to train a model.

   
8. Click Publish.

   

9. In the Publish SignImageCollection window, click Publish survey.

   

   The survey is published to your ArcGIS organization, along with all relevant feature services and form items.

10. When publishing is complete, click OK.

    

11. On the side menu, click the More actions button. Click Manage in Survey123 website.

    

    In a new browser window, the published survey opens to the Overview page on the Survey123 website. To allow others to use this survey, you'll share it with your organization.

12. On the ribbon, click Collaborate.

    

13. On the Share survey tab, for Who can submit to this survey, choose Members of my organization and click Save.

    

    The Share survey window appears.

14. In the Share survey window, click Save.

    

    You've now created a survey to collect images that you can use to train an image classification model.

    You can share the survey with colleagues and start collecting various road signs, including photos taken in different light conditions, and from different angles and distance. The more variety and examples, the better the resultant model will be able to predict what it is observing. Include approximately 500 images for each category of image you want to classify.

15. Close Survey123 Connect.

Use the survey

Now that the survey is published, you can download it to your mobile device and use it.

If you cannot capture photos of street signs in your local area, you can use a sample geodatabase of photos created for this tutorial to move on to the next section, and skip these steps.

1. On your mobile device, open the Survey123 field app.

   Note:

   If you don't have the Survey123 field app, go to Google Play, the App Store, the Microsoft Store, or the Survey123 product page. If downloading from the product page, under ArcGIS Survey123 field app, click the download link for your computer or device's operating system. Run the downloaded installer and follow its instructions to install the field app.

2. Tap Sign in with ArcGIS Online. Sign in to your ArcGIS organizational account.

   

   Note:

   If you're using ArcGIS Enterprise, tap Manage ArcGIS connections, and sign in to your ArcGIS Enterprise environment.

   The app opens to the My Survey123 page. To collect records for the survey, you'll need to download it.

3. Tap Download surveys.

   

   Note:

   If you already have surveys downloaded on your device, to download a new survey, tap the Menu button and choose Download Surveys.

   The Download Surveys page shows all surveys that have been shared to your organization.

4. In the search bar, type SignImageCollection. From the search results, locate your SignImageCollection survey and tap Download.

   

   The survey is downloaded to your device.

5. Once the survey is downloaded, tap the back arrow to return to the survey gallery.

   The SignImageCollection survey appears in the survey gallery.

   

6. Tap the SignImageCollection survey. On the survey contents page, tap Collect.

   

7. For the Sign image question, tap the camera button.

   Note:

   The camera button includes the option to add photos from a file or use the device camera to collect photos. If you cannot field test the survey, you can use this collection of sample images. Open the photos on your computer and use your device to take photos of the images on the computer screen.

   

8. Take a photo of a stop or speed sign.

   

9. For the Sign type question, choose Stop sign or Speed sign, depending on what sign you photographed.
10. Tap the submit button.

    If your device is online, you're prompted to choose Send now, Continue this survey, or Save in Outbox.

11. Choose Send now.

    Continue to capture photos of stop and speed signs until you have submitted at least 500 of each. Try adding photos taken from different directions, from different angles, and under various lighting conditions.

    More photos equate to a better trained model and a higher success rate in identifying signs accurately.

    Note:

    If you cannot capture photos of street signs in your local area, you can use a sample geodatabase created for this tutorial. The following steps will describe how to use your own data or the sample geodatabase to train a model.

You now have the input data required to build an image classification model.

Train a model

Next, you'll use ArcGIS Notebooks to create an image classification model. You'll need to create a notebook that has advanced plus GPU capability. This capability uses 30 credits per hour.

Alternatively, you can download and use the referenced notebook locally, but you must ensure you have suitable hardware.

Note:

If you are interested in learning how to use ArcGIS Notebooks to write Python code for ArcGIS, complete the Hello, Notebook! tutorial to get started. If you see a Download button but do not see an Open Notebook button, you may not be signed in. If signing in doesn't give you access to the Open Notebook button, your account is not enabled for notebook use. Contact the ArcGIS administrator for your organization for access to Notebooks.

You may also be interested in the other Notebook tutorials available in the Learn Python with ArcGIS Notebooks tutorial series.

The Enabling ArcGIS Notebooks for your organization in ArcGIS Online guide may be helpful for administrators who have not enabled ArcGIS Notebooks before.

Set up ArcGIS Notebooks

First, you'll create a notebook in ArcGIS Notebooks. If you are unable to access ArcGIS Notebooks, you can download a trained model for use in the next section of this tutorial.

1. Download the SignImageClassificationModel notebook.
2. Rename the downloaded file by adding your initials to the .ipynb file.

   Item names in an ArcGIS organization must be unique. If someone else in your organization has completed this tutorial, there may already be a notebook file with the default name uploaded.

3. If necessary, sign in to your ArcGIS organization.
4. On the ribbon, click the Content tab.
5. Click New item.

   

6. Drag or browse to your .ipynb file and add it to the New item window.

   

7. In the New item window, set the following parameters:
   • For Title, type Sign Image Collection Model_(YourName).
   • For Folder, choose or create a suitable project folder.
   • For Tags, type SignImageCollectionModel.
   • For Summary, type Sign Image Collection Model.

   

8. Click Save.

   The item is created and its details page appears.

9. On the details page, click the Settings tab.

   

10. For Notebook Runtime, choose ArcGIS Notebook Python 3 Advanced with GPU support – 9.0.

    

11. Click Save.
12. Click the Overview tab.

    

13. Click Open notebook.

    

    Instructions to run the notebook and create the model files are included in the notebook text.

14. Complete the steps in the notebook and return to this tutorial.

    The notebook steps include the following:

    • Setting up the environment
    • Downloading training images from the feature layer
    • Training the model
    • Testing the model (optional)

    Note:

    The notebook is designed so it can be run using the supplied data. It is important to follow the steps described inside the notebook, especially restarting the kernel when directed.

    If you've collected your own images, you can change the item IDs referenced in the notebook to create a model from your own images.

Download model results

Once the notebook is successfully finished, a package of files is generated. You'll download these files for inclusion in your test survey.

1. In ArcGIS Notebooks, choose Files.
2. Browse to /arcgis/home/tmp/fa40cf680eb4436daf4109b887b52b30/models/

   

   Note:

   The default names are based on the date that the notebook is run. Your file names will vary from the image above.

3. Download the .zip file.

   Note:

   If you did not train a model, you can download a model for use in the next section of this tutorial.

4. Extract the contents of the .zip file to a suitable location on your computer.

You've downloaded the image classification model output files that you'll use in the Survey123 field app to automatically identify speed and stop signs.

Test the model

Next, you'll add the image classification model files to a survey to identify speed and stop signs.

Ideally, you would go into the field to test this survey, but if you can't, or if you don't have speed and stop signs nearby, you can also simulate this testing with photos on your computer.

Create and publish a survey

First, you'll create a survey using Survey123 Connect that will use the model files you created to classify images as either stop or speed signs.

1. Download the SignImageClassificationModelTesting .xlsx file.
2. Start Survey123 Connect. If necessary, sign in to your ArcGIS organizational account.
3. Drag the .xlsx file that you downloaded onto the gallery of surveys.

   

   A new survey is created with the name SignImageClassificationModelTesting.

4. When the importing is complete, click OK.

   

5. On the side menu, click Files.

   

   All the files that pertain to the SignImageClassificationModelTesting survey are visible. Your survey has a media folder. You'll copy and paste two model files from the folder you downloaded into the media folder.

   

6. Open the folder containing the model outputs. Find and copy the model files with the file extensions .emd and .tflite.

   

   Note:

   The names of these files are based on the date that the notebook is run. Your file names will vary from the image above.

   The .emd and .tflite files will provide your survey with the ability to detect speed and stop signs using your mobile device's camera.

7. Return to the survey's SignImageClassificationModelTesting folder for your survey. Open the media folder and paste the .emd and .tflite files into the media folder.

   

8. Close the file browser windows and return to Survey123 Connect.

   The new survey has the following features:

   • The Sign Image question looks the same as the regular camera button in Survey123 but it will use the model that you have included in the survey to interpret the contents of the photo captured.
   • The Sign Type question will automatically check the type of sign that is detected in the photo.
   • The Is this correct question allows users to verify the success of the camera.

   

   You're ready to publish your survey.

9. On the side menu, click Publish.

   

10. In the Publish SignImageClassificationModelTesting window, click Publish survey.

    

    Survey123 Connect publishes the content to your ArcGIS organization. The publishing may take longer than normal because you are including the image classification files in the survey.

11. When publishing is complete, click OK.

    

12. On the side menu, click the More actions button and choose Manage in Survey123 website.

    

    The published survey opens to the Overview page on the Survey123 website. To allow others to use this survey, you'll share it with your organization.

13. On the ribbon, click Collaborate.

    

14. On the Share survey tab, for Who can submit to this survey, choose Members of my organization. Click Save.

    

15. In the Share survey window, click Save.

    You've now created a survey to test an image classification model.

    The next step is to use the survey to test the model. You'll want to test the model's accuracy by taking photos in different light conditions, and from different angles and distance. The more variety you test successfully with, the better confidence you can have when sharing these models.

Use the survey

Now that the survey is published, you'll download it to your mobile device. You can go outdoors and take photos of stop and speed signs.

Note:

If you cannot field test the survey, you can use this collection of sample images. Open the photos on your computer and use your device to take photos of the images on the computer screen.

1. Open the Survey123 field app on your mobile device.
2. If necessary, sign in to your ArcGIS organization.

   The app opens to the My Survey123 page. To collect records for the survey, you'll need to download it.

3. Tap the Menu button and choose Download Surveys.

   

   The Download Surveys page shows all surveys that have been shared to your organization.

4. In the search bar, type SignImageClassificationModelTesting. From the search results, locate your SignImageClassificationModelTesting survey and tap Download.

   

   The survey is downloaded to your device.

5. Once the survey is finished downloading, tap the back arrow to return to the survey gallery.
6. Tap the SignImageClassificationModelTesting survey. On the survey contents page, tap Collect.

   

7. For the Sign Image question, tap the camera button.

   

8. Take a photo of a stop or speed sign.

   The camera preview displays a classification (stop or speed) and confidence value, shown at the top of the screen.

   

   After the image is captured, you are taken back to the survey page. The Sign Type question is automatically updated with the same value as was shown in the camera preview. This image classification has been done by the model files that were included in the survey.

   

   Next, you'll need to validate the smart attributes result.

9. If the model correctly predicted the sign type, for the Is this correct question, tap Yes. Otherwise, tap No.

   

10. Tap the submit button.

    If your device is online, you're prompted to choose Send now, Continue this survey, or Save in Outbox.

11. Tap Send now.

    

12. Continue to capture photos of stop and speed signs until you have submitted at least 10 of each.

    

You've now used the camera to classify photos containing stop signs or speed signs. If you're satisfied with the results, you can use these model files in any survey to identify stop or speed signs. If you're not satisfied with the results, you may need to train the model again.

Consider the following options to improve the quality of the model:

• Review the images used to train the model.
• Ensure the sign is in the middle of the image and crop if necessary.
• Ensure you have a variety of lighting, angles, and distances to the object photos.
• Continue to train the data by increasing epochs (iterations) until the validation loss (valid_loss) and train loss (train_loss) are less than 1. It is recommended that you increase epochs by 5 for each train.

You can find more tutorials in the tutorial gallery.

Acknowledgements

• Photos used in this tutorial were captured by members of the Esri Melbourne Research & Development Center.
• Streets basemap sources: Esri, HERE, Garmin, FAO, NOAA, USGS, OpenStreetMap contributors, and the GIS User Community.