Capture training images

To create an image classification model, you need a large collection of images that identify each class of feature that you need your model to recognize. You'll identify two classes: stop signs and speed signs.

For road signs, you should have approximately 200 images for each class. Every image does not need to be a unique road sign, but at least 20 percent of them should be unique. Multiple photos of the same sign (with different lighting, angle, and distance to the object) are acceptable. Varied lighting, angle, and distance to the object photos enhance the model, resulting in better accuracy and higher confidence and acceptance of prediction.

Note:

The use of image classification for field work is in its infancy and is considered technology for early adopters. Creating models that result in higher accuracy and confidence require multiple iterations as well as testing and experimentation. This lesson provides a workflow and a framework to demonstrate the steps required to create models for image classification in the field. For support in using these steps in your image classification project, please go to the Survey123 early adopter community forum.

Create and publish a survey

First, you'll create a survey using Survey123 Connect that captures an image, the sign type, and allocates each record a unique ID.

  1. Go to the SignImageCollection item on ArcGIS Online and click Download.

    Download button

  2. Open ArcGIS Survey123 Connect.
    Note:

    If you don't have ArcGIS Survey123 Connect, you can download it from the Survey123 product page. Under Survey123 Connect desktop app, download the installer for the appropriate operating system. Run the installer and follow its steps to install the program.

  3. On the ribbon, click the menu button and click Sign in.

    Sign in button

  4. Sign in to your ArcGIS organizational account.
    Note:

    If you don't have an organizational account, you can sign up for an ArcGIS free trial.

  5. Drag and drop the .xlsx file that you downloaded onto the gallery of surveys in Survey123 Connect.

    Survey form dragged onto list of surveys

  6. When the import is completed, click OK.

    XLSForm import complete window

  7. Review the survey questions.

    The Sign Image question includes a camera button, which allows survey takers to attach a picture using their mobile devices. You'll use this survey to collect images to be used by the model to train and identify images used by Survey123 SmartCamera functionality.

    Camera button
  8. Click Publish.

    Publish button

  9. In the Publish SignImageCollection window, click Publish survey.

    Publish survey button

    The survey is published to your ArcGIS organization, along with all relevant feature services and form items.

    Publishing complete message

  10. When publishing is complete, click OK.
  11. On the side menu click the three dots and in the More Actions windows that displays, choose Manage in Survey123 website.

    Manage in Survey123 website option

    In a new browser window, the published survey details opens to the Overview page in the Survey123 website. To allow others to use this survey, you'll share it publicly.

  12. On the ribbon, click Collaborate.

    Collaborate button

  13. In the Share survey tab, for Who can submit to this survey?, choose Members of my organization and click Save.

    Share survey tab

    You've now created a survey to collect images that can be used to train an image classification model.

    Feel free to share the survey with colleagues and start collecting various road signs, remembering to include photos taken in different light conditions, from different angles and distance. The more variety and examples the better the resultant model will be able to predict what it is observing. You should have approximately 200 images for each category of image you want to classify.

Use the survey

Now that the survey is published, you can download it to your mobile device and use it.

If you're unable to capture photos of street signs in your local area, you can use a sample geodatabase of photos created for this lesson to move on to the next section, and skip these steps.

  1. On your mobile device, open the Survey123 field app.
    Note:

    If you don't have the Survey123 field app, go to the Survey123 product page. Under ArcGIS Survey123 field app, click the download link for your computer or device's operating system. Run the downloaded installer and follow its instructions to install the field app.

  2. Click Sign in. Sign in to your ArcGIS organizational account.

    Sign in button

    The app opens to the My Surveys page. To collect records for the survey, you'll need to download it.

  3. Click the menu button and choose Download Surveys.

    Download Surveys option

    The Download Surveys page shows all surveys that have been shared to your organization.

  4. In the search bar, type SignImage Collection. From the search results, locate your SignImageCollection survey and click Download.

    SignImageCollection survey

    The survey is downloaded to your device.

  5. Once the survey is downloaded, click the back arrow to return to the survey gallery.

    The SignImageCollection survey appears in the survey gallery.

    Survey in My Surveys page

  6. Click the SignImageCollection survey. On the survey contents page, click Collect.

    Collect button

  7. For the Sign image question, click the camera button.
    Note:

    The camera button includes the option to add photos from file or use the device camera to collect photos. If you cannot field test the survey, you can use this collection of sample images. Open the photos on your computer and use your device to take photos of the photo on the computer screen.

    Camera button

  8. Take a photo of a stop or speed sign.

    Photo of a stop sign

  9. For the Sign type question, choose either Stop sign or Speed sign, depending on what sign you photographed.
  10. When you're satisfied, click the check button.

    If your device is online, you're prompted to choose either Send now, Continue this survey, or Save to outbox.

  11. Choose Send now.

    Sending window

  12. Continue to capture photos of stop and speed signs until you have submitted at least 200 of each. More photos equate to a better trained model and a higher success rate in identifying signs accurately. Try adding photos taken from different directions, angles and under various lighting conditions.
    Note:

    If you're unable to capture photos of street signs in your local area, you can use a sample geodatabase created for this lesson. The following steps will describe how to use your own data or our sample geodatabase to train a model.

You now have the input data required to build an image classification model.


Train a model

Next, you'll use ArcGIS Notebooks to create an image classification model. You'll need to create a notebook that has advanced plus GPU capability. This capability will use 30 credits per hour.

Alternatively, you can download and use the referenced notebook locally, but you must ensure you have suitable hardware.

Note:

Interested in learning how to use ArcGIS Notebooks to write Python code for ArcGIS? Open this Hello Notebook! lesson to get started. If you see a Download button but do not see an Open Notebook button, you may not be signed in. If signing in doesn't give you access to the Open Notebook button, your account is not enabled for notebook use. Please contact the ArcGIS administrator for your organization to get access to Notebooks. The Hello Notebook! lesson consumes approximately 2 ArcGIS Online credits, using a Premium Content layer from Living Atlas.

You may also be interested in the other Notebook lessons available in the Learn Python with ArcGIS Notebooks path.

The guide Enable ArcGIS Notebooks for Your Organization in ArcGIS Online may be helpful for administrators who have not enabled ArcGIS Notebooks before.

Set up ArcGIS Notebooks

First, you'll create a notebook in ArcGIS Notebooks.

  1. Go to the SignImageClassificationModel on ArcGIS Online and click Download.

    Download

  2. Rename the downloaded file by adding your initials to the .ipynb file.

    Items in an ArcGIS organization must be unique, if someone else in your organization has used this lesson, there may already be a file with the default name uploaded.

  3. If necessary, click Sign In and sign in to your ArcGIS organizational account.
  4. On the ribbon, click Content.
  5. Click New item.

    New item button

  6. Drag and drop or browse to your .ipynb file then add it to the New item pane.

    New item window

  7. In the New item pane, set the following parameters:
    • For Title, type Sign Image Collection Model_(YourName).
    • For Folder, choose or create a suitable project folder.
    • For Tags, type SignImageCollectionModel.
    • For Summary, type Sign Image Collection Model.
  8. Click Save.

    Item details

    The item is created and its details page appears.

  9. In the details page, click Settings.

    Settings button

  10. For Notebook Runtime, choose ArcGIS Notebook Python 3 Advanced with GPU support – 4.0.

    Notebook Runtime option

  11. Click Save.
  12. Click Overview.

    Overview button

  13. Click Open Notebook.

    Open Notebook button

    Instructions to execute the notebook and create the model files are included in the notebook text.

  14. Complete the steps in the notebook and then return to this lesson.

    The notebook steps include:

    • Setting up the environment.
    • Downloading training images from the feature layer.
    • Training the model.
    • Testing the model (optional).
    Note:

    The notebook is designed so it can be run using the supplied data. It is important to follow the steps described inside the notebook, especially restarting the kernel when directed.

    If you've collected your own images, you can change the item IDs referenced in the notebook to create a model from your own images.

Download model results

Once the notebook is successfully executed, a package of files is generated. You'll download these files for inclusion in your test survey.

  1. In ArcGIS Notebooks, choose Files.
  2. Browse to /arcgis/home/tmp/fa40cf680eb4436daf4109b887b52b30/models/

    Generated files

  3. Download the .zip file.
  4. Extract the contents of the .zip file to a suitable location.

You've downloaded the image classification model files that can be used with the Survey123 SmartCamera to identify speed and stop signs.


Test the model

Next, you'll add the image classification model files to a survey that includes the Survey123 SmartCamera to identify speed and stop signs.

Ideally, you should go out into the field to test this survey, but if you can't, or if you don't have speed and stop signs nearby, you can also simulate this testing with photos on your computer.

Create and publish a survey

First, you'll create a survey using Survey123 Connect that will use the model files you created to classify images as either stop or speed signs.

  1. Go to the SignImageClassificationModelTesting item on ArcGIS Online and click Download

    Download button

    Note:

    If you created model files using the notebook, you'll include those in this survey. If you do not have your own model files, you can download a collection of sample model files for use.

  2. Open Survey123 Connect. If necessary, sign in to your ArcGIS organizational account.
  3. Drag and drop the .xlsx file that you downloaded onto the gallery of surveys.

    Add file to create new survey

    A new survey is created with the name SignImageClassificationModelTesting.

  4. When the importing is complete, click OK.

    XLSForm import complete window

  5. Click Files.

    Files button

    The contents of the survey folder are displayed in the file browser.

    Survey files

  6. In the survey folder, create a new subfolder. Name the subfolder models.

    New folder

  7. From the model output location, find and copy the three model files with the file extensions .dlpk, .emd, and .tflite.
  8. Paste the three files into the models folder.

    Three files pasted into models folder

  9. Close the file browser and return to Survey123 Connect.

    The new survey has the following features:

    • The camera button in the Sign Image question looks different than the regular camera button in Survey123. This is the SmartCamera. It will use the model that you have included in the survey to interpret the contents of the photo captured.
    • The Sign Type question will automatically check the type of sign it detects when using the smart camera.
    • The Is This Correct question allows users to verify the success of the smart camera.

    Review survey questions

  10. When finished, click Publish.

    Publish button

  11. Click Publish survey.

    Publish survey button

    Survey123 Connect publishes the content to your ArcGIS organization. The publishing may take longer than normal because are including the image classification files in the survey.

  12. When publishing is complete, click OK.

    OK button

  13. Click the More Actions button and choose Manage in Survey123 website.

    Manage in Survey1213 website option

    The published survey opens to the Overview page in the Survey123 website. To allow others to use this survey, you'll share it publicly.

  14. On the ribbon, click Collaborate.

    Collaborate option

  15. In the Share survey tab, for Who can submit to this survey?, choose Members of my organization. Click Save.

    Share survey tab

    You've now created a survey to test an image classification model.

    The next step is to use the survey to test the smart camera. Similar to when you were collecting images to train the image classification model, you should test with photos taken in different light conditions, from different angles and distance. The more variety you test successfully with, the better confidence you can have when sharing these models with others.

Use the survey

Now that the survey is published, you'll download it to your mobile device. You can go outdoors and take photos of stop and speed signs.

Note:

If you cannot field test the survey, you can use this collection of sample images. Open the photos on your computer and use your device to take photos of the photo on the computer screen.

  1. Open the Survey123 field app on your mobile device.
  2. If necessary, sign in to your ArcGIS organization.

    The app opens to the My Surveys page. To collect records for the survey, you'll need to download it.

  3. Click the menu button and choose Download Surveys.

    Download Surveys button

    The Download Surveys page shows all surveys that have been shared to your organization.

  4. In the search bar, type Sign image classification model testing. From the search results, locate your SignImageClassificationModelTesting survey and click Download.

    Download button for the survey

    The survey is downloaded to your device.

  5. Once the survey is finished downloading, click the back arrow to return to the survey gallery.
  6. Click the SignImageClassificationModelTesting survey. On the survey contents page, click Collect.

    Collect button

  7. For the Sign image question, click the SmartCamera button.
    Note:

    If you're testing the survey using photos on your computer, prepare in advance and display the photo in a file browser or other suitable application.

    SmartCamera button

  8. Take a photo of a stop or speed sign.

    Add a photo of a sign

    The smart camera displays a classification (stop or speed) and confidence value, shown at the bottom of the screen.

  9. When you're satisfied with the photo, click the camera button to capture the photo and return to the survey page.

    The Sign Type question is automatically updated with the same value as was shown in the camera preview. This image classification has been done by the model files that were included in the survey.

    Verify auto sign identification

    Next, you'll need to validate the smart camera result.

  10. Click the check button.

    When you click the button, you are prompted to answer a question verifying the identification of the sign.

    Required question

  11. If the model correctly predicted the sign type, for the Is this correct? question, click Yes.

    Is this correct question

  12. Click the check button.

    If your device is online, you're prompted to choose either Send now, Continue this survey or Save in Outbox.

  13. Click Send now.

    Send now option

  14. Continue to capture photos of stop and speed signs until you have submitted at least 10 of each.

    Additional photos

You've now used the smart camera to classify photos as containing either stop signs or speed signs. If you're satisfied with the results of the smart camera, you can use these model files in any survey to identify stop or speed signs. If you're not satisfied with the results, you may need to train the model again.

Consider the following options to improve the quality of the model:

  • Review the images used to train the model.
  • Ensure the sign is in the middle of the image and crop if necessary.
  • Ensure you have a variety of lighting, angle, and distance to the object photos.
  • Continue to train the data by increasing epochs (iterations) until the validation loss (valid_loss) and train loss (train_loss) is less than 1. It is recommended to increase epochs by 5 for each train.

You can find more lessons in the Learn ArcGIS Lesson Gallery.