Schedule automated near real-time data updates

Water levels in reservoirs change constantly: decreasing during the dry months when water is being consumed and increasing during the rainy season.

In this tutorial, you'll configure a layer in ArcGIS Online to consume a near real-time web service provided by the Texas Water Development Board. You'll create a feature service in ArcGIS Online using water data for Texas. You'll create a feature view to preserve symbology when the data is updated and configure a script to overwrite the feature service when new data is available. Finally, you'll schedule a task on a local computer to automate the updates.

This tutorial was last tested on April 24, 2023.

View final result

Requirements

• ArcGIS organizational account with a Publisher, Facilitator, or Administrator role in an ArcGIS organization (see options for software access)

Publish a feature service

In this tutorial, you’ll create a hosted feature layer in ArcGIS Online to display near real-time reservoir data for Texas. First, you’ll download reservoir data for Texas. Next, you’ll use this information to publish a hosted feature layer and configure a view of it. You'll configure a Python script to update the feature layer and set up a Windows task to run it. The Texas reservoirs data feature layer will be overwritten as new data becomes available, so any web maps based on the feature view will always display the latest information.

Access near real-time reservoir data for Texas

First, you'll access reservoir information from the Water Data for Texas website. Then, you'll download the data in GeoJSON format and store it locally.

1. In a browser, go to the Water Data for Texas website. Scroll down and under the Statewide Conservation Storage chart, click Recent Averaged Storage Conditions and click GeoJSON.

   

   The website redirects you to https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson, which contains the near real-time reservoir data for Texas. This is a GeoJSON file. For each reservoir in Texas, it contains information such as location (coordinates), name, conservation capacity, percent full, and last updated time (time stamp).

   

   Note:

   The Water Data for Texas website is dynamic. It is frequently updated, as soon as new data is available. The time stamp value for each reservoir displays the time of the last update.

2. Click on the page and press Ctrl + A to select all of the text on the recent conditions page, then right-click and click Copy. You can also press Ctrl + C to copy the text.

   

3. Open a text editor such as Notepad++ and press Ctrl + V to paste the text into a new file.
4. Save the file to your local computer. In the Save as window, navigate to a folder on your computer such as C:\Temp\, then type recent-conditions.geojson for the file name. For Save as type choose All Files.

   Note:

   

   Check the file in your local folder after you save it to verify that the file extension is .geojson and not .geojson.txt.

Add an item to ArcGIS Online

You'll use the local GeoJSON file of reservoir data for Texas to create a hosted feature layer in ArcGIS Online.

1. Sign in to your ArcGIS organizational account.

   Note:

   If you don't have an organizational account, see options for software access.

2. If necessary, click Content, and click My Content.

   

3. Click New item.

   

4. Drag the recent-conditions.geojson file that you just saved onto the upload pane.
5. Accept the default option to Add recent-conditions.geojson and create a hosted feature layer and click Next.
6. Accept the defaults set on the Fields page and click Next
7. On the next page of the New item pane, enter the following:

   • For Title, type TX_reservoirs_recent_ and add your name or initials.
   • For Tags, type Texas, Reservoir, Recent Conditions and press Enter after each.
   • For Summary, type Near real-time reservoir information for Texas.

8. Click Save.

   The process will take a few moments. After it completes, the Overview page for the TX_reservoirs_recent_(your name) hosted feature layer appears. In addition to this layer, a copy of the TX_reservoirs_recent GeoJSON file is added to your My Content section.

You accessed near real-time reservoir information for Texas and created a hosted feature service in ArcGIS Online. Next, you'll use the feature service to create a feature view. This feature view can be customized, shared publicly, and added to your apps. Creating the feature view allows the feature service to be updated without disrupting the symbology and pop-ups configured for the view. This simplifies the process of updating the data later.

Create a hosted feature layer view

Now you'll create a hosted feature layer view from the hosted feature layer, share the hosted feature layer view publicly, update its symbology, and configure its pop-up.

Create the hosted feature layer view

The Texas reservoirs hosted feature layer that you created will be overwritten when new information is available. The overwrite process will remove any custom symbology and pop-up settings and restore it to the default settings. To avoid changing how the data is displayed in your web maps and apps, you will create a hosted feature layer view in ArcGIS Online and configure its symbology and pop-up settings. These will remain the same and display the latest data after the hosted feature layer is overwritten.

1. On the TX_reservoirs_recent_(yourname) feature layer Overview page, click Create View Layer.

   

2. In the Create View Layer window, in the Layers pane, the TX_reservoirs_recent_(yourname) - points layer is selected by default. Click Next twice.

   In the Included layers pane you can set layer definitions To learn more about these options, view the Create hosted feature layer views documentation.

3. In the Create pane, enter the following:

   • For Title, type Texas Reservoirs Recent Conditions (yourname).
   • For Folder, you can optionally specify a folder where you want to save the view layer.
   • For Tags, type Water, Texas, Reservoirs, Environment, Natural Resources, Live Feeds, livefeeds, pressing Enter after each.
   • For Summary, you can optionally provide additional text about the view layer.

4. Click Create. Once the view has been created a Success pop-up window appears. Close this window.

   A new hosted feature layer view item is created from the hosted feature layer and the item details page for the new feature layer view appears.

5. On the Overview page, click Share, choose Everyone (public), and click Save.

   The hosted feature layer view is now public and accessible to everyone.

6. Click Content.

   There are now three items in your My Content page: the GeoJSON file, the hosted feature layer, and the view.

   

Note:

It is a good practice to use a hosted feature layer view to share a hosted feature layer, keeping the original hosted feature layer private. You can change the pop-up and the symbology of the hosted feature layer view without modifying the original hosted feature layer. During an update process, the hosted feature layer can be overwritten, and the new values will be reflected in the hosted feature layer view. The symbology of the hosted feature layer view is saved with it and is not affected when the original hosted feature layer is overwritten.

Change symbology

The hosted feature layer view has the reservoir data for Texas. You'll customize its symbology to better represent the information in the layer using the reservoirs' capacity for different feature sizes and using the current percent-full level for different colors.

1. Click the Texas Reservoirs Recent Conditions_(Your Name) feature layer to open the hosted feature layer view item page.

   The item page for Texas Reservoirs Recent Conditions_(Your Name) appears on the Overview tab.

2. Click the Open in Map Viewer Classic drop-down arrow and select Open in Map Viewer.

   

3. If necessary, in the Layers pane, click the Texas Reservoirs Recent Conditions layer to activate it.
4. On the Settings (light) toolbar, click Styles.

   

5. Under Choose attributes click the + Field button.

   

6. In the Add fields window that appears, choose percent_full and conservation_capacity and click Add.

   

   The attributes are added to the Styles pane.

7. Drag percent_full above conservation_capacity.

   

8. Under Pick a style, for Color and Size, click Style options.

   

9. For Counts and Amounts (Color), click Style options.

   

10. In the Style options pane, on the histogram, click the minimum value break, type 50 and press Enter.

    

11. Click the color ramp under Symbol style.

    

12. In the Symbol style pane that opens, under Fill, click the blue-gray color ramp (it is named Blue 10), and click OK.

    

    To see the name of a color ramp, point to the color ramp.

13. In the Style options pane, click Done.
14. For Counts and Amounts (size), click Style options.

    

15. Click the symbol under Symbol style. In the Symbol style pane that appears, under Outline, expand Custom Color and type #283849 and press Enter.

    

16. On the Style options pane, click Done, and click Done two more times.

    The symbology for the layer is now set. Next you'll configure the map and save it.

17. On the Contents (dark) toolbar, click Basemap and click Light Gray Canvas.

    

    The symbology for the layer and basemap are now set.

    

    The final map shows the recent conditions of reservoirs in Texas. The color of each point represents the percentage full of each reservoir, and the size of the point represents the total reservoir capacity.

Configure the pop-up

You will improve the pop-up for the layer to display only the relevant information. You'll show the name, capacity, and percent full for each reservoir.

1. On the Settings toolbar and click Configure Pop-ups.

   

2. On the Pop-ups pane, expand Title, delete the existing text and click the Add Field button.

   

   The Add field window appears.

3. From the drop-down, choose full_name.

   The pop-up title will now show the contents of the attribute full_name.

4. On Fields list, click the Options button and choose Delete.

   

5. Click the Add Content button and click Text

   

   

   A text box appears.

6. In the text box, type {percent_full}% full as of {timestamp} and click OK.

   

   The attribute field names in curly braces will be replaced with the values from those fields when the pop-up is opened by clicking a feature.

   The pop-up will now show the title and text you configured when a feature is clicked.

   

   You can still improve the format of the fields by configuring how each of the fields is displayed.

7. On the Settings toolbar, click the Configure Fields tab and click the percent_full field.
8. Change the Display name to Percent Full (%), and in the Significant Digits drop-down menu, choose 0 Decimal Places.

   

9. Click Done.
10. On the Contents toolbar, click Save and open and click Save as.
11. In the Save map window, enter the following:

    • For Title, type Texas Reservoirs (Your Name)
    • Add Tags, Water, Texas, and Reservoirs.
    • For Summary, type A near real-time map of Texas reservoirs showing percent full and capacity.

12. Click a reservoir on the map.

    The pop-up displays the name of the reservoir, the percent full, and the date and time last updated.

You've created a hosted feature layer view in ArcGIS Online from the reservoirs hosted feature layer. The hosted feature layer view allows you to customize the display of the Texas reservoir layer and retain these customizations even when the hosted feature layer is overwritten. Next, you'll overwrite the Texas reservoirs hosted feature layer using a simple Python script. The updated information will be reflected automatically in the hosted feature layer view.

Update a hosted feature layer

Now you'll configure a short script to overwrite the Texas reservoirs hosted feature layer. First, you'll download the overwrite script from ArcGIS Online. Then, you'll retrieve the hosted feature layer parameters such as item title and item ID. Finally, you'll run the script locally using the Python Command Prompt. The overwrite process allows you to update the hosted feature layer with the latest information.

Download the overwrite script

First, download the script for overwriting a hosted feature layer in ArcGIS Online.

1. Download the OverwriteFS.zip file to your computer.

   The .zip file contains two files: the Python script (OverwriteFS.py) and a document (ReadMe.pdf).

2. Extract the contents of the .zip file to a folder on your computer, for example, C:\Temp\.
3. Open a text editor such as Notepad or Notepad++ to a new, blank text file.
4. In ArcGIS Online, under My Content, click the TX_reservoirs_recent_(yourname) hosted feature layer.

   You will need to know the title and the item ID for this hosted feature layer in order to update it. You will save this information in a text document for reference.

5. On the TX_reservoirs_recent item Overview page, copy the item title and paste it into the text file.

   

6. Copy the item ID at the end of the item's URL and paste it into your text file. Save the file, but keep it open, you will need this information soon.

   In this example, the item title is TX_reservoirs_recent_(yourname) and the item ID is 4309a6d6bb77477f986acba13b5f9ef6. The item ID is unique for each item in ArcGIS Online. Your item ID will be different.

Store ArcGIS Online credentials locally

ArcGIS Online requires login credentials to overwrite items. You'll store ArcGIS credentials locally in a secure profile. This profile will be used in the Python script to overwrite the reservoirs hosted feature layer.

1. Open the Windows Start menu, search for Python Command Prompt, and open it.
2. Type cd and type the path to the folder where you extracted the OverwriteFS.py script and press Enter.

   For example, if you extracted the script to C:\Temp, you would type cd C:\Temp.

   

3. Type python and press Enter.

   

   This starts the Python console in the Python Command Prompt window. The Python version is noted in the console and the input prompt changes to >>>.

   Next you will import the arcgis Python module.

4. In the Python console, type import arcgis and press Enter.

   

   Note:

   The Python command prompt and the arcgis Python module are included with the ArcGIS Pro installation.

   Now you will store your ArcGIS Online credentials locally by adding a profile.

5. Type the command gis = arcgis.GIS(profile='learn_user', username= '<account>', password= '<pass>'), replace <account> with your username account and <pass> with your password. Press Enter.

   

   Note:

   Your ArcGIS Online credentials will be securely saved in the learn_user profile. You can learn more about storing ArcGIS Online credentials locally in a profile by visiting the Working with different authentication schemes page.

6. Type exit() to close the Python console and return to the command prompt.

Overwrite the hosted feature layer

Now use the overwrite Python script, the ArcGIS Online profile, and the hosted feature layer parameters (item ID and title) to update the reservoirs hosted feature layer.

1. In the Python command prompt, in the C:\Temp\ folder that contains the OverwriteFS.py script, run the following command: python OverwriteFS.py learn_user <item id> <item title> <GeoJSON URL>, replacing <item id> and <item title> with the parameters of the hosted feature layer you saved in your text file.

   In this code, learn_user is the ArcGIS Online profile saved locally. Replace <item id> and <item title> with the parameters of the hosted feature layer that you saved in your text file. In the example, you would replace <item id> with 3b2b385c923147349dec02025e10aa55 and <item title> with TX_reservoirs_recent_(yourname). You would replace <GeoJSON URL> is the URL of the data source with the updated data, https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson.

   Note:

   Because you've already created the hosted feature layer using the most recent data, the script won’t make any changes. Instead, you can temporarily use a file from https://downloads.esri.com/learnarcgis/schedule-automated-near-real-time-data-updates/recent-conditions.geojson, which contains a snapshot of the recent-conditions.geojson file for December 26, 2019.

   If you run it with the URL to the live Texas water site, the command will resemble the following (the item ID will be different):

   python OverwriteFS.py learn_user 20af483d68e54ff5835753de0e90ce27 TX_reservoirs_recent_(yourname) https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson

   If you download the snapshot of the recent-conditions.geojson file for December 26, 2019, save it to the C:\Temp\ folder that you are working in (replace the existing file). The command will resemble the following (the item ID will be different, and the path will be a local file path instead of a URL):

   python OverwriteFS.py learn_user 20af483d68e54ff5835753de0e90ce27 TX_reservoirs_recent_(yourname) c:\Temp\recent-conditions.geojson

   Press Enter to run the command.

   The command prompt displays the results. The Performing Overwrite… Success! Elapsed Time message at the end means that the script correctly updated the hosted feature layer.

   

2. In a browser, open the Texas Reservoirs map in Map Viewer Classic. Click any reservoir to view its pop-up.

   The pop-up shows the updated time and percent-full values. The symbology of the hosted feature layer view remains as you set it.

You ran a Python script to overwrite a hosted feature layer in ArcGIS Online. Next, you'll create and schedule a task in Windows to automatically run the overwrite script at regular intervals.

Create and schedule a task

You ran a Python script to update a hosted feature layer. Now, you'll automate running the script by scheduling a task in Windows. First, you'll create a .bat file. Then, you'll create a task in Windows Task Scheduler. The task will regularly run the .bat file, which will run the overwrite Python script, which will check for updates and overwrite the hosted feature layer if new data is available.

Create a .bat file

Windows .bat files are simple text files with a set of instructions to be run by the command line interpreter.

1. Open a text editor such as Notepad or Notepad++.
2. Copy and paste the following text into the text editor:

   "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:\Temp\OverwriteFS.py" learn_user 3b2b385c923147349dec02025e10aa55  TX_reservoirs_recent_(yourname) https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson

   You can also type the text it as a single line.

3. Change the item ID from 3b2b385c923147349dec02025e10aa55 to the item ID of your hosted feature layer.
4. Change TX_reservoirs_recent_(yourname) to the name of your item.
5. Save the file in your C:\Temp\ folder, and type TXreservoirs_overwrite.bat for the file name. In the Save as type drop-down menu, choose All Files (or if you are using Notepad++, choose Batch file).

   

   Next, you'll create a task to run the .bat file.

Create a basic task

Now that you have a .bat file, you’ll schedule a task to run it at regular intervals.

1. Open the Windows Start menu. Search for and open Task Scheduler.
2. In the Actions panel, click Create Basic Task.

   

   The Create Basic Task Wizard window appears and starts on the Create a Basic Task tab.

3. In the Create Basic Task Wizard window, for Name, type TX reservoirs, and for Description type Task to run the TXreservoirs_overwrite.bat file to update the feature service.

   

4. Click Next.

   The window updates to the Task Trigger tab.

5. For When do you want the task to start?, confirm Daily is selected and click Next.

   

   The wizard updates to the Daily tab.

6. For Start date, choose today’s date. Set the start time to be one hour after the current time (4:35 p.m. in the example).

   

7. For Recur every, type 1 to make the task run every day and click Next.
8. On the Action page, confirm that Start a Program is selected as the task action and click Next.
9. On the Start a Program page, in the Program/script text box, type C:\Temp\TXreservoirs_overwrite.bat and click Next.

   Note:

   If you saved the .bat file in another folder, ensure you enter the correct path to the .bat file. You can also click the browse button and browse to the location of the file.

   

   The Finish tab appears in the wizard.

10. Check the box for Open the properties dialog for this task when I click Finish box and click Finish.

    

11. Under Security options, check the box for Run whether user is logged on or not and check Do not store password. Click OK.

    

The task will run at the scheduled time, updating the hosted feature layer with the latest information from the Water Data for Texas website. The hosted feature layer view will show the updated information from the hosted feature layer while retaining the same symbology and pop-up configuration.

In this tutorial, you downloaded data from the Water Data for Texas website and created a hosted feature layer. You created, customized, and shared a hosted feature layer view of the hosted feature layer to isolate the public item from the update process. You adapted, downloaded, and used a Python script to obtain the latest data from the website and update the hosted feature layer. You also created a .bat file and scheduled a Windows task to run the Python script. When the task runs, the overwrite Python script will update the hosted feature layer. You configured the task to run daily, so any web maps or apps that display the hosted feature layer view will show the latest data each day.

Acknowledgements

• Water Data For Texas from Texas Water Development Board (TWDB).
• Cartography, symbology, and images by Emily Meriam.