A browser tab opens to a gallery showing any existing data pipelines you own.

The Data Pipelines Editor appears.

This editing environment allows you to add data inputs, provides access to tools to transform data, and allows you to write the processed data out to a feature layer.

A browser tab opens to an overview of the Capital Project Tracker dataset. This provides valuable information, such as how often the dataset is updated, when it was last updated, and a description for each column in the .csv table.

The Export dataset window appears.

When you click the API endpoint button, the Export format switches to JSON. The API endpoint allows access to the dataset through a URL.

While Data Pipelines accepts JSON data formats, you'll change the data format to CSV.

Next, you'll copy the URL for this dataset.

Now, you'll you add this .csv table to your data pipeline as a URL input.

The Add a URL window appears.

The Response format parameter is set automatically.

The URL element is added to the canvas.

The element's name is derived from the name of the .csv table you accessed. You'll change its name next.

Since the element is selected in the canvas, the URL pane is open. Here, you can configure or reconfigure any selected element on the canvas.

Next, you'll preview the dataset you added.

The Capital Project Tracker window appears. It's currently showing the table preview. By previewing your data, you'll know what your data will look like when you run the data pipeline.

The top of the table indicates the number of records is 1,006. This may not match the number of rows on the OpenData website. If the numbers do not match, it is because some of the records contain a character that makes the record multiple lines. In the URL panel, you can account for this.

The preview refreshes. The number of records is now 1,000.

Since this is only a table and no geometry has been defined, a map preview is not available. You'll allow for a map preview in a later section.

This lists all the fields in the dataset and their field types. Throughout the rest of the tutorial, you'll use a number of these fields to transform your data, including currentphase, designstart, latitude, and longitude.

If there were any warnings or errors in your preview dataset, they would be listed here.

The Tools pane appears. The tools listed, by category, allow you to manipulate the datasets in your data pipeline. You'll add the Filter by attribute tool to remove any row whose current phase is not construction. You'll also filter out any rows that have a latitude or longitude value of 0.

An element is added to the canvas. It needs to be connected to an existing element that contains data and configured.

The two elements are connected. Data will flow from the .csv file into the Filter by attribute tool when the data pipeline runs.

Next, you'll configure the filter to exclude any records that have a value of 0 for latitude or longitude and only show those rows that have a current phase value of construction.

The Query builder window appears.

Next, you'll add another expression for longitude.

The preview window appears. Below the title of this table is a count of the number of records. Previously, it was more than 1,000 records. Now, because of the filters you applied, it's less than 100.

These values meet the criteria of your query.

When the tool element was added to the canvas, it was given the default name of Filter by attribute. You'll change its name to make it more meaningful.

Before adding more elements, you'll save your data pipeline.

The Save data pipeline window appears.

The data pipeline is saved and the title appears above the Tools pane.

The Create geometry element is added to the canvas and its associated pane appears.

The two elements are connected. Next, you'll configure the Create geometry element. Since this table contains latitude and longitude values, you can create a point geometry.

Additional parameters appear. These parameters are used to determine which fields in your table contain X, Y, and Z values. Your dataset does not have Z values; this parameter will not be used.

The locations of the capital projects are visible on the map. You can click features to see their attributes in a pop-up window.

The Project geometry element is added to the canvas.

The two elements are connected. Next, you'll configure the Project geometry element.

The Browse coordinate systems window appears.

For a projected coordinate system, you'll use NAD 1983 (2011) StatePlane New York Long Isl FIPS 3104 (Meters). Its ID number is 6538.

The spatial reference of the data will be projected to a more suitable coordinate system.

The Calculate field element is added to the canvas.

The two elements are connected. Next, you'll configure the Calculate field element. You'll start by providing the new field with a name.

Next, you'll write an expression to calculate the field. This tool uses ArcGIS Arcade expressions to calculate fields.

The Arcade expression window appears. Here, you can write Arcade expressions to calculate field values. You'll copy and paste code that returns the number of years and days since a capital project's design began.

//Convert time between 2 fields to count years and days 

//Determine the total number of days
var TotalDays = DateDiff(now(), $record.designstart, "days")

//Determine the number of days
var RemainderDays = Floor(TotalDays % 365)

//Determine the number of years
var RemainderYears = Floor(DateDiff(now(), $record.designstart, "years"))

//Format the final text to account for year(s) and day(s)
if(RemainderYears == 1 && RemainderDays == 1){
  return RemainderYears + " year and " + RemainderDays + " day"
}
else if (RemainderYears == 1 && RemainderDays != 1){
  return RemainderYears + " year and " + RemainderDays + " days"
}
else if (RemainderYears != 1 && RemainderDays == 1){
  return RemainderYears + " years and " + RemainderDays + " day"
}
else{
  return RemainderYears + " years and " + RemainderDays + " days"
}

The desired format of this calculation is X years and Y days. To do this, the expression first determines the number of days since a capital project design started. Since the number of days may be more than one year, the code divides the number of days by 365 and returns the remainder value. This returns the Y value in the desired format. Then, the expression calculates the number of years since the project began. This is the X value in the desired format. The last part of the expression, starting on line 12, formats the years and days text to make them singular or plural based on the number of days or years since the design started.

The Elapsed_Time field is added to the table and calculated.

For each project, the number of years and days since the project began is recorded in an understandable format.

You'll rename this element to clarify the field that it calculates.

Finally, you'll save your data pipeline.

A browser tab opens to an overview of the 2020 Neighborhood Tabulation Areas (NTAs) - Tabular dataset. Like the Capital Project Tracker dataset, this page provides an overview of the dataset and how frequently it's updated.

The Export dataset window appears. Since this dataset contains fewer than 1,000 rows, you won't need to change to URL like you did with the Capital Project Tracker dataset.

• Click API endpoint.
• For Data format, choose GeoJSON.
• For Version, select SODA2.

As you did with the .csv file previously, you'll copy the URL for this dataset so you can add it to your project.

The Add a URL window appears.

The Response format parameter is automatically set to GeoJSON.

The URL element is added to the canvas.

Again, the name is unintuitive, so you'll rename this element.

Next, you'll preview the dataset you added.

In the preview window, observe the fields provided with this dataset. The ntaname field is the attribute that you'll be adding to the capital projects point layer using a spatial join.

A map appears with the neighborhoods drawn as polygons.

The two elements are connected. Because you copied this element, the coordinate system is already selected. Now, the neighborhoods dataset uses the proper coordinate system.

The Join element is added to the canvas.

Next, you'll connect the Calculate field and the Project geometry elements to the Join element. The Join element has two input ports. The upper input port is for the target dataset. The target dataset is the dataset that will have additional attributes added to it. The lower input port is for the join dataset. This is the dataset that will share its attributes to the target dataset. In this case, you want the capital projects to receive the attributes from the neighborhood dataset. Therefore, the Calculate field element is the target dataset and it will be connected to the upper input port.

The two input elements are connected to the Join element. In the Join panel, the Target dataset and Join dataset are filled in based on the elements you linked.

Next, you'll set the Join element to use a spatial relationship.

Additional parameters appear. The Target geometry and Join geometry parameters are automatically completed. But you still need to choose a Spatial relationship. This defines how the target and join datasets are joined. Since the capital project points fall inside of the neighborhood polygons, you'll use the Intersects relationship.

The preview window appears. This dataset continues to represent the capital project points. In the table preview, the fields that initially appear are from the capital project points.

The fields at the far end of the table are the fields from the neighborhoods. Now, for each capital project, you know the neighborhood that it falls within.

As you perform more joins, the number of fields becomes cumbersome, especially since many of these fields have not been requested by your stakeholders. Later, you'll remove the unnecessary attribute fields.

In the next section, you'll add a second spatial join. To prevent confusion, you'll rename the first Join element.

The item details page for the NYC_Community_Districts layer appears.

Next, you'll add the feature layer to your content using the URL you copied.

Now you have a feature layer in your content that you can add to your data pipeline.

The Select a feature layer window appears showing your My content folder that contains the NYC_Community_Districts layer.

A Feature layer element is added to the canvas.

The BoroCD field is the attribute that you'll be adding to the capital projects point layer using a spatial join.

A map appears with the community districts drawn as polygons.

The two elements are connected. Now, the community districts dataset uses the proper coordinate system.

The Join element is added to the canvas.

Next, you'll connect the first Join and the Project geometry elements to the second Join element. The first Join element will be the target dataset parameter and the Project geometry element will be the join dataset parameter.

Next, you'll set the second Join element to use a spatial relationship.

The preview window appears.

The first fields you see are from the capital projects dataset. Next, you see the fields from the neighborhood dataset. Finally, at the end of the table, you see the fields from the community districts dataset. Now, each capital project has information about the community district that it falls within.

Since there is another Join element on the canvas, you'll rename this second Join element for clarity.

The Select fields element is added to the canvas.

Next, you'll connect the Join and Select fields elements.

Next, you'll choose which fields you want the output dataset to contain.

The Select fields window appears. You'll choose the fields that are of interest to your stakeholders. You'll also choose the GEOMETRY field. This field is necessary for you to be able to display your output dataset as points. Otherwise, the output would be a nonspatial feature layer or hosted table.

• fmsid
• currentphase
• GEOMETRY
• Elapsed_Time
• ntaname
• BoroCD

The fields that will be included in the output are listed.

The preview window appears.

Instead of having an overwhelming number of fields, your output dataset will only contain these six fields that were requested by the stakeholders.

Since you included the GEOMETRY field in the selected fields, a map of the capital project points is visible.

The Update fields element is added to the canvas.

Next, you'll connect the Select fields and Update fields elements.

Next, you'll choose which fields you want to update and configure them. When updating fields, you can update their name and field type. You'll update three of the fields' names.

The first field you'll update is fmsid. This field originated from the Capital Project Tracker dataset and contains a project identification number.

The Select field window appears. You'll choose the field that you want to update.

Next, you'll provide an updated name for this field.

The first field has been updated. You'll update two more fields, the ntaname and BoroCD fields.

Next, you'll add more fields to update.

The three fields you set to update are listed.

In the table preview, the column headings have been updated. Your table's field headings are more intuitive for your stakeholders.

The Feature layer element is added to the canvas.

Next, you'll connect the Update fields and Feature layer elements.

Next, you'll configure the output settings for the feature layer that will be created. With ArcGIS Data Pipelines, it's also possible to have the output replace an existing feature layer or add and update features in an existing feature layer.

• Under Output settings, ensure that Output method is set to Create.
• For Output name, type DPR Capital Projects.

What you see in the preview window is what will be written to your feature layer when you run the data pipeline.

Your data pipeline is complete.

If your data pipeline's elements are disorganized, the Auto layout diagram button repositions elements to better see the flow of inputs, tools, and outputs.

The elements on the canvas are repositioned.

The Latest run details window appears and opens to the Run details tab. This window provides you with information as the data pipeline runs. It also displays any warnings or errors that occur during processing.

After the data pipeline completes, you'll explore your results. Processing takes about a minute.

This tab lists any outputs created by the data pipeline. The DPR Capital Projects feature layer is listed.

Next, you'll review your feature layer's item details and share the feature layer with your organization.

A browser tab opens to the DPR Capital Projects item details page.

This page provides information about the feature layer created by the data pipeline. Next, you'll share the results with your organization.

The Share window appears.

The DPR Capital Projects layer is now shared with your organization for others to access. When the data pipeline runs, it'll update this feature layer for anyone who adds it to the maps or apps.

Next, you'll view the result on a map.

A map opens and the DPR Capital Projects feature layer is added.

A pop-up appears with the attributes that you specified in the data pipeline.

This dataset is available to be symbolized, analyzed, and configured further for your stakeholders' web maps and apps.

The Feature layer parameter appears. This parameter tells the data pipeline which feature layer in your organization to replace when the data pipeline runs in the future.

Now, when the data pipeline runs again in the future, it will overwrite the existing feature layer data and avoid errors.

Now, you'll create a task. A task allows you to control how frequently your data pipeline runs.

The Create task window appears. Here, you'll choose the data pipeline that you created.

You'll schedule your data pipeline to run automatically, allowing you to saturate the target feature layer with the latest information. In a production environment, you might set this to run monthly, daily, or more frequently based on how often your input datasets update.

First, you'll give this task a title.

You'll have this data pipeline run every 15 minutes.

Next, you'll ensure that the data pipeline only runs once.

The task is visible and informs you when it will run next.

After the task runs, you can see the task run history.

The Task runs pane shows the task and the status of completed runs. A green check mark indicates that the run succeeded. A red hexagon indicates that the run failed.

Under Output results, an overview of the results from the data pipeline is shown.

You can see that the data was replaced as you specified.