Deep learning in Raster Analysis

ArcGIS Image Server allows you to use statistical or machine learning classification methods to classify remote sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Image Server for object detection and image classification. Training samples of features or objects of interest are generated in ArcGIS Pro with classification training sample manager tools, labeled using the Label Objects for Deep Learning tool, and converted to a format for use in the deep learning framework. These training samples are used to train the model using Train Deep Learning Model raster analysis tool or ArcGIS API for Python. The trained model, together with the model definition file (and optionally a Python raster function script file), is packed and shared as a deep learning package (.dlpk) item that is then used to run the raster analysis inference tools. The inference tools either extract specific features or classify the pixels in the imagery. The .dlpk item can be used multiple times as inputs to the raster analysis tools, allowing you to assess multiple images over different locations and time periods, after the model has been trained.

	Create training samples with the Label Objects for Deep Learning tool in ArcGIS Pro and use the Export Training Data for Deep Learning raster analysis tool in ArcGIS REST API or ArcGIS API for Python to prepare the data for the data scientist.
	Train a deep learning using the Train Deep Learning Model raster analysis tool. You may also use the ArcGIS REST API to train the model with a deep learning framework. The output deep learning package (*.dlpk), consisting of the model definition file and the trained deep learning model or Python script, is shared to your portal as a .dlpk item.
	The data scientist delivers a deep learning package, consisting of the model definition file and the trained model or Python script, back to the ArcGIS Image Server raster analysis environment. With the .dlpk item on portal, run the Detect Objects Using Deep Learning tool, or the Classify Pixels Using Deep Learning tool, or the Classify Objects Using Deep Learning tool to generate an output.

Features and capabilities

Deep learning raster analysis tools allow you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks for image classification and object detection.
• Use a deep learning model to either classify image pixels or detect or classify objects such as airplanes, trees, vehicles, water bodies, buildings, and oil well pads.
• Integrate external deep learning model frameworks, including PyTorch, Keras, and TensorFlow.
• Process a collection of rasters during model inferencing to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing detected objects to be used for additional analysis or workflows.
• The deep learning inference tools can use the CPU for distributed processing, or use the powerful GPUs on each server node if available.
• You can use multiple GPUs per server node to run deep learning inference job with the model configurations predefined in ArcGIS, including TensorFlow Object Detection API and DeepLab, Keras MaskRCNN, PyTorch fastai Single Shot Detector.
• If the input raster comes with the appropriate map space to image space transform, training image chips can be exported in image space in Export Training Data For Deep Learning.

Get started with deep learning

The creation and export of training samples are done within ArcGIS by a knowledgeable image analyst because they require strong knowledge of image classification workflows. The deep learning model can be trained in ArcGIS using the Train Deep Learning Model raster analysis tool or ArcGIS API for Python arcgis.learn. Alternatively, the deep learning model can be trained outside ArcGIS using a third-party deep learning API. Once the model is trained, the .dlpk item is added to the Portal for ArcGIS and ready for model inference tasks. Run the raster analysis tools to detect and classify objects or classify pixels from Map Viewer, ArcGIS API for Python, ArcGIS REST API, or ArcGIS Pro. The entire deep learning workflow can be completed by one analyst that has experience with deep learning models and ArcGIS image classification.

A separate deep learning setup application is also provided, which will install the proper deep learning framework APIs (TensorFlow, Keras, PyTorch and all other required Python libraries) in the ArcGIS Image Server default arcgispro-py3 environment. It is the prerequisite to run deep learning tasks on ArcGIS Image Server.

Refer to the help documentation for help configuring and deploying ArcGIS Enterprise for raster analytics. See the additional steps for configuring your Raster Analysis server to use multiple GPUs per server node.

1. Create and export training samples.
   1. Use the Label Objects For Deep Learning tool in ArcGIS Pro to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning raster analysis tool to convert the source imagery and training samples into deep learning training data. The source imagery may be an image service. The output of the tool is the data store string where the output image chips, labels and metadata files will be stored. An additional output of the tool is a template .emd file to be populated by the data scientist.
2. Train the deep learning model.
   1. Use the image chips generated in step 1 above to train a model using the Train Deep Learning Model raster analysis tool. Alternatively, the users can train deep learning models using the ArcGIS API for Python arcgis.learn, or third-party deep learning framework APIs.
   2. If the deep learning model is trained using Train Deep Learning Model raster analysis tool, the output will be a deep learning model package item published on the portal. The deep learning model package (.dlpk) can be directly consumed in the next step.
   3. If a third-party deep learning framework API is used to train the model, users need to update the .emd file with the name of the deep learning framework, the model configuration, other optional properties, and optionally a custom Python raster function for model inference. For information about required and optional parameters in the .emd file, see the Esri model definition file section below.
3. Run the inference raster analysis tools from Map Viewer, ArcGIS API for Python, ArcGIS REST API, or ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning, Classify Objects Using Deep Learning, or the Classify Pixels Using Deep Learning raster analysis tools to process your imagery. If your trained model incorporated custom Python raster functions with additional variables such as padding or confidence threshold for fine-tuning the sensitivity, these variables will appear on the ArcGIS Pro or Map Viewer raster analysis tool's dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function.

      The output of the Detect Objects Using Deep Learning tool, or the Classify Objects Using Deep Learning tool, is a feature class showing the objects detected or classified by the model, and the Classify Pixels Using Deep Learning tool outputs a classified raster.

      Note:

      The deep learning model framework must be installed on the Raster Analysis service machines to run the inference tools because the inference logic is built into the framework Python API and is required for computation.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model. There are required and optional parameters in the file as described in the table below.

Model definition file parameter	Explanation
Framework	The name of a deep learning framework used to train your model. The following deep learning frameworks are supported: TensorFlow Keras PyTorch InferenceFunction
ModelConfiguration	The name of the model configuration. The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. There are existing open source deep learning workflows that define standard input and output configuration and inferencing logic. ArcGIS supports a set of predefined configurations: TensorFlow ObjectDetectionAPI DeepLab Keras MaskRCNN KerasClassifier If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you will need to describe the inputs and outputs in the .emd file or in the custom Python file, in full.
ModelFile	The path to a trained deep learning model file. Many model framework file formats are based on Protocol Buffers, so the trained deep learning model is a .pb file, but other file formats are supported.
ModelType	The type of model. The type can be one of the following: ImageClassification—For classifying pixels ObjectDetection—For detecting objects or features ObjectClassification—For classifying objects or features
InferenceFunction (Optional)	The path of the inference function. An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS deep learning raster analysis tools: Detect Objects for TensorFlow Classify Pixels for Tensor Flow Detect Objects for Keras Detect Objects for PyTorch Classify Objects for PyTorch If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required together with the trained model. In this case, set InferenceFunction to the Python module path.
ImageHeight (Optional)	The number of rows in the image being classified or processed.
ImageWidth (Optional)	The number of columns in the image being classified or processed.
ExtractBands (Optional)	The band indexes or band names to extract from the input imagery.
Classes (Optional)	Information about the output class categories or objects.

Below is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

Deep learning package (.dlpk) item

Deep learning raster analysis tools require a deep learning model package (.dlpk) as input. A deep learning model package is composed of the Esri model definition JSON file (.emd), the deep learning binary model file, and optionally, the Python raster function to be used.

When you have all components ready, you can compress all files into a .zip file, and upload the .zip file to your portal as a .dlpk item. For more details, please refer to Add files from your computer. You can also change your .zip file extension to .dlpk so that when adding item, the .dlpk item type will be automatically detected.

Note:

You must compress all files included in the deep learning model package at the root folder level.