Advanced Python Scripting for ArcGIS Pro

Chapter 1

Introducing advanced Python scripting

1.1 Introduction

Python has become one of the most widely used programming languages, and this increase includes geospatial applications. Python is employed for many different tasks, from automating data processing using desktop software, to web scraping for downloading structured data, to developing machine-learning algorithms for classifying imagery hosted in the cloud. Python is a versatile, open-source programming language supported on different platforms. These features contribute to its growing popularity in the geospatial community. Python is also the preferred scripting language for working with ArcGIS Pro.

This book represents the logical follow-up to Python Scripting for ArcGIS Pro, also published by Esri Press (2020), which introduces the fundamentals of Python and teaches you how to write basic scripts to automate workflows. Advanced Python Scripting for ArcGIS Pro picks up where Python Scripting for ArcGIS Pro left off by focusing on more advanced scripting techniques and the development of tools and notebooks to be shared with others. This book also includes working with third-party packages and the ArcGIS API for Python, which opens new and exciting possibilities to use Python for geospatial applications.

This book is written for ArcGIS Pro and Python 3. The topics covered in this book require substantial previous experience in writing Python scripts for ArcGIS. The fundamentals of Python and ArcPy, including setting up a Python editor and writing basic scripts for data processing using ArcPy, are covered in Python Scripting for ArcGIS Pro.

1.2 Python scripting in ArcGIS Pro using ArcPy

ArcGIS Pro provides support for the use of Python as a scripting language, including the ArcPy package installed as part of ArcGIS Pro. ArcPy provides access to all the tools available in ArcGIS Pro, including those that are part of ArcGIS Pro extensions. This feature makes Python scripting an attractive and efficient method for automating tasks in ArcGIS Pro.

Python scripting has become a fundamental tool for GIS professionals to extend the functionality of ArcGIS Pro and automate workflows. Python is the scripting language of choice to work with ArcGIS Pro and is included in every ArcGIS Pro installation. Python is also directly embedded in many tools in ArcGIS Pro. For example, Python is one of the standard expression types for field calculations. As another example, several geoprocessing tools in ArcGIS Pro consist of Python scripts, even though the casual user does not necessarily notice it (or need to).

One of the goals for using the current book is to learn how to develop new geoprocessing tools that expand the functionality of ArcGIS Pro but that look and feel like regular tools that are part of the software. This is accomplished using Python script tools and Python toolboxes. A secondary goal is to become familiar with the ArcGIS API for Python to expand the use of Python to working with web GIS. This is accomplished using notebooks.

1.3 Python versions and ArcGIS

Compared with other programming languages, Python has gone through a limited number of versions, reflecting a philosophy of incremental change and backward compatibility. Python 3 was released in 2008 as a major overhaul, with the primary goal to clean up the code base and remove redundancy. The most recent version, at the time of writing, is 3.8, with 3.9 under development.

Some of the changes in Python 3 are fundamental, which result in breaking with the backward compatibility philosophy of Python. As a result, not all code written in Python 3 works in Python 2. Some of the new functionality added in Python 3 was also added to Python 2, a process known as backporting. With careful attention to detail, it is therefore possible to write code that works in both versions.

The two versions of Python will continue to coexist for some time, but officially Python 2 will no longer be maintained past 2020. This means that any existing code will continue to work, but there will be no further improvements to version 2.

ArcGIS Desktop 10.x uses Python 2 whereas ArcGIS Pro uses Python 3, which has several implications. If you are going to write scripts for both versions of ArcGIS or are planning to migrate scripts and tools from ArcGIS Desktop 10.x to ArcGIS Pro, you must learn some of the differences between the two versions of Python. Resources and utilities exist to assist with this conversion, which are covered in chapter 8.

The purpose of this book is to focus on writing scripts and developing tools for ArcGIS Pro using Python 3. Although Python code is not 100 percent backward compatible between versions 3 and 2, it is, in principle, possible to write Python code that works for both versions. However, because of fundamental differences between ArcGIS Desktop 10.x and ArcGIS Pro, many scripts and tools written for one version are unlikely to work in the other. Nonetheless, sometimes the differences are small, and strategies to identify and correct for these differences are covered in chapter 8.

Note: Many GIS users will continue to use both ArcGIS Desktop 10.x and ArcGIS Pro for some time. At the time of writing, the most current versions are ArcMap 10.7.1 and ArcGIS Pro 2.5. The installation of ArcMap 10.7.1 includes the installation of Python 2.7.16, and the installation of ArcGIS Pro 2.5 includes the installation of Python 3.6.9. These two versions can run on the same computer. When working with ArcGIS Pro 2.5, you should use only version 3.6.9.

1.4 ArcGIS API for Python and Jupyter Notebook

Python and ArcPy make it possible to extend the functionality of ArcGIS Pro using scripts and tools. ArcGIS Pro is a software application that runs on desktop computers and is primarily designed to work with local datasets. Increasingly, however, geospatial data and their applications reside on the web, referred to as web GIS. Web GIS is a type of distributed information system that allows you to store, manage, visualize, and analyze geographic data. ArcPy has limited functionality to work directly with web GIS. The ArcGIS API for Python is a different Python package from Esri to work directly with web GIS. This API complements the use of ArcPy for desktop GIS.

Code that uses the ArcGIS API for Python is typically written in Jupyter Notebook, an open-source web application that works like a Python editor and provides built-in visualization capabilities. Notebooks can also be used directly within ArcGIS Pro. Details on using the ArcGIS API for Python are covered in chapter 9.

1.5 The structure of this book

Advanced Python Scripting for ArcGIS Pro consists of nine chapters that focus on developing tools for ArcGIS Pro and writing more advanced scripts. Sample code is provided throughout the text.

Chapter 1 introduces Python scripting for ArcGIS Pro and illustrates several example scripts, tools, and notebooks that were developed using Python.

Chapter 2 demonstrates how to create custom functions and classes in Python. Custom functions and classes make it easier to organize more complex code and use parts of your code in multiple scripts. Custom functions and classes are widely used in script tools and Python toolboxes.

Chapter 3 explains how to create custom script tools, which make Python scripts available as regular geoprocessing tools with a familiar tool dialog box. Script tools are one of the preferred methods for sharing Python scripts with other users and make it easier to add a Python script as a tool to a larger sequence of operations.

Chapter 4 covers how to create Python toolboxes as an alternative to Python script tools. In a Python toolbox, the tool dialog box is written in Python itself, which is often more robust.

Chapter 5 outlines strategies for sharing tools with others, including how to organize your files, work with paths, and provide documentation for tools.

Chapter 6 covers the use of managing packages using conda. Packages allow you to add functionality to Python, and conda is a convenient way to install and manage these packages as well as Python environments, which control which packages are available.

Chapter 7 describes the use of selected built-in modules and third-party packages other than ArcPy, which can greatly enhance the functionality of your scripts. The modules and packages include ftplib, urllib, openpyxl, json, NumPy, Pandas, and Matplotlib.

Chapter 8 explains the key steps in migrating scripts and tools from ArcGIS Desktop 10.x to ArcGIS Pro, including the use of several utilities to facilitate this process.

Chapter 9 introduces ArcGIS API for Python, which makes it possible to use Python to work with web GIS. This chapter also introduces Jupyter Notebook as the preferred way to write and document Python code using this API. The resulting notebooks can also be shared with others.

1.6 A note about code in this book

Most of the code in this book is written for ArcGIS Pro 2.5, which uses Python 3.6.9. Most of the code will work in earlier versions of ArcGIS Pro, except for the most recently added functionality. As new functionality is added to future releases of ArcGIS Pro, the code in this book will continue to work for the foreseeable future. However, much of the code will not work in ArcGIS Desktop 10.x. Some of the code in this book also uses the ArcGIS API for Python version 1.7.0. This is the version that is installed with ArcGIS Pro 2.5, but the ArcGIS API for Python can also be installed separately. If installed separately, Python 3.5 is required to use the ArcGIS API for Python.

Note: The update cycle of the ArcGIS API for Python does not follow the same schedule as ArcGIS Pro. For example, at the time of writing, version 1.8.0 of the ArcGIS API for Python has been released, whereas ArcGIS Pro 2.5 installs with version 1.7.0. This version will be updated with future releases of the ArcGIS Pro software. The differences in these versions are typically small.

The code in this book employs the coding conventions of the official Style Guide for Python Code, also referred to as PEP 8. The complete style guide can be found at http://www.python.org/dev/peps/pep-0008/. Although not required, following coding guidelines improves the consistency and readability of your code.

1.7 Working with Python editors

Writing scripts and developing tools requires a Python editor. You are expected to be already familiar with using a Python editor and configuring it to use the correct environment. Details on working with Python editors are covered in Python Scripting for ArcGIS Pro.

The code is this book is not specific to one Python editor. IDLE is installed by default with every Python installation, and therefore most code illustrations in this book use IDLE as the Python editor of choice. Other recommended editors include PyCharm and Spyder, and some code illustrations use these editors as well. You are free to use the Python editor of your choice. Regardless of which editor is used for code illustrations, the Python code is the same for any Python editor. To use a Python editor with the code in this book, however, it must be configured to work with the default environment arcgispro-py3 or a cloned environment. Chapter 6 provides details on using conda to manage environments, but the configuration of Python editors is covered in Python Scripting for ArcGIS Pro.

You can also use the Python window in ArcGIS Pro to write and test Python code. However, the Python window is most suited to running short snippets of code for testing purposes. The more complicated and longer scripts developed in this book require the use of a dedicated Python editor, such as IDLE, Spyder, or PyCharm.

1.8 Exploring example scripts, tools, and notebooks

This section uses several examples to illustrate how Python is used to create scripts, tools, and notebooks. The examples were obtained from Esri and the ArcGIS user community. One of the reasons for presenting these examples is for you to become more familiar with looking at Python code and tools developed by others. One of the best ways to learn how to write code and develop tools is to work with existing examples. You are not expected to fully understand all the code at this point, but the examples will give you a flavor of what is to come in this book.

Example 1: Terrain Tools

The Terrain Tools were developed by Esri and extend what is available in ArcGIS Pro by providing capabilities for creating alternative terrain representations. These representations include different types of hillshade surfaces and contours, which can greatly enhance the cartographic display of terrain data.

The tools are made available as a collection of tools in a toolbox. Each tool consists of a tool dialog box and has a corresponding Python script. Although these scripts are written in Python, the functionality of the script can be accessed the same way as any other geoprocessing tools. The figure illustrates what the toolbox looks like in ArcGIS Pro.

The scroll icon indicates that these tools are written in Python, also referred to as Python script tools.

The tool dialog boxes look like those of regular geoprocessing tools in ArcGIS Pro. As an example, consider the Illuminated Contours tool. The tool provides an analytical version of the hand-drawn Tanaka method of symbolizing contours that includes coloring and varying the thickness of contour lines. Assuming a certain lighting direction, contours are drawn lighter on parts of the terrain that are illuminated and darker on parts of the terrain that are not illuminated.

The tool dialog box looks much like the regular Contour tool available in ArcGIS Pro.

The Illuminated Contours tool has five parameters, two of which are optional. The required parameters include the input raster, which is a digital elevation model or DEM, as well as the contour interval to be used and the output contour feature class. The optional parameters include the base contour and z-factor to be used. The result of the tool is a new polyline feature class, in which each contour is broken up into segments with new attributes for the color (grayscale, from white to black) and the appropriate thickness.

An example of the resulting illuminated contours is shown, with the contours overlaid on top of the regular DEM shown in grayscale from dark (low elevation) to light (high elevation). The assumed lighting direction is from the northwest, as revealed in the different levels of illumination of the contours.

The Illuminated Contours tool effectively carries out a series of steps, which can be accomplished by running regular geoprocessing tools and applying symbology. Some of these steps include creating contours from a DEM, creating a default hillshade, converting hillshade brightness values, reclassifying this grid of values into five-degree intervals, converting the reclassified grid to polygons, intersecting the contour polylines with these polygons, and assigning symbology on the basis of the new attributes of the polylines. The purpose of the script tool is to automate these steps and provide a user-friendly interface.

A single Python script is used in this tool, and the script can be opened to get an inside look at what the tool does. When you open the script in a Python editor, it looks like the figure.

When scrolling through the script, you will find the equivalent of the tasks you would need to carry out in ArcGIS Pro using existing tools. For example, making sure you have a license for the Spatial Analyst extension; running geoprocessing tools such as Hillshade, Contour, Reclassify, and Intersect; and applying symbology using a layer file. You could complete these steps using existing tools and a few manual manipulations, but the Python script tool includes all of them in a single easy-to-use tool dialog box.

One of the nice things about working with Python script tools is that you can view the underlying code. Not only can you learn from the code from others, you can also copy it and make a modified version of it for your own work.

Detailed documentation, all the source code, and example datasets to experiment with these tools can be found in ArcGIS Online at www.arcgis.com by searching for Terrain Tools Sample.

Example 2: Random Sample

The Random Sample tool was developed by the author and is discussed in more detail in chapter 3. This tool creates a random sample based on an input feature class and a user-defined number of features. The output is saved as a new feature class. The tool is created as a Python script tool. The tool dialog box is shown in the figure.

The tool provides functionality not available in ArcGIS Pro. Several online resources such as Stack Exchange (http://stackexchange.com) and Esri Support (http://support.esri.com) provide various code solutions to select features at random from a feature class, but employing these solutions requires substantial coding skills. By developing a Python script tool, the script becomes more user-friendly. As a Python developer, you can write this type of script, develop and test the Python script tool, and then make the tool available to other users who can use the tool without having to learn Python.

The Python script for this tool is shown in the figure.

The code for this tool is explained in detail in chapter 3, including the steps to develop and test the tool dialog box. The strategy for sharing this type of tool is covered in chapter 5. By the end of this book, you will be able to develop tools like this Python script tool.

Example 3: 3D Fences toolbox

The 3D Fences toolbox was developed by Esri’s Applications Prototype Lab. This toolbox makes it possible to create 3D fence diagrams on the basis of point data with a z-dimension field and at least one value field. An example application of this tool is to use sampling points of measurements of oil in seawater after an oil spill. Not only does each sampling point have an x,y coordinates, it also has a z-dimension (depth) and a measurement (oil concentration).

The tool creates a vertical subset of the 3D data—i.e., a slice—and transforms this subset onto a 2D plane. The value of interest, for example, oil concentration, is interpolated using Empirical Bayesian Kriging (EBK). The results are transformed as points into the original coordinate space as a fence. This transformation allows for a closer examination of 3D data, which is more difficult to do using the original point cloud of measurements.

This tool is relatively sophisticated, but it is also written entirely in Python. The tool is made available in two different versions in a Python toolbox as shown in the figure.

Details on Python script tools and Python toolboxes are covered in chapters 3 and 4, respectively. At this stage, it is enough to know that, in both cases, all the code is written in Python, and the tool dialog boxes look just like regular geoprocessing tools in ArcGIS Pro.

The tool dialog box has a lot of options for inputs, outputs, and analysis settings, reflecting the relatively complex nature of the interpolation. Like regular geoprocessing tools, some of the parameters have suggested defaults.

Some of the key parameters of the tool are the input point features with a z-dimension and a field for the measurement of interest, the interpolation settings, and the preexisting 2D linear features along which the fence will be created.

The output is a 3D point feature class. The following example (courtesy of the tool’s author) shows the results from the Feature-based Fences tool as a scene in ArcGIS Pro, with the original points used in the interpolation shown in red, and the resulting 3D fence as a color ramp.

The Python code associated with these tools is relatively long and complex, as could be expected for a sophisticated tool. The entire code is more than 1,000 lines, although the script also includes notes, comments, and blank lines to facilitate reading.

Even though the code appears complex at first, you probably recognize some existing tools, such as Copy Features, Add Field, and the Empirical Bayesian Kriging tool. The entire workflow is elaborate and would be cumbersome to complete step by step in ArcGIS Pro. Developing a tool of this complexity requires advanced coding skills and a significant time investment. Once created, however, the tool can be used many times, and it can be shared with other users.

Documentation and all the source code for this tool can be found in ArcGIS Online at www.arcgis.com by searching for 3D Fences toolbox.

Example 4: Notebook for wildfire analysis

Python and ArcPy make it possible to extend the functionality of ArcGIS Pro, as illustrated in the previous examples. The following example uses the ArcGIS API for Python to work with web GIS. The example is one of the sample notebooks provided with the documentation of the ArcGIS API for Python. The specific notebook illustrates an analysis of the Thomas Fire in 2017 in California. The figure shows the top portion of the notebook as part of the online documentation.

A notebook shows Python code combined with text, graphics, and other elements. As you run part of the code, the results update interactively. A notebook provides a different interface to working with Python code compared with more traditional Python editors, and it does not produce a tool for use in ArcGIS Pro with the familiar interface of a tool dialog box. Instead, users interact directly with the code and display the results within the notebook.

Notebooks can be opened directly in ArcGIS Pro. The figure shows the same notebook opened in ArcGIS Pro.

A user can inspect the code, update datasets or analysis parameters, and run the code to view the updated results within the notebook. The following example shows a side-by-side comparison of imagery before and after the wildfire. A user can enter a new address to be geocoded, run the code, and view the updated imagery.

You can also use the ArcGIS API for Python to perform many different types of analysis, similar to the geoprocessing tools in ArcGIS Pro. The figure shows an example of the use of map algebra to calculate a normalized burn ratio to determine the impacts of the fire by using the before-and-after imagery.

The results are symbolized and added to a map display in the notebook. The maps show the burned areas (in red) on top of a background satellite image.

This notebook can be found under the sample notebooks at https://developers.arcgis.com/python/sample-notebooks by searching for Thomas Fire.

Although a notebook does not have the familiar interface of a geoprocessing tool, it provides a more interactive approach to work with code and the results. Notebooks can be shared with others and hosted in an ArcGIS Enterprise portal. The use of notebooks makes it easier to document and share workflows. Creating notebooks using the ArcGIS API for Python is covered in detail in chapter 9.

You can benefit from these examples because they provide insight into why you would develop these tools and notebooks in the first place. Perhaps you have been using ArcGIS Pro for a while and have wondered why there is not a tool for a certain task. Or you have established a workflow that requires many repetitive tasks, and you are looking for a