Sand Mining in Western Wisconsin: Data Collection

Introduction:

Once a research question has been proposed the next concrete step is to collect data. To understand the impacts that frac sand mining has on surrounding areas it was required to find and acquire datasets from numerous government sources. This data then had to be put through a sting of tools (see “Python Scripting Two” below) to normalize and prepare for use. All the data sets were acquired for use in Trempealeau County Wisconsin.

Methods:

Figure one. Information databases typically come in a compressed file format (zipped). This saves memory space and make for quicker downloads. The above image shows and extraction program in the process of decompressing zipped files.

The first objective of this exercise was to actually find and download data for Trempealeau County. There were four different government websites visited for six datasets (Figure five). One had to find each particular dataset by selecting different parameters before downloading. All the datasets were initially downloaded as zip files and required extraction before use (Figure one).

Figure two. Naming folders appropriately helps a researcher to recall where information is held.

 The compressed files were extracted into labeled folders in a permanent drive. Folder names reflect the source and subject of each dataset. This helps to keep files from being lost or misinterpreted.

    This project focuses on the early steps of manipulating data. This involved the fine process of changing file types, reorganization, examining feature attribute tables. Ultimately this led to making relationships among individual feature classes with the use primary and foreign keys.

Figure three. Results of clipping a rail network via county borders. Rails that exist within county in red. Rails in green are not needed for the area of interest so removing them is a good practice in data management. 

Data management is a cornerstone for any well-organized project. Having files that cover more than the necessary ground is a waste of space, clipping features removes information and preserves what is important for the project. See figure three for an example. A somewhat contradictive issue in regards to file size also became apparent when downloading data for this project. The digital elevation model (DEM) downloaded from the USGS did not cover the entirety of the county. There was one small section on the county’s south point that was omitted meaning that a second DEM needed to be downloaded (See figure four). An image mosaic needed to be completed between these two raster files. The output file was in unsigned 16 bit format and only had one radiometric band which is not an issue as this file only had one band to start out with.  

Figure four.  Digital elevation data comes as a raster image that covers a large area. In the case of Trempealeau County two images (blue tint and red tint) are needed for full coverage. 

Once all the information was downloaded I went on a search through each datasets respective metadata file to assess their reliability. Observe figure five for results.

Figure five. Metadata accuracy information for downloaded datasets.

Conclusions:

All data has limitations and knowing what they are is key to its accurate representation. The datasets that were downloaded range in spatial and temporal accuracy and are a blend between vector and raster type files. For a large scale map that depicts a village or town sized area these could lead to issues in regards to accurate representation. The spatial resolution is far too course and changes over time are far more pronounced meaning the datasets that haven’t been updated lately would be a key concern. This project focusses on Trempealeau County, an area with a relatively large scale, meaning that the precision is not as imperative.

Python Scripting Two: Batch Processing

   As mentioned in the last blog, Python scripting for ArcGis can be used to automate tedious and repetitive tasks. In the below example (figure one) a “for loop” was used to name, locate, project, extract (the vector equivalent of this tool would be “clip”) and copy to a different geodatabase.

Figure one. Python coding used to batch process a raster dataset. 

 While creating this code I experienced several setbacks and many error messages, most of which were due to my own inexperience with coding. The data set I was working with only totaled three raster files meaning that it would have been much faster for me to have run each raster individually.  But it is the principle of the matter which is important and making a batch data processing code is immensely important for a dataset that totals many more files than the one used for this exercise.

Results:

Figure two. Elevation data raster for Trempealeau County WI. 

Figure three. Land use in Trempealeau County (too many specific types of land use to fit in legend, see figure four for simplified labels). 

Figure four. Generalized land use map for Trempealeau County.

Python Scripting One: An Introduciton to Coding for ArcMap

As a basis for future GIS work having a basic understanding of the coding language called Python as it relates to ArcGIS is indispensable.  This post is collective of some introductory exercises that were completed for lab. Python is a coding language used in ArcGIS as a way to complete tasks which would otherwise be very difficult or time consuming if using the more conventional functions of ArcGIS programs. While Python scripting has a much steeper learning curve relative to using GIS tools individually and using a model builder, it is not limited in the same ways. For example, one can create a script to automate complex yet tedious and repetitive tasks. 

Figure one: Collective demos for python scripting. 

There were three different sections for the introductory Python scripting material.  The first section was simply running ESRI ArcMap tools with a script. One will notice this as the green, pound signed block located roughly in the middle of the figure one (it was pound signed so as to speed up diagnostics for later operations). A buffer tool was run shortly before a clip tool.

The second part is listed above the clip and buffer tools is where variables are being defined so the tools can be used for multiple datasets without being rewritten for each time. One only needs to redefine the set parameters for each dataset making it fairly easy to adapt from one place to another.

The third and final section of this short introduction to Python was coding to automatically fetch and display file metadata such as projection. A secondary line of code was used to list raster files in a geodatabase and display the file name and spatial reference. In figure one this is expressed below the green buffer and clip tool code.