Module 4- Coastal Flooding- Applications in GIS

 Increased occurrence of coastal flooding and sea level rise are two effects of climate change that threaten coastal cities. This week we performed a coastal flooding assessment to compare the rate of damage before and after Hurricane Sandy hit the New Jersey coastline. Then we used elevation models to delineate coastal flood zones in Naples, Florida.

For our hurricane damage assessment, we subtracted the pre-Sandy raster from the post-Sandy raster. It was very clear when inspecting the resulting rate of change raster what areas had the greatest concentration of damage. We then compared the resulting image to a building footprint from 2019 that showed what buildings were present at that time.

After learning how to use the reclassify and region grouping tools to create a predictive map of a storm surge, we created a map using two elevation models, a USGS DEM and a LiDAR derived DEM to predict areas and buildings effected by a 1-meter storm surge. We then analyzed the type of buildings affected and, presuming the LiDAR data was more accurate the errors of omission and errors of commission associated with the data.

Module 3- Visibility Analysis- Applications in GIS

    This week’s lesson centered around 3D visualization techniques in ArcGIS as well as performing line of sight and viewshed analyses. Before working with 3D visualization there are several important core concepts to understand.

     We covered how to determine which of the 3 elevation types to utilize and how to set or change elevation type in ArcGIS Pro. The 3 elevation types are:

    - on the ground (typically used for vegetation or buildings)

                - relative to the ground (traffic cameras, subsurface features)

                -an absolute height (airplane paths, satellites)

    Other important terms are:

        -  cartographic offset, or adjusting the height of an entire layer so that the features are not obscured by others layers within the scene. In our case the houses were obscured by the trees, so we set the cartographic offset to 50 to make them visible.

        - vertical exaggeration, emphasizing vertical features on a map, particularly useful in cases where the horizontal extent is vast.

                          This image gives and examples of cartographic offset. The red dots represent houses in the  neighborhood. However, without sufficient cartographic offset some of the houses were obscured. 50 feet was sufficient to see all the houses clearly.

    We covered the process of extruding data. Extruding data from a 2D to a 3D feature can be a way to visualize more than just the literal dimensions of an object. For our exercise we extruded building using their absolute value in order to visualize the most valuable properties in a city.

For this map the yellow symbolizes residential properties, the pink represents commercial properties and the gray represents public housing. The height of the buildings isn't their actual height but instead representative of their absolute property value.

  Additionally we covered how to apply visual effects using symbology, and setting illumination properties for global and local scenes to see the effects of shadows and illumination on the scene. 

    After we covered introductory 3D concepts we performed both a line of sight analysis and a viewshed analysis. 

    The 3 main steps to performing a sight line analysis are to determine observers and targets, construct sight lines, and determine lines of sight. We used ArcGIS 3D analyst tools to complete these steps. We considered line of sight from 2 observation points along a parade route we considered only those with visibility within 600ft which left clear gaps in coverage from a security standpoint.

                        The centermost portion of the parade route was completely absent of coverage with the line of  sight analysis at 600ft.

For our viewshed analysis we assessed a proposed campground lighting system to determine what height was necessary to obtain sufficient campsite coverage.

Viewshed analysis photos assessing a proposed campground lighting system. Ideal coverage was more than 2 observers.

    

    Finally we covered how to create a site-specific 3D scene in ArcGIS Pro from 3D datasets. One important tip I took note of in this section is that when you are extruding features from 2D to 3D symbology (such as a building), one way to ensure accuracy is to extrude the features at an absolute height, and then convert the newly extruded features to multipatch features before switching the elevation type to "on the ground". Drawing at an absolute height ensures the slope of the terrain doesn't interfere with the rendering of the feature, and if you change the elevation type before converting the data to a new feature class it will not keep the features as they were drawn at an absolute height but will re-draw them at ground level. I also found it worth noting that extruding the features does not automatically make a new, shareable 3D feature class.

    Overall I feel like this exercise gave me new insight into what 3D tools and options are available to me with ArcGIS Pro.

-   

- 

Module 2- Applications in GIS- Forestry and LiDAR

    This week we worked with Virginia LiDAR data from Shenandoah National Park to create a DEM and DSM, forest height and forest biomass (canopy density) layers. I found the LiDAR data a bit tricky. The layers themselves flowed smoothly but, when viewing in 2D, I never could get my LiDAR data to populate on my map without zooming in so far that it rendered the visual pointless. Regardless, I was able to work around this by creating a 3D map with the LiDAR layer, the 3D model had less trouble populating than the 2D although I still couldn't view it zoomed out as much as I needed to for my final map.

We used the Point File Information tool to summarize the contents of the LiDAR layer and then created a DEM layer using the LAS Dataset to Raster tool with ground points. We created a DSM layer using Non-Ground Points. For the heigh layer we subtracted the DEM from the DSM using the minus tool.

To calculate biomass density, we used the LAS to multipoint tool and created a layer representing the ground and a layer representing the vegetation. We converted these layers to rasters with a “count” assignment. We used the IS NULL tool to create a binary file for each layer, then use the Con tool on these created layers to accept as true if value of 0 is encountered and to pull a value of 1 from the original raster to pull the ground and vegetation layers separately.

We used the Plus tool to combine these layers and the Float tool to transform the information from an integer to a float. Finally, we calculated density using the Divide tool to compare the vegetation raster that was created using the con tool with the float data. I realized ¾ of the way through this process that this would have been a great time to have used the Model Builder we learned about in GIS Programming last term, but since I had already created so many files it had become counterproductive to use it at this point. If I was doing these calculations frequently, I would create a model to use to make the process more efficient and leave less room for error. Eventually I had so many features I was working with, and I was trying to remember which one was from which step even with intentional naming patterns. A model would have been very useful.

Finally, we created a histogram chart to display the height data created earlier in the lab. The exercise had said negative values were in error, but I was having a hard time finding how to exclude the handful of negative data points. For our vector data we excluded the point in the symbology pane, but this wouldn’t work for the raster data set. Eventually I discovered a button called “Selection” which I could toggle on and off that, when combined with a SQL clause, allowed me to exclude data as needed. I used this to remove all data points below 0 from my histogram.

Here are 3 maps that I created from some of the many layers I created in this process.

Ideally I would have zoomed out a bit more for this lidar image but I this was the farthest I could zoom out without it switching to only showing the extent lines.

         

 

Module 1- Applications in GIS- Crime Analysis

 A new term, a new course. We kicked off the Applications in GIS course learning about how GIS can be used in crime analysis. We made a choropleth map and a kernel density map depicting 2018 burglary rates in Washington DC. Then we created 3 hotspot maps (Grid Overlay, Kernel Density and Local Moran’s I) using 2017 homicide data for Chicago. We determined which analysis method was a better predictor of areas at higher risk of future homicide when compared to 2018 Chicago homicide data.

Crime density was highest within the Grid Overlay map, and lowest within the Local Moran’s I map. There was a correlation between crime density and total area of the of the hotspot, with Grid Overlay having the lowest total area since we set our parameters to consider only the top 20% of crime areas.

I did make an obvious mistake when I initially created my Grid Overlay map. I calculated the top 20% manually since there isn’t a tool for this task. There was no way to filter by attribute for this task either, but initially I used the select by attributes option and selected by objectID the last 62 features. I realized when I was completing my write up that even though I had sorted by homicide count in my attribute table, the object ID was a standard and would not have changed when I chose to sort by homicide count. There isn’t an option to select by the field number on the far left which would have been an accurate choice. I suppose I could have added a new field and numbered them that way but it was quicker to just manually select from 249 on towards the end after they were already sorted by homicide count. This was an obvious error, and I wasn’t sure how I overlooked it but fortunately I caught it because it would have greatly skewed my results. My initial (wrong) Grid Overlay had a significantly lower density than the other two analysis methods. One fact to consider when running these analyses is that the more manual tasks are involved in an analysis method, the more room there is for error.

Here are my 3 hotspot maps for the city of Chicago.

Module 6- Working With Geometries- GISProgramming

 This week we focused on working with geometries: reading geometries, working with multipart features and writing geometries. We got to work with the search cursor more but this time we worked with tuples which I found a bit more complicated when trying to print my script.

 

We worked with a rivers shapefile for this lab. Our first task was to create/open a new empty txt file that could be written in. The goal of this lab was to populate this text file with the name, x, y, coordinates, OID and to number the vertices for each row of the features in the rivers class.

We began with a search cursor. Initially when setting up my cursor I was not putting the search variables in the correct order. I put “NAME” first and when I tried to run my script I kept getting an integer error that cleared when I moved it to it’s proper order in the cursor.

 Once I got this sorted, I kept getting errors when trying to print the “NAME”. I tried using the getValue method that we used in the last lab to no avail. I kept getting an error that the “tuple” object has no attribute “getValue”. I also tried setting the NAME as a variable, like I did when I had issues in the last lab, but it wasn’t successful. I looked in the exercises we worked on before this assignment, but we never printed off any non-numerical variables, so I wasn’t sure how to approach this. I knew it was a tuple and I noticed as I reviewed last week’s script that last week’s search cursor was not so I tried variations on ways that can call a tuple value, but they didn’t work. Finally,  I referenced the documentation https://pro.arcgis.com/en/pro-app/latest/arcpy/data-access/searchcursor-class.htm and was able to print using the f’({})’ option they showed.

 

The goal was to populate a txt file, but to ensure I was printing the information properly without having to open the file each time I ran a matching print statement after my writing function. I focused on getting my print statement to work correctly while leaving the writing function commented out until I was ready to try to run it. When I finally got my print statement to print appropriately, I tried to move it to my txt file. I simply copied the entire parenthesis part of the statement and put it after my write to file function. However, I received an error that only one argument could be added. I was able to solve this by concatenating the individual statements as opposed to using a comma. After doing this I was able to run the script to add all the values that way, but my file was still blank when I checked to see if it filled in the correct data. I re-read the chapter on this and realized I left out my file “close” statement. I added the close statement, but I added it within the “for” loop instead of outside of it. This led to a value error that was resolved when I moved it outside of the loop. My txt file finally(!) populated but at this point it looked like it didn’t have the OID and Vertex numbers. I tried several options including str() before I realized it WAS giving me the first two numbers, but it was leaving out the space between them and the coordinates. I was easily able to resolve this by adding an extra space between each item I wanted in the write to file. I did not need this space for the print statement. It was very rewarding to see my populated text file.

 Here is a flowchart for my script along with an image of the resulting txt file.

                                    

Module 5- Exploring and Manipulating Data- GISProgramming

 This week we learned how to check for, describe and list data, create and populate a geodatabase with existing data in other folder, how to use a SQL statement within a python script using cursors, we reviewed lists and dictionaries and populated a dictionary from a selected feature class using a cursor.

Create and populate a new geodatabase

We began by creating a new geodatabase in our results folder and populating it with shapefile data provided in our data folder.  During this step I ran into some mistakes with my files early on. For both the creation of my geodatabase and the next step of populating my geodatabase I left out a “/” in my output. This led to me initially create a geodatabase named “resultsbnl26” in my module 5 folder as opposed to a geodatabase named bnl26 in my results folder. After I corrected this error, while populating my geodatabase I wound up creating a single shapefile named bnl26 to my results folder and nothing in my geodatabase folder. I realized quickly just how important that slash is.

Create a Search Cursor

We then were tasked to create a Search Cursor for the cities layer and to retrieve and print the name, population and feature of all cities that were listed as a “County Seat”. I created my cursor, ran my “for” loop and immediately had no issues printing my “name” statement but I ran into several snags trying to print the population. I kept getting an error that the program couldn’t concatenate an integer, so I tried converting it to a string, but I was not able to. I spent awhile trouble shooting and trying different methods of formatting my print statement but finally I had to set a separate variable to get the value of the population and then set that variable as a string instead of putting the getValue method within the print function. During this troubleshooting process I had a moment where I falsely thought I had figured out the population issue when, instead of an error, I got a syntax issue on the next line that wouldn’t allow my code to begin to run. I spent awhile wrongly troubleshooting that line (which was my feature print statement) until I finally removed it altogether to see if that fixed the issue. It simply moved the syntax error to the next line so at that point I knew that I still had an error in the previous line with my population print statement. This was a good reminder that sometimes the errors are not what or where we think they are.

During this section I also struggled to figure out how to use the newline character outside of a string since my previous experience with it had been inside a string and most of the options I was seeing online were within a string. I finally realized I needed to “+” it within the print statement rather than trying to tack it onto the end of my variable.

Create and populated a dictionary from feature class data

Our final task was to create and populate a dictionary that contained the names and population of each county seat. This task was my biggest struggle since we had only done the required tasks separately so far. I created my blank dictionary and I knew I needed a “for” loop and to somehow use the search cursor again. For some reason I tried to set my “for” loop as “for county in county_seats” (county_seats was my dictionary), and had a second “for” loop for the search cursor within the first for loop but realized that the first for loop was unnecessary. I knew how to update the dictionary but I kept accidentally populating my dictionary with [NAME : Pop_2000] (the titles of the rows) instead of getting the individual features names. At this point I realized I needed to add the getValue method to the line of code somehow, but I kept trying to add it within the “update” dictionary statement. I realized that I was making the same mistake I did earlier during the previous task and that I should set key and value variables that used the getValue method and then use those variables within the dictionary and it populated smoothly after that. I did run into a repeat of an issue I had earlier in the term where instead of printing off my final list it printed each iteration. I accidentally printed each iteration of my dictionary as it ran the “for” loop. This was easily corrected by adjusting the indent.

After I had completed all my tasks I went in and added the necessary print statements and retrieved messages for each task.

Here are the many screenshots of my final output as well as the flowchart I created based on my script.

 

Final thoughts

It was amazing to see the progress we have made in such a short time. I have really enjoyed learning to code, and I am disappointed that we are coming to the end of the term (only one week left). I am actively looking for ways to get more practice with these tools and I am hoping I can utilize them in my next few classes. One issue I am running into is that since it is all so new to me, I keep forgetting what options are even available to me. Another issue is discerning what variables to set for each task but this week I felt a lot more confident in my ability to “figure it out” when I hit a snag. I noticed that when I stepped away for a bit and came back to the task at hand fresh, I was able to solve my issues more quickly. This is a weakness of mine because I get very hyper focused on things I perceive as a puzzle and I don’t want to step away, but I am actively choosing to find a task I can do that forces me to take my mind off the “puzzle” of whatever error I am struggling with so that I can come back and see the code more clearly.

 

Module 4- GIS Programming- Geoprocessing

 This week we were tasked with using the coding skills we've gained over the past few lessons to run various geoprocessing tools in ArcGIS Pro. We used both the Model Builder feature and ran additional geoprocessing tools in Notebook. I love the functionality of the Model Builder and I could see how this would have been useful previously. I remember running a lot of individual tools during my Introduction to GIS final project and this would have more easily streamlined it (and possibly could have reduced some of unnecessary geoprocessing I did blundering around on my own for the first time. As someone who prefer visual context for tasks (and isn't GIS all about providing visuals?) this particular feature is very appealing and I look forward to using it more.

We built a simple model that clipped a provided soil shapefile into a specific basin area, then erased the areas that were classified as "Not prime farmland". This created the following output:

Our second task, in Notebook, was to add XY point data to a provided hospital shapefile, and then to run buffer analysis with the dissolve feature added. Although we worked through buffer analysis during the exercises, we had not added XY data to a shapefile before. We were told to reference the ArcGIS help pages for this task. I appreciated this because it gave me practice referencing online materials for solving new-to-me coding problems. I realize that after this class I will be constantly learning and will often need to reference internet resources in this manner. Even with the reference page I still find myself overanalyzing and second guessing (and therefore making mistakes) with my script. This seems to be a consistent theme for me in this class since I want to understand what each piece of the script is doing but with patience and practice I am understanding more and more. 

We are not supposed to share our actual codes on our blog for obvious reasons but here is a screenshot of my output in Notebook for this exercise:

And after my code was run it nicely populated directly onto my map as pictured: