Create GeocodeEvictions_DC2024.sas by RAGarMir · Pull Request #88 · NeighborhoodInfoDC/Requests

RAGarMir · 2024-10-22T20:29:44Z

I have hit a roadblock. I am unable to get the Data Step (from step 2) to create a dataset the macro can read. The code I used for the Data Step starts after "**Creates new data set with new street_address and zip_char variables **;" I get the following message when running macro from Step 3, "ERROR: [DC_mar_geocode] The input data set {EVICTIONS_ADDRESSES} does not exist or could not be opened.
the following error code when running the Macro from Step 3."

ptatian · 2024-10-22T20:36:59Z

Hi @RAGarMir The curly braces ( { } ) in the sample code were meant to indicate that you should replace what is there with the appropriate names. But you should not keep the { }. If you do that, I believe your code will run properly.

RAGarMir · 2024-10-31T15:57:27Z

@ptatian Looks like the program is running fine and I was also able to clean up all the addresses that did not match up. It should be ready for final review!!

ptatian · 2024-10-31T18:00:18Z

Thanks, @RAGarMir. I will review. But I need you to place a copy of the StreetAlt_GeoEvict_2024.txt file on SAS1, and your program should reference that copy. Otherwise, I will not be run the code.

And just to confirm, did you edit any of the addresses directly in the input data file (FY24DetailedData(records).csv)? Or are all the corrections done through the StreetAlt_GeoEvict_2024.txt file?

RAGarMir · 2024-10-31T18:45:28Z

Thanks, @RAGarMir. I will review. But I need you to place a copy of the StreetAlt_GeoEvict_2024.txt file on SAS1, and your program should reference that copy. Otherwise, I will not be run the code.
Where on SAS1 should I copy the file? Should it be the in the same Library and subdirectory location as in my local drive? What are the changes I should make to how the program is currently referenced to ensure the copy is also referenced?
And just to confirm, did you edit any of the addresses directly in the input data file (FY24DetailedData(records).csv)? Or are all the corrections done through the StreetAlt_GeoEvict_2024.txt file?
I made some changes to the input data file as well. It was mostly spelling mistakes for places like Minnesota Ave.

ptatian · 2024-10-31T19:16:57Z

Actually, I'm sorry. StreetAlt_GeoEvict_2024.txt is fine where it is!

Thank you.

Based on review

ptatian

Hi @RAGarMir . Thanks for doing this. You did a good job of implementing the macro according to the instructions. I added to your code some things that you would not have known to do, but I included notes in the review to point out those changes so that you can pick up some new lessons from them. Also, there are a few changes below that I'd like to ask you to make.

Street name mispellings

It is preferrable to correct the street name mispellings in the data by using the Street_alt file, rather than editing the input data. For example,

WARNING: [DC_mar_geocode] Street not found: HAWTHORONE ( street_address=86 Hawthorone Court #86 / _N_=768 )

Can be addressed by adding this line to Street_alt:

HAWTHORONE,HAWTHORNE

Same for WLIMINGTON, 14H, etc. SInce it is all working, you don't need to change anything. I just mentioning for future reference.

Address matching

Other data entry problems do need to be addressed in the input data, like you have done. There is still one address that is not being read in properly because of a data entry error.

WARNING: [DC_mar_geocode] Street not found: 19TH STREETSE ( street_address=1701 19th StreetSE #203 / _N_=1548 )

The quadrant SE got added to the street name. So if you would please fix that, that would be great.

This address also shows up in the "UNMATCHED OR NONEXACT ADDRESSES" list in the output. Scanning this list, you can see that 46 addresses do not geocode to the same exact address. For example, "231 37th Street SE" was geocoded to "232 37th St SE." This could be because of data entry error (for example, perhaps it should have been 231 37th PLACE SE, which is a valid address), or it could be a problem in our address file. Since we have no way of knowing for sure, we will just have to leave these addresses as is.

In any case, the approximate matches look like they are close to the entered one, so for things like summarizing data by wards or neighborhoods, we should be OK. If we want to match the data to parcel records, however, we will not want to use any of the non-exact matches, which would match to the wrong parcel.

Other data problems

In the frequency tables of DISPOSITION, one row has CANCELED spelled CANCELLED. Since we want categorical variable values to be consistent, we should correct this. Again, it is generally better to correct errors in the code rather than editing data files, which is less transparent. In addition, this problem may appear in future data files, so handling it code will be valuable. In the data evictions_addresses; data step, please add an IF/THEN statement to correct the mispelled value.

We also have 6 rows with missing values for YEAR, and 1 with YEAR=2025. The year is usually part of CASE_NUM, but for the 6 missing the CASE_NUM does not have a part that looks like a year. And for the 2025 one, the case number is 3278-25. So, not anything we can do about that.

BTW, the data do show that almost all these evictions (>98%) are cases from 2022 or later.

Please make the changes indicated above, and then tag me again on the pull request. Thank you!

cc: @lhendey

ptatian · 2024-11-01T20:43:54Z

+        replace;
+
+     getnames=yes;
+     guessingrows=max;


guessingrows=max; makes Proc Import look at the entire file when deciding on variable types. Without this statement, the code was creating street_number as a numeric var, but it contains some character elements (e.g., "5337/5341") that were being read as missing values.

ptatian · 2024-11-01T20:44:35Z

+run;
+
+proc print data=work.evictions (obs=10);
+run;


I added (obs=10) to limit the printed output to 10 rows.

ptatian · 2024-11-01T20:46:08Z

+**Creates new data set with new street_address and zip_char variables **;
+data evictions_addresses;
+	Set work.evictions (rename=(CASE__=Case_num)); 
+  street_address = catx( " ", street_number, street, type, quadrant );


Proc Import doesn't handle column names like "CASE #" ver well, so I used (rename=(CASE__=Case_num)) to rename the case number variable to a more natural name.

ptatian · 2024-11-01T20:47:30Z

+
+  label
+    Case_num = "Eviction case number"
+  ;


I used a label statement to a descriptive label to street_address and Case_num. Ideally, we want to label all variables in permanent data sets. Do we have a code book or something that describes what all the fields are? If so, we should use that to add labels for the remaining variables.

ptatian · 2024-11-01T20:48:55Z

+proc print data=work.evictions_addresses (obs=40);
+  id Case_num;
+  var street_address street_number street type quadrant zip_char;
+run;


I limited the rows printed with (obs=40) and also what colums are output using the ID and VAR statements, assuming that we were mainly interested in seeing the new street_address var and its components.

ptatian · 2024-11-01T20:52:30Z

+    out=Evictions_2024_dc,
+    outlib=Requests,
+    label="Scheduled and executed evictions, 2024, DC",
+    sortby=Case_num eviction_date,


I adjusted the output data set name and label. Since we will likely have more eviction data sets, the data set name should include the year.

ptatian · 2024-11-01T20:53:41Z

+    /** Metadata parameters **/
+    printobs=0,
+    freqvars=year DISPOSITION ward2022,
+    revisions=%str(New file.)


I supressed the printed listing of data (printobs=0,) and added frequency tables for three variables (freqvars=).

ptatian · 2024-11-01T20:54:50Z

+  )
+
+%Dup_check(
+  data=evictions_addresses,


I added this macro from our macro library to print out obs that have the same case numbers. As you can see, there are many and we need to be aware of that if we want to map or do other analysis with the data.

ptatian · 2024-11-01T20:57:30Z

+  informat _all_ ;
+  format _all_ ;
+  format EVICTION_DATE mmddyy10.;
+


Proc Import add lots of unneeded formats and informats. I use these first two statements to clear them. But then the date format is needed for EVICTION_DATE, so I restore that one.

ptatian · 2024-11-01T20:59:52Z

+%DC_mar_geocode(
+data = evictions_addresses,
+staddr = street_address,
+out = GeocodedEvictionsData,


I removed ZIP codes from the geocoding. Since many addresses are missing ZIP codes, we were getting a lot of unmatched addresses. From the LOG, 333 unmatched.

_________ Geocoding Summary _____________________________ Address data: WORK._DCG_INDAT Output data: WORK._DCG_OUTDAT STREET lookup data: MAR.GEOCODE_94_DC_M ZIP lookup data: SASHELP.ZIPCODE Geocoding method: Street level Run date: 31Oct2024 Obs processed: 2,923 Elapsed time: 00:00:03 Obs per minute: 53,781 Street matches: 2,555 ZIP matches: 35 Not matched: 333 _________________________________________________________

It works better without the ZIP codes.

_________ Geocoding Summary _____________________________ Address data: WORK._DCG_INDAT Output data: WORK._DCG_OUTDAT STREET lookup data: MAR.GEOCODE_94_DC_M CITY lookup data: MAPSGFK.USCITY_ALL Geocoding method: Street level Run date: 01Nov2024 Obs processed: 2,923 Elapsed time: 00:00:27 Obs per minute: 6,478 Street matches: 2,922 City matches: 1 Not matched: 0 _________________________________________________________

This is a bit of a quirk in the geocoding macro.

RAGarMir · 2024-11-13T15:49:43Z

Added if/then statement to correct "cancelled" Spelling

Thanks, @RAGarMir. Unfortunately, this code does not work (that is, it does not make the change we want). Can you tell why?

@ptatian I am not sure why I does not work. After running the code and looking at the outputs I assume that the if/then statement is in the wrong place and the line with the "Cancelled" disposition is filtered out at an earlier step.

Re: StreetAlt_GeoEvict_2024.txt
I updated the file to remove the line referencing LACKLAND

Condition statements in SAS are case-sensitive by default. If you write disposition = 'cancelled', that is only true if the value is all lower case. But, in the data the disposition values are uppercase (CANCELLED).

It's better, however, not to assume the case of the values. In a future data file they might be different. You can use the upcase() function to convert the values of disposition to all upper case for the comparison: upcase(disposition) = 'CANCELLED'.

And, then, you want to assign the new value in uppercase as well so that they are consistent. 'CANCELED'

I reviewed the default StreetAlt file and found some other problem entries. I've fixed the file in the MAR library but for your StreetAlt_GeoEvict_2024.txt file, you will want to delete these rows as well.

MC KENNA,MCKENNA SURCUM CORDA,SURSUM CORDA SURSUM CURDA,SURSUM CORDA VANDENBURG,VANDENBERG YONT,YOUNT

RAGarMir · 2024-11-13T15:55:18Z

@ptatian I resolved the issues mentioned above. I did come across a new issue. I am getting the following error code:
ERROR: [StreetAlt] Invalid entry for correct spelling of street name LACKLAND WAY G BA as LACKLAND

It looks like "LACKLAND" is not in the ValidStreets.html file. I am unsure whether I should add it to the file or find another way to resolve the error.

ptatian · 2024-11-14T18:45:16Z

Thanks, @RAGarMir. Unfortunately, this code does not work (that is, it does not make the change we want). Can you tell why?

ptatian · 2024-11-14T18:50:39Z

@RAGarMir Regarding Lackland, what the error message means is that Lackland was used as a correct street name spelling in StreetAlt, but there is no such street in DC. It may have been in the original StreetAlt file, and perhaps there was such a street earlier.

The solution is simply to delete this row from Prog\2024\StreetAlt_GeoEvict_2024.txt

LACKLAND WAY G BA,LACKLAND

RAGarMir · 2024-11-15T19:38:43Z

@ptatian I updated the StreetAlt_GeoEvict2024.txt file and removed all problem entries.

I also updated the GeocodeEvictions_DC2024.sas file to include the upcase function within the added if/then statement. I ran the code without errors. I was not able to tell if the upcase function did fix the problem with the disposition='CANCELLED' entry.

ptatian · 2024-11-15T20:26:23Z

Thanks, @RAGarMir . Look at the frequency procedure output for the DISPOSITION variable to check whether the CANCELLED entry is still present in the data.

RAGarMir · 2024-11-15T21:19:33Z

@ptatian I could not find the CANCELLED entry in the output. I am assuming everything worked correctly.

ptatian · 2024-11-15T21:23:01Z

That sounds good!

ptatian · 2024-11-19T20:34:57Z

@RAGarMir This code is ready to go, but I'd like to move it to a new Evict repo that I have created. We will not merge this pull request in the Requests repo, but finish it up in the new repo.

Please see https://github.com/NeighborhoodInfoDC/Evict/issues/1 for instructions.

Create GeocodeEvictions_DC2024.sas

b70ad11

RAGarMir requested a review from ptatian October 22, 2024 20:29

Update GeocodeEvictions_DC2024

41f4019

ptatian added 2 commits November 1, 2024 16:40

Update GeocodeEvictions_DC2024.sas

2b7c436

Based on review

Add label for street_address

903e149

ptatian requested changes Nov 1, 2024

View reviewed changes

Update GeocodeEvictions_DC2024.sas

569790a

RAGarMir commented Nov 13, 2024

View reviewed changes

ptatian requested changes Nov 14, 2024

View reviewed changes

RAGarMir added 2 commits November 14, 2024 15:46

Update StreetAlt_GeoEvict_2024.txt

05c07b0

Updated GeocodeEvictions_DC2024.sas & StreetAlt_GeoEvict_2024.txt

83b8f7f

Uh oh!

Conversation

RAGarMir commented Oct 22, 2024

Uh oh!

ptatian commented Oct 22, 2024

Uh oh!

RAGarMir commented Oct 31, 2024

Uh oh!

ptatian commented Oct 31, 2024

Uh oh!

RAGarMir commented Oct 31, 2024

Uh oh!

ptatian commented Oct 31, 2024

Uh oh!

ptatian left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RAGarMir Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ptatian Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RAGarMir commented Nov 13, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ptatian commented Nov 14, 2024

Uh oh!

RAGarMir commented Nov 15, 2024

Uh oh!

ptatian commented Nov 15, 2024

Uh oh!

RAGarMir commented Nov 15, 2024

Uh oh!

ptatian commented Nov 15, 2024

Uh oh!

ptatian commented Nov 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ptatian left a comment •

edited

Loading

RAGarMir Nov 14, 2024 •

edited

Loading

ptatian Nov 14, 2024 •

edited

Loading