Unknown problem causing data loss

[regan-sarwas]

From Dylan Schertz:

On the sheep survey in GAAR we had my first significant data loss from Park Observer (we were able to patch things together from the pilot GPS). I wish I was more directly involved in the download so I would be more sure of somethings but they had loaded a POZ onto the computer and it wasn't giving them any data so they called me over. When I looked at it the POZ was about 24 kb (or so) so it seems like there was actually no data in the POZ. I did later confirm the POZ was empty by unzipping it and looking at all the tables. I looked at the iPad and saw data but also saw two duplicate error messages (something about mission not found unable to save). I switched the active survey to see if the other one also had data from the same day or it was all recorded on the one, there was nothing in the other survey and when I switched back to the first survey the data that was previously there had disappeared. I took this iPad outside collected data multiple times and exported that data successfully. I am not sure what happened that one time.

The version number was the full release of 2.0, iOS 14.5, and I was using the attached obsprot I wrote (a slight modification of the short form survey).

In the interest of not having this happen again I uninstalled park observer from that iPad, reinstalled it, and loaded the obsprot that I pulled from another iPad. I used this iPad on later days and had no issues with it.

I am curious if you have any thoughts as to what may have happened

sheep_medium_v1_30June2021.obsprot.txt

[regan-sarwas]

My initial guess is that for some reason the app was unable to create a "mission" in the database, and without that, all subsequent points were drawn on the screen, but not inserted into the database.

To clarify: the CoreData referential integrity (i.e. a missing mission) is not checked until the app tries to save the CoreData to disk. The app can draw GPS points and Observation on the map without a mission, but cannot be saved and will display an error message. This would explain the behavior seen above.

[regan-sarwas]

Assumption: Some object in the core data model requires a non-optional Mission and it's mission property is nil when CoreData tries to save, which causes an exception and the error message. Only three CoreData entities have a mission property and all three are required:

1. GpsPoint
2. MissionProperty
3. Observation

The ObjC interface requires that the mission property be a nullable on the three classes, so it is possible to create them without a mission. However, the managed objects can only be created in CoreData with the NSManagedObject.init(context:) constructor, the NSManagedObject.init(entity: insertInto:) constructor, or the NSEntityDescription.insertNewObject(forEntityName: into:) method. ParkObserver provides no additional init() constructors. Searching the code for (context: and (entity:, reveals that it the CoreData classes are never constructed this way. Similarly insertNewObject is only called 7 times, once within each of the 7 coredata classes, and always in the static new(in:) function provided by the class.

• GpsPoint: .new(in:) is called only once outside of the tests. That call in SurveyController.addGpsLocation() is followed immediately with a call to GpsPoint.initializeWith(mission: location:) which requires and sets a non-optional mission.
• MissionProperty: .new(in:) is called only once (outside of the tests) and that is by the static new(mission: ...) function provided by the class. That function requires and sets a non-optional mission. That method is only called once by the ObservationPresenter
• Observation: .new(in:) is called only once (outside of the tests) and that is by the static new(_ mission: ...) function provided by the class. That function requires and sets a non-optional mission. That method is only called once by the ObservationPresenter

Therefore it appears impossible to create an object that requires a mission without a mission. Maybe one of the mission properties is set to nil after creation

a search for .mission = in the code base reveals that outside of the tests, these three classes never have the mission property changed. It is only set in the functions identified above. Of note the ObservationPresenter has an optional mission property, but as expected, there is a guard against a null value before it is used.

Conclusion: It appears impossible for any of the 3 core data classes that require a non-null mission to be in a state without a mission when the app tries to save the core data objects.

[regan-sarwas]

Two new assumptions:

1. The error is a red herring. there is still a referential integrity problem (we can check), but it is not with the mission.
2. Something in the app is triggering a rare bug in Apple's core data code. This is highly unlikely and will be impossible to pursue without a reproducible test case.

The other required CoreData properties are:

AdhocLocation.map/latitude/longitude
AngleDistanceLocation.observation/angle/distance/direction
GpsPoint.latitude/longitude/timestamp
Map.name

[regan-sarwas]

Continuing the search for a referential integrity problem with CoreData that could cause the save to fail.

The following discussion ignores exceptions in the test code

• AdhocLocation
  This class is only created with the static .new(in:) function which is only called once by the ObservationPresenter.createAdHocLocation(in:) function. This function sets the required properties with guaranteed non-nil values.
• AngleDistanceLocation
  The properties angle, distance, and direction are all declared as non optional, so they are never nil (they will get default values on object creation. This class is only created with the static .new(in:) function which is only called once by the ObservationPresenter.createAngleDistanceLocation(in:) function. This function DOES NOT set the required observation property, because it does not yet exist. When the observation is created, it is passed the AngleDistanceLocation which will be assigned to the observation and also create the reverse relationship (via the CoreData rules). Therefore even though it appears that this required property in never set, it should not be nil (unless there is some interruption and the observation is not created -- This should be explored).
• GpsPoint
  Similar to discussion above every GpsPoint creation is done with a non nil CLLocation which has a non-nil timestamp and non-nil latitude/longitude, so while the properties must be declared as optional, and are nil for a very short time, they will always be set to a non-nil value in the create/initialize process. Every assignment (only the timestamp in SurveyController) to these properties is wrapped in an if let statement to ensure the new value is not nil.
• Map
  The class' static new(in:) function is only called by the class' static findOrNew(matching: in:) which is given a non-nil MapInfo to set the .name property with the non-nil MapInfo.title property. The Map.name property is never assigned outside of the object creation.

[regan-sarwas]

Solution may be to save more often. However, I am already saving more than I remembered. Saving is done in the following conditions:

• When the user stops track logging
• When the user switches to a different app (this app goes into the background)
• When the user switches to a different survey
• The user close/saves (i.e. does not cancel) the attribute editing form - that is observation or mission property creation.

The code could also do a save after every N GPS points in the track log. where N represents a few minutes of data collection.

The error messages are not erased, so they will accumulate if the user cannot save. This should be enough of an annoyance to get their attention that something is wrong.