Jsondb - #67
Open
z9security wants to merge 86 commits into
Open
Conversation
Replace traditional relational columns with JSON document storage pattern. Each table now stores data as JSON blobs with SQLite handling ID generation. Changes: - Add JsonDocumentRepository<T> base class for JSON serialization - Consolidate 11 migrations into single _0000_InitialCreate with JSON schema - Convert all 9 repositories to use JSON document storage - Add JSON indexes for credential number and assignment lookups - Remove old Aporta.sqlite file from repo - Delete unused BaseRepository.cs All 72 tests pass with the new storage model. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allow IDs to be assigned by code instead of auto-increment, enabling sync from upstream systems where IDs are pre-assigned. Changes: - Remove AUTOINCREMENT from schema (except event table) - Add optional explicitId parameter to Insert method - Add Upsert method for insert-or-update by ID - EventRepository overrides Insert to always use AUTOINCREMENT - Add tests for explicit IDs and upsert functionality In production, either auto-assign (Aporta standalone) or explicit (upstream sync) mode is used, not mixed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Standalone tool to migrate from column-based schema (v0-10) to JSON document storage. Supports upgrading from any old schema version. Features: - Embeds old migrations to bring DB up to v10 first - Transforms all tables to JSON format - Creates backup before migration - Dry-run mode for preview - Automatic rollback on error Usage: dotnet Aporta.Migration.dll <database-path> [--dry-run] Tests cover: - Full migration from v10 - Partial migration from v5 (applies missing migrations) - Dry-run mode - Already-migrated database (no-op) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- JSON schema uses version 100 (old column-based schema was 0-10) - App refuses to start with old schema, directs user to migration tool - Migration tool sets version to 100 after migrating Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Aporta now listens as a Z9/Open panel for host connections. The service handles identification handshake, DbChange, and DevActionReq messages with dummy responses as a foundation for data mapping. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Uses SpCoreProtoUtil.ToBigInteger() for proper card number conversion, auto-creates Person for each credential, falls back to card number as name when Cred.Name is blank, and skips credentials with no card number. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Panel now makes the outbound TCP connection to the Z9/Open Community host instead of listening. Upstream host is configured via command line args --z9OpenCommunityHost and --z9OpenCommunityPort (default 9723). Service only starts when --z9OpenCommunityHost is provided. Also updates Z9.Protobuf.Community from 1.0.0 to 1.0.2. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Read Logging:File:Path from configuration instead of hardcoding /var/log/aporta.log, which requires root permissions. Development defaults to relative "aporta.log" in working directory. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Send IDENTIFICATION first (initiating side sends before reading) - Use IoControllerCommunity DevMod instead of IoControllerZ9Spcore - Set SupportsIdentificationPassword/Upstream to true - Make panel Id configurable via --z9OpenCommunityId arg Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Added TCP/IP connection support for OSDP readers: - Bus.cs: Added ConnectionType enum (Serial/Tcp), TcpHost, TcpPort properties - OSDPDriver.cs: Handle TCP connections using TcpClientOsdpConnection Added automatic OSDP driver configuration from Z9 Dev messages: - Z9OpenCommunityProtocolService.cs: Process Dev messages to extract OSDP CredReader configurations, fires OsdpConfigurationReceived event - StartupWorker.cs: Subscribe to OsdpConfigurationReceived, automatically enable OSDP driver extension, add TCP bus and device Updated Z9.Protobuf.Community to 1.0.3 for community proto field number fixes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- OSDPDriver: Add NotifyEndpointOnlineStatus() to raise OnlineStatusChanged events when OSDP devices come online/offline - OSDPDriver: Add try-catch in PanelOnConnectionStatusChanged to prevent silent async exceptions - ExtensionService: Forward OnlineStatusChanged events from drivers - Z9OpenCommunityProtocolService: Add endpoint-to-config mapping and SendCredReaderOnlineEvent() to send EVT messages to Z9 - StartupWorker: Disable OSDP extension in Z9 Open Community mode until config is received from the host, subscribe to OnlineStatusChanged Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix host/port parsing in Z9OpenCommunityProtocolService to correctly handle serialPortAddress that already contains port (e.g., "localhost:9843") - Add constructor to OnlineStatusChangedEventArgs for endpoint notifications - Add --cleanDatabase CLI argument to delete database on startup for testing - Reorder startup to disable OSDP extension BEFORE ExtensionService.Startup() to prevent loading stale configuration - Use direct database access to disable OSDP since ExtensionService isn't started yet when we need to clear the config - Add debug logging to OSDPDriver for troubleshooting device online flow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add SendAccessEvent method to Z9OpenCommunityProtocolService for sending
access granted/denied events back to Z9
- Add OnAccessCredentialReceived handler in StartupWorker to forward card
reads from OSDP readers to Z9 as access denied events (unknown credential)
- Fix OSDPDriver endpoint ID parsing for TCP connections where the port name
contains a colon (e.g., "localhost:9843"). The previous Split(":")[0] logic
broke when the port name itself contained colons.
- Add ParseEndpointId helper method to properly extract port name and OSDP
address from endpoint IDs regardless of connection type
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Encode credentials from Z9 as bit strings using BinaryFormatter.Encode() - Decode card reads using BinaryFormatter.Decode() to extract credNum - Store DataFormat, DataLayout, CredTemplate from Z9 downloads - Update SendAccessEvent to include decoded credNum on EvtCredRef - Add conditional Z9.Protobuf project reference for z9main development Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace in-memory dictionaries with database storage for Z9 Open protobuf types. Add ProtoJsonRepository base class using Google.Protobuf JsonFormatter/JsonParser for serialization with proper oneof extension handling. New tables: cred_template, data_layout, data_format, priv, sched, hol, hol_cal, hol_type. All types now persist across restarts. Also fixes migration system bug that used version as array index. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enable Aporta to grant access when a known credential is swiped: AccessService.cs: - Add AccessDecisionMade event for external systems to subscribe - Add 35-bit credential decoding fallback to match raw card reads against credentials stored as decimal numbers - Allow access decisions without door strike (for community controllers) Z9OpenCommunityProtocolService.cs: - Only set EvtSubCode for denied events (granted events don't need one) StartupWorker.cs: - Auto-create Door entities when OSDP readers come online - Subscribe to AccessDecisionMade event to forward decisions - Add retry logic for endpoint lookup during door creation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Z9CredRepository to store full Z9 Cred proto with privilege bindings - Add migration _0002_AddZ9CredTable for z9_cred table - Implement HasAccessPrivilege() in AccessService to check privilege bindings - Store Z9 Cred proto when processing credentials from Z9 server - Check for PrivType.Door or precision access bindings before granting access Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Includes schedule utilities for evaluating time-based access restrictions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Check Z9 Cred status before granting access: - Disabled credential -> ACCESS_DENIED_INACTIVE - Not yet effective (future date) -> ACCESS_DENIED_NOT_EFFECTIVE - Expired (past date) -> ACCESS_DENIED_EXPIRED - No privilege bindings -> ACCESS_DENIED_NO_PRIV Add corresponding EventReason enum values and map to EvtSubCode for proper event reporting to Z9 host. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… checking Upgrade NuGet package to 1.0.6, fix protobuf API usage for DoorAccessPriv (use Elements collection) and SchedElement (use SchedDays/Start/Stop), add DoorUnid to OsdpReaderConfig for door-to-reader mapping, implement schedule restriction checking for access privileges, and simplify csproj references by removing conditional local Z9.Protobuf paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lookup 1.0.7 adds the missing privUnid field to CredPrivBinding, enabling proper privilege resolution for schedule-restricted access bindings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Process Z9 Actuator Dev objects to extract door strike output config, match actuators to readers via shared door (logicalParentUnid), and wire DoorStrikeEndpointId when the OSDP output endpoint comes online. Handle DevActionReq DOOR_MOMENTARY_UNLOCK from the Z9 host. Filter OnlineStatusChanged to only process reader endpoints (not outputs) to prevent duplicate door creation. Fix OpenDoor to fire strike asynchronously so the access processing task completes immediately, preventing the deduplication check from dropping subsequent card reads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subscribe to OSDP state changes and send DoorUnlocked/DoorLocked events when the strike output changes, and DoorOpened/DoorClosed events when the door contact input changes. Process Door and Sensor dev objects from Z9 to wire door contact inputs alongside existing strike outputs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire REX sensor from Z9 as OSDP input endpoint on door, send EXIT_REQUESTED event on REX press, and fire door strike if activateStrikeOnRex is enabled. Upgrade Z9.Protobuf.Community to 1.0.8 for DoorConfig proto support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract strikeTime from community proto DoorConfig and use it for door strike pulse duration instead of hardcoded 3 seconds. Defaults to 3000ms when not configured. Upgrade Z9.Protobuf.Community to 1.0.9. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track per-door strike state and send DoorForced/DoorNotForced events to Z9 when the door contact opens while the strike is not energized. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a credential has extDoorTime=true, use the door's extendedStrikeTime instead of the normal strikeTime for access grant strike duration. Also remove unnecessary IsOpen check from OSDPDriver.AddUpdateDevice - devices should be registered in the panel regardless of connection state, as the reader may come online later. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces DevStateRecord-based state tracking using z9/open proto types, replacing the need for ad-hoc state queries. DevStateService is wired into InputService, OutputService, StartupWorker (door aspects), and the SpCore protocol service (batch/continuous DevStateRecord forwarding). Adds Flex REST API endpoint at flex/devStateRecord/list for read-only state queries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…to Flex API Events cleanup: Delete EventsController, EventService, and EventServiceTests — Monitoring.razor already uses FlexApi exclusively. Move EventType enum to EventData.cs and delete Event.cs. Credentials migration: Enhance FlexCredController with Save override (auto-assign unid, ensure default CredTemplate, SignalR notify), Delete override, and Enroll endpoint. Add FlexPostAsync/SaveCredAsync/DeleteCredAsync/EnrollCredAsync to FlexApiService. Rewrite Credentials.razor to use Flex API exclusively. Delete CredentialsController, PeopleController, CredentialService, PeopleService. Remove service registrations from Startup.cs and paths from Paths.cs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sensors.razor, Actuators.razor, and Monitoring.razor were hybrid — listing via Flex but create/delete/available/state via legacy REST controllers. Now all three pages use Flex API exclusively, continuing the pre-fork alignment started with the Credentials migration. - Rewrite FlexSensorController and FlexActuatorController with full endpoints: list (enabled-only filter), save/assign, delete, available pool, and state get/set - Add 9 FlexApiService client methods for sensor/actuator CRUD and state - Strip InputService/OutputService to event-handler-only singletons (retain hardware→SignalR state relay) - Delete InputsController, OutputsController, Input.cs, Output.cs - Update Doors.razor to use FlexApi for available actuators Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Doors.razor now uses FlexApiService exclusively for list/create/delete and available readers. FlexDoorController gains custom List (top-level doors only), POST create, Delete (via DoorConfigurationService), and available/readers + available/endpoints routes. DoorsController is slimmed to read-only GETs for integration test compatibility. Virtual driver Configuration.razor also migrated from IDoorCalls to FlexApiService. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Launch Aporta as a process and exercise the Flex API through the typed Kiota FlexClient, catching swagger-vs-implementation drift at the deserialization layer. Tests cover auth (401, valid, bad creds), all 13 list endpoints, CRUD for schedule and holiday type, pagination, and session termination. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix three issues preventing DevStateRecords from reaching SpCoreDriver: - SendAllDevStateRecords skipped empty responses, deadlocking batch polling - Door contact/REX sensors used wrong door unid (LogicalParentUnid vs config) - Concurrent protobuf access between batch serialization and UpdateAspect Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Persist a CONTROLLER_STARTUP event to z9_evt on application startup (before connecting to the host), then send it when the host identifies. References the top-level IoControllerCommunity device. Unit tests updated with CompleteIdentification helper that consumes the startup event. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Send MOMENTARY_UNLOCK event when a host-initiated DevActionReq triggers a door momentary unlock, providing an audit trail distinct from REX- triggered unlocks. Update README completed/TODO sections accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Z9/Flex Community swagger defines paths without a /flex/ prefix (e.g., /cred/list, /door/list). This removes the prefix from all controller routes, updates FlexAuthFilter to use namespace-based controller detection instead of path-based, and updates all client URL references accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Send TAMPER/TAMPER_NORMAL events with DevAspectState tracking when an OSDP reader reports a tamper state change, and CRED_READER_POWER_CYCLE when a reader reports it has power-cycled. Bump Z9.Protobuf.Community to 1.0.12 for the new TamperState enum. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…luation Evaluates all schedules once per minute (aligned to minute boundary) using SchedEvaluator.InSched, tracks active/inactive state, and reports transitions as SCHED_ACTIVE (127) / SCHED_INACTIVE (128) events. Events are persisted to z9_evt and sent over the wire when connected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Populate EvtDevRef and EvtCredRef in InsertAccessEvt so monitoring page shows device and credential columns for access events - Add EvtCode 17 "Card Read" to FlexEvtCodeHelper - Revert auto-credential creation on badge swipe; swipes now only produce RAW_CRED_READ events - Add enroll-from-read flow: raw reads API endpoint, enroll-from-read controller endpoint, and updated Credentials.razor enrollment UI to select from raw card reads instead of unassigned credentials - Update Selenium tests to verify new enrollment dropdown and monitoring page device/credential assertions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hand-maintained FlexEvtCodeHelper with Messages.GetEvtCodeText() and Messages.GetEvtSubCodeText() from Z9.Protobuf.Community 1.0.13. Display strings now match the commercial Z9 product. FlexMapper populates EvtCodeText/EvtSubCodeText on FlexEvt during conversion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace silent auto-authentication with a visible login form. FlexApiService gains LoginAsync/Logout/IsAuthenticated; App.razor gates on auth state and restores sessions from sessionStorage across page reloads. Selenium tests log in through the UI and capture a login screen screenshot. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ide Unid assignment Credentials page: rename Person to Credential labels, add Door Access column showing assigned doors with schedule names, add Edit Access modal for managing door access privileges with schedule selection. Add Schedules page for creating/editing schedules with time intervals and day-of-week selection. Fix server-side Unid auto-assignment in FlexCrudControllerBase so new records created via web UI get unique IDs (previously defaulted to 0, causing database collisions). Selenium tests cover the full flow: create schedule, edit credential access to use it, verify Access Denied when schedule is not active. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create, edit, and delete binary card formats (DataFormat + DataLayout) from a new Card Formats page under Configuration > Access. Each format defines min/max bits and a list of binary elements (static, parity, or field). Selenium test validates creating a Standard 26-Bit format. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the Holidays configuration page for creating, editing, and deleting holidays with repeat and schedule preservation options. Exposes the Holidays checkbox on schedule time intervals so they can match during holidays. Includes Selenium tests verifying the holiday-aware access control flow: schedule without holidays causes denial on a holiday day, enabling holidays on the schedule restores access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uses <clear /> to override any developer-local NuGet sources, ensuring all packages resolve from nuget.org only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 0359564.
Replace the local project reference to z9flex-community with a NuGet PackageReference (v1.0.1). Also remove NuGet.Config which contained a local package source that is no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Picks up SpCoreHostConnection with optional password support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sets SpCoreDevUse to IoControllerSecondary in identification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds exponential backoff for SpCoreHostConnection reconnects. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates the Aporta database to use JSON clobs for all datatypes, and have ids not assigned by sqlite, but able to be auto-assigned in code, or assigned from the outside