diff --git a/README.md b/README.md
index fd2808a..f130af1 100644
--- a/README.md
+++ b/README.md
@@ -457,9 +457,9 @@ MIT License - see LICENSE file for details.
## Support
-- **Documentation**: [Wiki](https://github.com/yourusername/erpnext_github_integration/wiki)
-- **Issues**: [GitHub Issues](https://github.com/yourusername/erpnext_github_integration/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/yourusername/erpnext_github_integration/discussions)
+- **Documentation**: [Wiki](https://github.com/Yankyyyy/erpnext_github_integration/wiki)
+- **Issues**: [GitHub Issues](https://github.com/Yankyyyy/erpnext_github_integration/issues)
+- **Discussions**: [GitHub Discussions](https://github.com/Yankyyyy/erpnext_github_integration/discussions)
## Changelog
diff --git a/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js b/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js
index 058a240..03cb860 100644
--- a/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js
+++ b/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js
@@ -144,10 +144,13 @@ frappe.ui.form.on("GitHub Settings", {
frappe.call({
method: 'erpnext_github_integration.github_api.sync_all_repositories',
callback: function(r) {
- frappe.msgprint(__('All repositories sync initiated. Check background jobs for progress.'));
- },
- error: function(err) {
- frappe.msgprint(__('Error initiating sync: {0}', [err.responseText || JSON.stringify(err)]));
+ if (r.message) {
+ frappe.msgprint({
+ title: __('Repositories Sync'),
+ indicator: r.message.failed > 0 ? 'red' : 'green',
+ message: __(`Success: ${r.message.success}
Failed: ${r.message.failed}`)
+ });
+ }
}
});
});
diff --git a/erpnext_github_integration/patches.txt b/erpnext_github_integration/patches.txt
index f15c3a9..cf7af62 100644
--- a/erpnext_github_integration/patches.txt
+++ b/erpnext_github_integration/patches.txt
@@ -3,4 +3,5 @@
# Read docs to understand patches: https://frappeframework.com/docs/v14/user/en/database-migrations
[post_model_sync]
-# Patches added in this section will be executed after doctypes are migrated
\ No newline at end of file
+# Patches added in this section will be executed after doctypes are migrated
+erpnext_github_integration.patches.add_github_username
\ No newline at end of file
diff --git a/erpnext_github_integration/patches/add_github_username.py b/erpnext_github_integration/patches/add_github_username.py
new file mode 100644
index 0000000..60dca5f
--- /dev/null
+++ b/erpnext_github_integration/patches/add_github_username.py
@@ -0,0 +1,12 @@
+import frappe
+
+def execute():
+ if not frappe.db.exists("Custom Field", {"dt": "User", "fieldname": "github_username"}):
+ frappe.get_doc({
+ "doctype": "Custom Field",
+ "dt": "User",
+ "fieldname": "github_username",
+ "label": "GitHub Username",
+ "fieldtype": "Data",
+ "insert_after": "username"
+ }).insert()
diff --git a/erpnext_github_integration/patches/after_install.py b/erpnext_github_integration/patches/after_install.py
index f7d822d..2d9cdb2 100644
--- a/erpnext_github_integration/patches/after_install.py
+++ b/erpnext_github_integration/patches/after_install.py
@@ -14,16 +14,6 @@ def create_custom_fields_and_scripts():
# Custom fields to be created
custom_fields = {
- 'User': [
- dict(
- fieldname='github_username',
- label='GitHub Username',
- fieldtype='Data',
- insert_after='email',
- unique=1,
- description='GitHub username for integration purposes'
- )
- ],
'Project': [
dict(
fieldname='repository',
diff --git a/technical_documentation.md b/technical_documentation.md
new file mode 100644
index 0000000..d37a43a
--- /dev/null
+++ b/technical_documentation.md
@@ -0,0 +1,307 @@
+### ERPNext GitHub Integration – Technical Documentation
+
+## Overview
+- Purpose: Synchronize GitHub repositories, issues, pull requests, branches, collaborators, and related activities into ERPNext, with UI actions and webhooks for near-real-time updates.
+- Platform: Frappe/ERPNext app.
+- Key components:
+ - Server modules: `github_client.py`, `github_api.py`, `api.py`, `webhooks.py`, `hooks.py`
+ - Data models (DocTypes): `GitHub Settings`, `Repository`, `Repository Issue`, `Repository Pull Request`, child tables for branches, members, assignees, reviewers
+ - Client scripts: `public/js/project_client.js`, `public/js/task_client.js`, `repository.js` (form UI actions)
+ - Patches and setup: `patches/after_install.py`, `patches/add_github_username.py`
+ - Scheduler: hourly sync
+
+## Installation and Packaging
+- Standard Python packaging:
+ - `pyproject.toml`: uses `flit_core` for build; Python 3.10+; linter config via Ruff.
+ - `setup.py`: defines package metadata; `include_package_data=True`.
+- App metadata:
+ - `modules.txt`: declares module: “Erpnext Github Integration”.
+ - `hooks.py` registers assets, events, scheduled jobs.
+
+## Configuration and Setup
+
+### Prerequisites
+- ERPNext v14+/Frappe v14+ (compatible code includes `frappe.has_role` fallbacks).
+- GitHub Personal Access Token (PAT) or OAuth app (project implements PAT-driven flows; OAuth fields available in settings).
+- Webhook secret (optional but recommended).
+
+### After-install patch
+- `erpnext_github_integration.patches.after_install.create_custom_fields_and_scripts`:
+ - Creates role `GitHub Admin`.
+ - Creates custom fields:
+ - `Project.repository` (Link → `Repository`)
+ - `Task.github_repo` (Link → `Repository`)
+ - `Task.github_issue_number` (Int)
+ - `Task.github_pr_number` (Int)
+ - Creates or updates `Custom Script` for `Task` and `Project` with GitHub actions.
+ - Seeds default `GitHub Settings` single if missing.
+ - Grants `GitHub Admin` CRUD on GitHub DocTypes via `Custom DocPerm`.
+ - Creates sample workflow states.
+- Patch `patches/add_github_username.py`:
+ - Adds `User.github_username` field if missing.
+
+### App Hooks
+- Assets:
+ - `app_include_js`: injects `/assets/erpnext_github_integration/js/project_client.js` and `/assets/erpnext_github_integration/js/task_client.js`.
+- Doc Events:
+ - `Repository.validate` → `erpnext_github_integration.api.validate_repository`: validates `full_name` and backfills `repo_owner`, `repo_name`, `url`.
+- Dashboard:
+ - `override_doctype_dashboards["Repository"]` → `api.get_repository_dashboard_data`.
+- Scheduler:
+ - Hourly: `github_api.sync_all_repositories`.
+
+## Data Model (DocTypes)
+
+### GitHub Settings (Single)
+- Fields:
+ - `auth_type` (Select: PAT/OAuth)
+ - `personal_access_token` (Password)
+ - `oauth_client_id` (Data)
+ - `oauth_client_secret` (Password)
+ - `webhook_secret` (Password)
+ - `default_organization` (Data)
+ - `default_visibility` (Select: Public/Private)
+ - `last_sync` (Datetime)
+ - `enabled` (Check)
+- Permissions: `System Manager` (R/W/C/D), `GitHub Admin` (R/W).
+
+### Repository
+- Naming: `autoname: field:full_name`.
+- Core fields:
+ - `full_name` (owner/repo, unique, required), `repo_name`, `repo_owner`
+ - `github_id`, `url`, `visibility` (Public/Private), `default_branch`
+ - `is_synced` (Check), `last_synced` (Datetime)
+ - Tables:
+ - `branches_table` → child `Repository Branch`
+ - `members_table` → child `Repository Member`
+- Permissions: `System Manager` (R/W/C/D), `GitHub Admin` (R/W).
+
+### Repository Branch (Child)
+- Fields: `repo_full_name`, `branch_name`, `commit_sha`, `protected` (Check), `last_updated` (Datetime).
+- `istable = 1`.
+
+### Repository Member (Child)
+- Fields: `repo_full_name`, `github_username`, `github_id`, `role` (member/maintainer), `email`.
+- `istable = 1`.
+
+### Repository Issue
+- Naming: `autoname: format:{repository}-#{issue_number}`.
+- Fields: `repository` (Link → `Repository`), `issue_number` (Int, required), `title`, `body` (Text), `state` (open/closed), `labels`, `url`, `github_id`, `created_at`, `updated_at`.
+- Table: `assignees_table` → child `Repository Issue Assignee`.
+
+### Repository Issue Assignee (Child)
+- Fields: `issue` (Link → `Repository Issue`), `user` (Link → `User`).
+- `istable = 1`.
+
+### Repository Pull Request
+- Naming: `autoname: format:{repository}-#{pr_number}`.
+- Fields: `repository` (Link → `Repository`), `pr_number` (Int, required), `title`, `body`, `state` (open/closed/merged), `author`, `head_branch`, `base_branch`, `mergeable_state`, `github_id`, `url`, `created_at`, `updated_at`.
+- Table: `reviewers_table` → child `Repository PR Reviewer`.
+
+### Repository PR Reviewer (Child)
+- Fields: `pull_request` (Link → `Repository Pull Request`), `user` (Link → `User`).
+- `istable = 1`.
+
+## Server Modules
+
+### github_client.py (GitHub API client)
+- Base URL `https://api.github.com`.
+- Headers: Authorization: `token `, Accept: `application/vnd.github.v3+json`, User-Agent: `erpnext-github-integration`.
+- Rate limiting:
+ - Checks `X-RateLimit-Remaining` and `X-RateLimit-Reset`; sleeps until reset if depleted.
+- Pagination:
+ - Follows RFC5988 `Link` header; `_get_with_pagination` accumulates all pages.
+- `github_request(method, path, token, params=None, data=None, retry=2)`:
+ - JSON body requests; handles 200/201/204; paginated responses; raises Frappe errors on failures with retries on rate-limit 403.
+
+### github_api.py (Integration logic)
+- Role check compatibility: `has_role(role)` supports older/newer Frappe.
+- `convert_github_datetime(dt)`: normalizes GitHub ISO timestamps to Asia/Kolkata (IST) timezone and returns MySQL-friendly string without tz.
+- Admin guard: `_require_github_admin()`.
+- Permission check for sync: `_can_sync_repo(repo_full_name)` (GitHub Admin or project manager of linked `Project`).
+- Connection and lookup:
+ - `test_connection()`: validates PAT via `/user`.
+ - `get_github_username_by_email(email)`: GitHub user search API.
+- Listing:
+ - `list_repositories(organization=None)`: `/orgs/{org}/repos` or `/user/repos` with pagination.
+ - `list_branches(repo_full_name)`, `list_teams(org_name)`, `list_repo_members(repo_full_name)`.
+- CRUD/actions:
+ - `create_issue(repository, title, body=None, assignees=None, labels=None)` → creates GitHub issue and mirrors a `Repository Issue` record.
+ - `bulk_create_issues(repository, issues)` → batch create in GitHub and mirror locally.
+ - `assign_issue(repo_full_name, issue_number, assignees)` → PATCH GitHub issue; updates local assignees table.
+ - `create_pull_request(repository, title, head, base, body=None)` → creates GitHub PR and mirrors `Repository Pull Request`.
+ - `add_pr_reviewer(repo_full_name, pr_number, reviewers)` → request reviewers on GitHub; updates local reviewers.
+ - `manage_repo_access(repo_full_name, action, identifier, permission='push')`:
+ - `add_collaborator`/`remove_collaborator`
+ - `add_team`/`remove_team` on an org repo (requires admin)
+- Sync:
+ - `sync_repo(repository)`:
+ - Fetches repo info, branches (latest commit dates via `/commits?sha=branch&per_page=1`), issues (state=all), PRs (state=all), members.
+ - Upserts `Repository`, clears/rebuilds `branches_table` and `members_table`, mirrors issues and PRs with child tables, converts timestamps to IST.
+ - `sync_repo_members(repo_full_name)`:
+ - Updates `Repository.members_table`.
+ - Syncs linked `Project.project_users` by matching `User.github_username` or email fallback; sets role “Project User”.
+ - `sync_all_repositories()`:
+ - Hourly scheduler job; iterates all `Repository` and calls `sync_repo`.
+ - Updates `GitHub Settings.last_sync`.
+- Analytics and webhooks:
+ - `get_repository_activity(repository, days=30)`:
+ - Summaries: commit count, issues count (excluding PRs), pulls count, returns details preview.
+ - `create_repository_webhook(repo_full_name, webhook_url=None, events=None)`, `list_repository_webhooks(repo_full_name)`.
+
+### webhooks.py (Inbound GitHub webhooks)
+- Entry: `github_webhook()` (guest allowed)
+ - Validates HMAC signature with `webhook_secret` if present using `X-Hub-Signature-256`.
+ - Robust header extraction `X-GitHub-Event` with fallbacks; infers event if header absent.
+ - Ensures `repository.full_name` exists locally; processes events inline (not background) for reliability.
+- Handlers:
+ - `_handle_issues_event`: upsert/delete `Repository Issue` and assignees based on `action`.
+ - `_handle_pull_request_event`: upsert `Repository Pull Request` and reviewers based on `action`.
+ - `_handle_push_event`: updates `Repository.last_synced` and branch commit SHA/`last_updated`; adds branch if new.
+ - `_handle_member_event`: add/remove collaborators in `members_table`.
+ - `_handle_repository_event`: updates repo attributes; handles rename (`full_name`, `repo_name`, `repo_owner`, `url`).
+
+### api.py (ERPNext-facing helpers)
+- Form validation and UX:
+ - `validate_repository(doc, method)`: validates `full_name` and populates `repo_owner`, `repo_name`, `url`.
+ - `get_repository_dashboard_data(data)`: defines dashboard sections for `Repository`.
+- User and project flows:
+ - `get_user_repositories()`: returns repos accessible to current user:
+ - All if `GitHub Admin`
+ - Project manager of linked projects
+ - Membership match by SQL on `members` JSON (optimizes for string search on `github_username`).
+ - `sync_user_github_profile()`: fills `User` fields from GitHub (name/bio/location) via username.
+ - `link_github_user_to_erp(github_username, erp_user)`: sets a user’s GitHub username.
+ - `get_repository_statistics(repo_full_name)`: counts issues/PRs by state, branches, members.
+ - `create_project_from_repository(repo_full_name, project_name=None)`: creates `Project` linked to repo.
+- Bulk import:
+ - `bulk_import_github_data(repo_full_name, import_type, force_update=False)`:
+ - `issues`: imports all non-PR issues (state=all).
+ - `pull_requests`: imports all PRs (state=all).
+ - Upsert with `force_update` toggle; records import/update/skip/errors.
+
+## Client/UI Behavior
+
+### Repository form actions (`repository.js`)
+- Buttons (under “GitHub” or “Actions” groups):
+ - Sync Repository Data → `github_api.sync_repo`
+ - Sync Members → `github_api.sync_repo_members`
+ - Create Issue → `github_api.create_issue` (prompt for title/body/assignees/labels)
+ - Create Pull Request → `github_api.create_pull_request` (prompt for title/head/base/body)
+ - Manage Access → `github_api.manage_repo_access` (collaborator/team add/remove, permission)
+ - Show Activity → `github_api.get_repository_activity` summary dialog
+ - Manage Webhooks → list existing hooks; create via `github_api.create_repository_webhook`
+ - Open in GitHub → opens `url`
+- Auto-fill on `full_name` change: sets `repo_owner`, `repo_name`, `url`.
+
+### Project form (`public/js/project_client.js`)
+- Buttons:
+ - Sync Members from Repository → `github_api.sync_repo_members`
+ - Sync Repository Data → `github_api.sync_repo` (note: the script uses `args: {repo_full_name: repo}` in one place and `sync_repo` expects `repository`; be mindful in use; the `Repository` form uses the correct signature)
+
+### Task form (`public/js/task_client.js`)
+- Buttons when `github_repo` set:
+ - Create GitHub Issue → `github_api.create_issue` (prompts, sets `github_issue_number`)
+ - Create Pull Request → `github_api.create_pull_request` (prompts, sets `github_pr_number`)
+ - Assign Issue → `github_api.assign_issue` (prompts for assignees)
+ - Add PR Reviewer → `github_api.add_pr_reviewer` (prompts for reviewers)
+
+## Permissions Model
+- `GitHub Admin` role:
+ - Created during install.
+ - Granted CRUD permissions on GitHub DocTypes via `Custom DocPerm`.
+ - Elevated actions require this role (bulk import, manage repo access, create/list webhooks, sync all).
+- Access in business logic:
+ - `_require_github_admin()` guards admin-only endpoints.
+ - `_can_sync_repo()` allows either `GitHub Admin` or `Project.project_manager` of a project linked to that repository.
+- Repository visibility is informational; actual GitHub API permissions are enforced via token scopes.
+
+## Security
+- PAT stored in `GitHub Settings` as Password field; accessed via `get_password`.
+- Webhook validation: HMAC-SHA256 using `webhook_secret` if configured; rejects invalid signatures.
+- ERPNext permission checks:
+ - Creation of `Task`, `Project`, `User` edits guarded with `frappe.has_permission` checks in helpers.
+
+## Error Handling and Logging
+- API errors:
+ - `github_client.github_request` raises Frappe exceptions on HTTP errors; retries on 403 rate-limit with backoff; paginates automatically.
+- Webhooks:
+ - Extensive `frappe.log_error` for missing data, unhandled events, processing errors; commits after each upsert to reduce partial failures.
+- Bulk import and sync:
+ - Records skipped/updated/imported counts; logs individual errors per record.
+
+## Rate Limiting and Performance
+- Client handles `X-RateLimit-Remaining` and `X-RateLimit-Reset` with sleep-and-retry.
+- Pagination used for list endpoints.
+- Scheduler runs hourly `sync_all_repositories`; prefer using webhooks for near real-time updates.
+- Activity endpoint returns only small previews in `details`.
+
+## API Endpoints (Whitelisted Methods)
+- Connection/lookup:
+ - `github_api.test_connection()`
+ - `github_api.get_github_username_by_email(email)`
+- Listing:
+ - `github_api.list_repositories(organization=None)`
+ - `github_api.list_branches(repo_full_name, per_page=100)`
+ - `github_api.list_teams(org_name, per_page=100)`
+ - `github_api.list_repo_members(repo_full_name, per_page=100)`
+- Repo lifecycle:
+ - `github_api.sync_repo(repository)`
+ - `github_api.sync_repo_members(repo_full_name)`
+ - `github_api.sync_all_repositories()`
+ - `github_api.manage_repo_access(repo_full_name, action, identifier, permission='push')`
+ - `github_api.create_repository_webhook(repo_full_name, webhook_url=None, events=None)`
+ - `github_api.list_repository_webhooks(repo_full_name)`
+ - `github_api.get_repository_activity(repository, days=30)`
+- Issues:
+ - `github_api.create_issue(repository, title, body=None, assignees=None, labels=None)`
+ - `github_api.bulk_create_issues(repository, issues)`
+ - `github_api.assign_issue(repo_full_name, issue_number, assignees)`
+- Pull requests:
+ - `github_api.create_pull_request(repository, title, head, base, body=None)`
+ - `github_api.add_pr_reviewer(repo_full_name, pr_number, reviewers)`
+- ERP-side helpers (api.py):
+ - `api.get_user_repositories()`
+ - `api.sync_user_github_profile()`
+ - `api.create_task_from_github_issue(issue_name, task_title=None)`
+ - `api.bulk_import_github_data(repo_full_name, import_type, force_update=False)` (admin)
+ - `api.link_github_user_to_erp(github_username, erp_user)`
+ - `api.get_repository_statistics(repo_full_name)`
+ - `api.create_project_from_repository(repo_full_name, project_name=None)`
+ - `api.can_user_sync_repo(repo_full_name)`
+
+## Webhooks Integration
+- Endpoint: `/api/method/erpnext_github_integration.webhooks.github_webhook`
+- Events handled:
+ - `issues`: open/edit/reopen/close/delete → upsert/delete `Repository Issue` + assignees.
+ - `pull_request`: open/edit/reopen/close/merged → upsert `Repository Pull Request` + reviewers.
+ - `push`: updates branch commit SHA and `last_updated`.
+ - `member`: add/remove collaborator in `members_table`.
+ - `repository`: `edited`/`renamed` → update repo attributes and `full_name`.
+- Security:
+ - Verify `X-Hub-Signature-256` with configured secret.
+- Operational note:
+ - Processing is synchronous (“immediate”) to avoid background job issues; could be moved to background once stable.
+
+## Desk/UI Highlights
+- `Repository` dashboard shows “Issues & PRs” and “Project Management” links.
+- Many UI actions are provided as custom buttons on `Repository`, `Project`, and `Task` for convenience.
+- `Repository.full_name` drives owner/name/url auto-fill.
+
+## Known Edge Cases and Notes
+- Timezone conversion uses IST; adjust if a different local timezone is desired.
+- Some client scripts pass `repo_full_name` instead of `repository` for `sync_repo` arguments; ensure to call with `repository=` for correctness as per `github_api.sync_repo`.
+- GitHub “issues” API includes PRs; code filters PRs out when needed.
+- `get_user_repositories()` uses a SQL LIKE on JSON field for members matching; effective but not relationally strict.
+
+## Extensibility
+- Add new DocTypes for additional GitHub entities (e.g., labels, milestones) following the same pattern (create list API call, mirror locally in child tables).
+- Add background job queues to decouple webhook processing for high volume.
+- Extend `GitHub Settings` with rate-limit thresholds, default sync intervals, custom event subscriptions.
+- Implement OAuth if needed: settings fields are present; add OAuth flow endpoints to exchange and store tokens per user or app.
+
+## Quick Start
+- Install the app via bench; run migrations; ensure `GitHub Settings` has a valid PAT and webhook secret.
+- Create `Repository` records with `full_name` values, or use `github_api.fetch_all_repositories(organization=...)` to populate.
+- Click “Create Repository Webhook” from a `Repository` form (Manage Webhooks) or configure in GitHub manually to point to the webhook URL.
+- Use “Sync Repository Data” to prime local data; rely on webhooks + hourly scheduler for continuous updates.
\ No newline at end of file