diff --git a/README.md b/README.md index fd2808a..f130af1 100644 --- a/README.md +++ b/README.md @@ -457,9 +457,9 @@ MIT License - see LICENSE file for details. ## Support -- **Documentation**: [Wiki](https://github.com/yourusername/erpnext_github_integration/wiki) -- **Issues**: [GitHub Issues](https://github.com/yourusername/erpnext_github_integration/issues) -- **Discussions**: [GitHub Discussions](https://github.com/yourusername/erpnext_github_integration/discussions) +- **Documentation**: [Wiki](https://github.com/Yankyyyy/erpnext_github_integration/wiki) +- **Issues**: [GitHub Issues](https://github.com/Yankyyyy/erpnext_github_integration/issues) +- **Discussions**: [GitHub Discussions](https://github.com/Yankyyyy/erpnext_github_integration/discussions) ## Changelog diff --git a/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js b/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js index 058a240..03cb860 100644 --- a/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js +++ b/erpnext_github_integration/erpnext_github_integration/doctype/github_settings/github_settings.js @@ -144,10 +144,13 @@ frappe.ui.form.on("GitHub Settings", { frappe.call({ method: 'erpnext_github_integration.github_api.sync_all_repositories', callback: function(r) { - frappe.msgprint(__('All repositories sync initiated. Check background jobs for progress.')); - }, - error: function(err) { - frappe.msgprint(__('Error initiating sync: {0}', [err.responseText || JSON.stringify(err)])); + if (r.message) { + frappe.msgprint({ + title: __('Repositories Sync'), + indicator: r.message.failed > 0 ? 'red' : 'green', + message: __(`Success: ${r.message.success}
Failed: ${r.message.failed}`) + }); + } } }); }); diff --git a/erpnext_github_integration/patches.txt b/erpnext_github_integration/patches.txt index f15c3a9..cf7af62 100644 --- a/erpnext_github_integration/patches.txt +++ b/erpnext_github_integration/patches.txt @@ -3,4 +3,5 @@ # Read docs to understand patches: https://frappeframework.com/docs/v14/user/en/database-migrations [post_model_sync] -# Patches added in this section will be executed after doctypes are migrated \ No newline at end of file +# Patches added in this section will be executed after doctypes are migrated +erpnext_github_integration.patches.add_github_username \ No newline at end of file diff --git a/erpnext_github_integration/patches/add_github_username.py b/erpnext_github_integration/patches/add_github_username.py new file mode 100644 index 0000000..60dca5f --- /dev/null +++ b/erpnext_github_integration/patches/add_github_username.py @@ -0,0 +1,12 @@ +import frappe + +def execute(): + if not frappe.db.exists("Custom Field", {"dt": "User", "fieldname": "github_username"}): + frappe.get_doc({ + "doctype": "Custom Field", + "dt": "User", + "fieldname": "github_username", + "label": "GitHub Username", + "fieldtype": "Data", + "insert_after": "username" + }).insert() diff --git a/erpnext_github_integration/patches/after_install.py b/erpnext_github_integration/patches/after_install.py index f7d822d..2d9cdb2 100644 --- a/erpnext_github_integration/patches/after_install.py +++ b/erpnext_github_integration/patches/after_install.py @@ -14,16 +14,6 @@ def create_custom_fields_and_scripts(): # Custom fields to be created custom_fields = { - 'User': [ - dict( - fieldname='github_username', - label='GitHub Username', - fieldtype='Data', - insert_after='email', - unique=1, - description='GitHub username for integration purposes' - ) - ], 'Project': [ dict( fieldname='repository', diff --git a/technical_documentation.md b/technical_documentation.md new file mode 100644 index 0000000..d37a43a --- /dev/null +++ b/technical_documentation.md @@ -0,0 +1,307 @@ +### ERPNext GitHub Integration – Technical Documentation + +## Overview +- Purpose: Synchronize GitHub repositories, issues, pull requests, branches, collaborators, and related activities into ERPNext, with UI actions and webhooks for near-real-time updates. +- Platform: Frappe/ERPNext app. +- Key components: + - Server modules: `github_client.py`, `github_api.py`, `api.py`, `webhooks.py`, `hooks.py` + - Data models (DocTypes): `GitHub Settings`, `Repository`, `Repository Issue`, `Repository Pull Request`, child tables for branches, members, assignees, reviewers + - Client scripts: `public/js/project_client.js`, `public/js/task_client.js`, `repository.js` (form UI actions) + - Patches and setup: `patches/after_install.py`, `patches/add_github_username.py` + - Scheduler: hourly sync + +## Installation and Packaging +- Standard Python packaging: + - `pyproject.toml`: uses `flit_core` for build; Python 3.10+; linter config via Ruff. + - `setup.py`: defines package metadata; `include_package_data=True`. +- App metadata: + - `modules.txt`: declares module: “Erpnext Github Integration”. + - `hooks.py` registers assets, events, scheduled jobs. + +## Configuration and Setup + +### Prerequisites +- ERPNext v14+/Frappe v14+ (compatible code includes `frappe.has_role` fallbacks). +- GitHub Personal Access Token (PAT) or OAuth app (project implements PAT-driven flows; OAuth fields available in settings). +- Webhook secret (optional but recommended). + +### After-install patch +- `erpnext_github_integration.patches.after_install.create_custom_fields_and_scripts`: + - Creates role `GitHub Admin`. + - Creates custom fields: + - `Project.repository` (Link → `Repository`) + - `Task.github_repo` (Link → `Repository`) + - `Task.github_issue_number` (Int) + - `Task.github_pr_number` (Int) + - Creates or updates `Custom Script` for `Task` and `Project` with GitHub actions. + - Seeds default `GitHub Settings` single if missing. + - Grants `GitHub Admin` CRUD on GitHub DocTypes via `Custom DocPerm`. + - Creates sample workflow states. +- Patch `patches/add_github_username.py`: + - Adds `User.github_username` field if missing. + +### App Hooks +- Assets: + - `app_include_js`: injects `/assets/erpnext_github_integration/js/project_client.js` and `/assets/erpnext_github_integration/js/task_client.js`. +- Doc Events: + - `Repository.validate` → `erpnext_github_integration.api.validate_repository`: validates `full_name` and backfills `repo_owner`, `repo_name`, `url`. +- Dashboard: + - `override_doctype_dashboards["Repository"]` → `api.get_repository_dashboard_data`. +- Scheduler: + - Hourly: `github_api.sync_all_repositories`. + +## Data Model (DocTypes) + +### GitHub Settings (Single) +- Fields: + - `auth_type` (Select: PAT/OAuth) + - `personal_access_token` (Password) + - `oauth_client_id` (Data) + - `oauth_client_secret` (Password) + - `webhook_secret` (Password) + - `default_organization` (Data) + - `default_visibility` (Select: Public/Private) + - `last_sync` (Datetime) + - `enabled` (Check) +- Permissions: `System Manager` (R/W/C/D), `GitHub Admin` (R/W). + +### Repository +- Naming: `autoname: field:full_name`. +- Core fields: + - `full_name` (owner/repo, unique, required), `repo_name`, `repo_owner` + - `github_id`, `url`, `visibility` (Public/Private), `default_branch` + - `is_synced` (Check), `last_synced` (Datetime) + - Tables: + - `branches_table` → child `Repository Branch` + - `members_table` → child `Repository Member` +- Permissions: `System Manager` (R/W/C/D), `GitHub Admin` (R/W). + +### Repository Branch (Child) +- Fields: `repo_full_name`, `branch_name`, `commit_sha`, `protected` (Check), `last_updated` (Datetime). +- `istable = 1`. + +### Repository Member (Child) +- Fields: `repo_full_name`, `github_username`, `github_id`, `role` (member/maintainer), `email`. +- `istable = 1`. + +### Repository Issue +- Naming: `autoname: format:{repository}-#{issue_number}`. +- Fields: `repository` (Link → `Repository`), `issue_number` (Int, required), `title`, `body` (Text), `state` (open/closed), `labels`, `url`, `github_id`, `created_at`, `updated_at`. +- Table: `assignees_table` → child `Repository Issue Assignee`. + +### Repository Issue Assignee (Child) +- Fields: `issue` (Link → `Repository Issue`), `user` (Link → `User`). +- `istable = 1`. + +### Repository Pull Request +- Naming: `autoname: format:{repository}-#{pr_number}`. +- Fields: `repository` (Link → `Repository`), `pr_number` (Int, required), `title`, `body`, `state` (open/closed/merged), `author`, `head_branch`, `base_branch`, `mergeable_state`, `github_id`, `url`, `created_at`, `updated_at`. +- Table: `reviewers_table` → child `Repository PR Reviewer`. + +### Repository PR Reviewer (Child) +- Fields: `pull_request` (Link → `Repository Pull Request`), `user` (Link → `User`). +- `istable = 1`. + +## Server Modules + +### github_client.py (GitHub API client) +- Base URL `https://api.github.com`. +- Headers: Authorization: `token `, Accept: `application/vnd.github.v3+json`, User-Agent: `erpnext-github-integration`. +- Rate limiting: + - Checks `X-RateLimit-Remaining` and `X-RateLimit-Reset`; sleeps until reset if depleted. +- Pagination: + - Follows RFC5988 `Link` header; `_get_with_pagination` accumulates all pages. +- `github_request(method, path, token, params=None, data=None, retry=2)`: + - JSON body requests; handles 200/201/204; paginated responses; raises Frappe errors on failures with retries on rate-limit 403. + +### github_api.py (Integration logic) +- Role check compatibility: `has_role(role)` supports older/newer Frappe. +- `convert_github_datetime(dt)`: normalizes GitHub ISO timestamps to Asia/Kolkata (IST) timezone and returns MySQL-friendly string without tz. +- Admin guard: `_require_github_admin()`. +- Permission check for sync: `_can_sync_repo(repo_full_name)` (GitHub Admin or project manager of linked `Project`). +- Connection and lookup: + - `test_connection()`: validates PAT via `/user`. + - `get_github_username_by_email(email)`: GitHub user search API. +- Listing: + - `list_repositories(organization=None)`: `/orgs/{org}/repos` or `/user/repos` with pagination. + - `list_branches(repo_full_name)`, `list_teams(org_name)`, `list_repo_members(repo_full_name)`. +- CRUD/actions: + - `create_issue(repository, title, body=None, assignees=None, labels=None)` → creates GitHub issue and mirrors a `Repository Issue` record. + - `bulk_create_issues(repository, issues)` → batch create in GitHub and mirror locally. + - `assign_issue(repo_full_name, issue_number, assignees)` → PATCH GitHub issue; updates local assignees table. + - `create_pull_request(repository, title, head, base, body=None)` → creates GitHub PR and mirrors `Repository Pull Request`. + - `add_pr_reviewer(repo_full_name, pr_number, reviewers)` → request reviewers on GitHub; updates local reviewers. + - `manage_repo_access(repo_full_name, action, identifier, permission='push')`: + - `add_collaborator`/`remove_collaborator` + - `add_team`/`remove_team` on an org repo (requires admin) +- Sync: + - `sync_repo(repository)`: + - Fetches repo info, branches (latest commit dates via `/commits?sha=branch&per_page=1`), issues (state=all), PRs (state=all), members. + - Upserts `Repository`, clears/rebuilds `branches_table` and `members_table`, mirrors issues and PRs with child tables, converts timestamps to IST. + - `sync_repo_members(repo_full_name)`: + - Updates `Repository.members_table`. + - Syncs linked `Project.project_users` by matching `User.github_username` or email fallback; sets role “Project User”. + - `sync_all_repositories()`: + - Hourly scheduler job; iterates all `Repository` and calls `sync_repo`. + - Updates `GitHub Settings.last_sync`. +- Analytics and webhooks: + - `get_repository_activity(repository, days=30)`: + - Summaries: commit count, issues count (excluding PRs), pulls count, returns details preview. + - `create_repository_webhook(repo_full_name, webhook_url=None, events=None)`, `list_repository_webhooks(repo_full_name)`. + +### webhooks.py (Inbound GitHub webhooks) +- Entry: `github_webhook()` (guest allowed) + - Validates HMAC signature with `webhook_secret` if present using `X-Hub-Signature-256`. + - Robust header extraction `X-GitHub-Event` with fallbacks; infers event if header absent. + - Ensures `repository.full_name` exists locally; processes events inline (not background) for reliability. +- Handlers: + - `_handle_issues_event`: upsert/delete `Repository Issue` and assignees based on `action`. + - `_handle_pull_request_event`: upsert `Repository Pull Request` and reviewers based on `action`. + - `_handle_push_event`: updates `Repository.last_synced` and branch commit SHA/`last_updated`; adds branch if new. + - `_handle_member_event`: add/remove collaborators in `members_table`. + - `_handle_repository_event`: updates repo attributes; handles rename (`full_name`, `repo_name`, `repo_owner`, `url`). + +### api.py (ERPNext-facing helpers) +- Form validation and UX: + - `validate_repository(doc, method)`: validates `full_name` and populates `repo_owner`, `repo_name`, `url`. + - `get_repository_dashboard_data(data)`: defines dashboard sections for `Repository`. +- User and project flows: + - `get_user_repositories()`: returns repos accessible to current user: + - All if `GitHub Admin` + - Project manager of linked projects + - Membership match by SQL on `members` JSON (optimizes for string search on `github_username`). + - `sync_user_github_profile()`: fills `User` fields from GitHub (name/bio/location) via username. + - `link_github_user_to_erp(github_username, erp_user)`: sets a user’s GitHub username. + - `get_repository_statistics(repo_full_name)`: counts issues/PRs by state, branches, members. + - `create_project_from_repository(repo_full_name, project_name=None)`: creates `Project` linked to repo. +- Bulk import: + - `bulk_import_github_data(repo_full_name, import_type, force_update=False)`: + - `issues`: imports all non-PR issues (state=all). + - `pull_requests`: imports all PRs (state=all). + - Upsert with `force_update` toggle; records import/update/skip/errors. + +## Client/UI Behavior + +### Repository form actions (`repository.js`) +- Buttons (under “GitHub” or “Actions” groups): + - Sync Repository Data → `github_api.sync_repo` + - Sync Members → `github_api.sync_repo_members` + - Create Issue → `github_api.create_issue` (prompt for title/body/assignees/labels) + - Create Pull Request → `github_api.create_pull_request` (prompt for title/head/base/body) + - Manage Access → `github_api.manage_repo_access` (collaborator/team add/remove, permission) + - Show Activity → `github_api.get_repository_activity` summary dialog + - Manage Webhooks → list existing hooks; create via `github_api.create_repository_webhook` + - Open in GitHub → opens `url` +- Auto-fill on `full_name` change: sets `repo_owner`, `repo_name`, `url`. + +### Project form (`public/js/project_client.js`) +- Buttons: + - Sync Members from Repository → `github_api.sync_repo_members` + - Sync Repository Data → `github_api.sync_repo` (note: the script uses `args: {repo_full_name: repo}` in one place and `sync_repo` expects `repository`; be mindful in use; the `Repository` form uses the correct signature) + +### Task form (`public/js/task_client.js`) +- Buttons when `github_repo` set: + - Create GitHub Issue → `github_api.create_issue` (prompts, sets `github_issue_number`) + - Create Pull Request → `github_api.create_pull_request` (prompts, sets `github_pr_number`) + - Assign Issue → `github_api.assign_issue` (prompts for assignees) + - Add PR Reviewer → `github_api.add_pr_reviewer` (prompts for reviewers) + +## Permissions Model +- `GitHub Admin` role: + - Created during install. + - Granted CRUD permissions on GitHub DocTypes via `Custom DocPerm`. + - Elevated actions require this role (bulk import, manage repo access, create/list webhooks, sync all). +- Access in business logic: + - `_require_github_admin()` guards admin-only endpoints. + - `_can_sync_repo()` allows either `GitHub Admin` or `Project.project_manager` of a project linked to that repository. +- Repository visibility is informational; actual GitHub API permissions are enforced via token scopes. + +## Security +- PAT stored in `GitHub Settings` as Password field; accessed via `get_password`. +- Webhook validation: HMAC-SHA256 using `webhook_secret` if configured; rejects invalid signatures. +- ERPNext permission checks: + - Creation of `Task`, `Project`, `User` edits guarded with `frappe.has_permission` checks in helpers. + +## Error Handling and Logging +- API errors: + - `github_client.github_request` raises Frappe exceptions on HTTP errors; retries on 403 rate-limit with backoff; paginates automatically. +- Webhooks: + - Extensive `frappe.log_error` for missing data, unhandled events, processing errors; commits after each upsert to reduce partial failures. +- Bulk import and sync: + - Records skipped/updated/imported counts; logs individual errors per record. + +## Rate Limiting and Performance +- Client handles `X-RateLimit-Remaining` and `X-RateLimit-Reset` with sleep-and-retry. +- Pagination used for list endpoints. +- Scheduler runs hourly `sync_all_repositories`; prefer using webhooks for near real-time updates. +- Activity endpoint returns only small previews in `details`. + +## API Endpoints (Whitelisted Methods) +- Connection/lookup: + - `github_api.test_connection()` + - `github_api.get_github_username_by_email(email)` +- Listing: + - `github_api.list_repositories(organization=None)` + - `github_api.list_branches(repo_full_name, per_page=100)` + - `github_api.list_teams(org_name, per_page=100)` + - `github_api.list_repo_members(repo_full_name, per_page=100)` +- Repo lifecycle: + - `github_api.sync_repo(repository)` + - `github_api.sync_repo_members(repo_full_name)` + - `github_api.sync_all_repositories()` + - `github_api.manage_repo_access(repo_full_name, action, identifier, permission='push')` + - `github_api.create_repository_webhook(repo_full_name, webhook_url=None, events=None)` + - `github_api.list_repository_webhooks(repo_full_name)` + - `github_api.get_repository_activity(repository, days=30)` +- Issues: + - `github_api.create_issue(repository, title, body=None, assignees=None, labels=None)` + - `github_api.bulk_create_issues(repository, issues)` + - `github_api.assign_issue(repo_full_name, issue_number, assignees)` +- Pull requests: + - `github_api.create_pull_request(repository, title, head, base, body=None)` + - `github_api.add_pr_reviewer(repo_full_name, pr_number, reviewers)` +- ERP-side helpers (api.py): + - `api.get_user_repositories()` + - `api.sync_user_github_profile()` + - `api.create_task_from_github_issue(issue_name, task_title=None)` + - `api.bulk_import_github_data(repo_full_name, import_type, force_update=False)` (admin) + - `api.link_github_user_to_erp(github_username, erp_user)` + - `api.get_repository_statistics(repo_full_name)` + - `api.create_project_from_repository(repo_full_name, project_name=None)` + - `api.can_user_sync_repo(repo_full_name)` + +## Webhooks Integration +- Endpoint: `/api/method/erpnext_github_integration.webhooks.github_webhook` +- Events handled: + - `issues`: open/edit/reopen/close/delete → upsert/delete `Repository Issue` + assignees. + - `pull_request`: open/edit/reopen/close/merged → upsert `Repository Pull Request` + reviewers. + - `push`: updates branch commit SHA and `last_updated`. + - `member`: add/remove collaborator in `members_table`. + - `repository`: `edited`/`renamed` → update repo attributes and `full_name`. +- Security: + - Verify `X-Hub-Signature-256` with configured secret. +- Operational note: + - Processing is synchronous (“immediate”) to avoid background job issues; could be moved to background once stable. + +## Desk/UI Highlights +- `Repository` dashboard shows “Issues & PRs” and “Project Management” links. +- Many UI actions are provided as custom buttons on `Repository`, `Project`, and `Task` for convenience. +- `Repository.full_name` drives owner/name/url auto-fill. + +## Known Edge Cases and Notes +- Timezone conversion uses IST; adjust if a different local timezone is desired. +- Some client scripts pass `repo_full_name` instead of `repository` for `sync_repo` arguments; ensure to call with `repository=` for correctness as per `github_api.sync_repo`. +- GitHub “issues” API includes PRs; code filters PRs out when needed. +- `get_user_repositories()` uses a SQL LIKE on JSON field for members matching; effective but not relationally strict. + +## Extensibility +- Add new DocTypes for additional GitHub entities (e.g., labels, milestones) following the same pattern (create list API call, mirror locally in child tables). +- Add background job queues to decouple webhook processing for high volume. +- Extend `GitHub Settings` with rate-limit thresholds, default sync intervals, custom event subscriptions. +- Implement OAuth if needed: settings fields are present; add OAuth flow endpoints to exchange and store tokens per user or app. + +## Quick Start +- Install the app via bench; run migrations; ensure `GitHub Settings` has a valid PAT and webhook secret. +- Create `Repository` records with `full_name` values, or use `github_api.fetch_all_repositories(organization=...)` to populate. +- Click “Create Repository Webhook” from a `Repository` form (Manage Webhooks) or configure in GitHub manually to point to the webhook URL. +- Use “Sync Repository Data” to prime local data; rely on webhooks + hourly scheduler for continuous updates. \ No newline at end of file