New: /edit/ Service (for editing .txt and .src in /media/source) - #368
New: /edit/ Service (for editing .txt and .src in /media/source)#368calaldees wants to merge 3 commits into
/edit/ Service (for editing .txt and .src in /media/source)#368Conversation
shish
left a comment
shish
left a comment
There was a problem hiding this comment.
I cherry-picked and merged several smaller parts of this to make it easier to focus on the main code~
FYI
without sources
2.8M tracks.json
280K tracks.json.br
352K tracks.json.gz
with sources
3.2M tracks.json
328K tracks.json.br
412K tracks.json.gz
|
|
||
|
|
||
| @litestar.get( | ||
| path="/files.json", |
There was a problem hiding this comment.
For the other backend bits I've found it easier to use full paths at all times (eg this being /api/edit/files.json rather than /files.json), then setting the frontend (eg Caddy) to pass the full path through instead of just the suffix -- that way the URL is always /api/edit/files.json, in both dev and prod, publicly and privately, internally and externally - and there's no need to handle special cases or configure it differently in different situations
| tags=("Public",), | ||
| summary="Single File Contents", | ||
| description=textwrap.dedent(""" | ||
| """), |
There was a problem hiding this comment.
If all the methods are public, the summary says the same as the method name, and it's simple enough to not need an extra description, I'd simplify this to just path and media_type 👀
| file_path_backup = self._get_file_backup(file_path) | ||
| file_path.copy(file_path_backup) |
There was a problem hiding this comment.
git commit $path -m 'changed via webui' feels better to me than adding our own version control system on top of the existing version control system
- full history
- can include extra metadata like author
- we aren't creating extra files which then need to be ignored by humans and code
- one workflow for looking at older versions (as opposed to eg checking
.oldfiles for some changes, and checkinggit logfor other changes, and trying to figure out which of the two version control systems has the most up to date data) - no need to create new tools (things like "revert to the version from $DATE" already exist)
(technically we already have two version control systems, because syncthing is set to do versioned backups of the whole source folder, including the .git folder, so this would be our third nested VCS)
| FROM base AS dependencies | ||
| COPY pyproject.toml uv.lock ./ | ||
| RUN UV_NO_SYNC=False uv sync --no-dev | ||
| FROM dependencies AS test_dependencies | ||
| RUN UV_NO_SYNC=False uv sync --all-extras | ||
|
|
||
| FROM base AS production | ||
| VOLUME /logs | ||
| VOLUME /queues | ||
| VOLUME /media/processed | ||
| EXPOSE 8000 | ||
| CMD ["python", "-m", "sanic", "api_queue.server.app", "--host", "0.0.0.0", "--port", "8000", "--single-process"] | ||
| # "--workers", "4" # workers are python.multithreading rather than async. For now, we can just stick with single process | ||
| FROM dependencies AS code | ||
| COPY ./api_queue ./api_queue | ||
| COPY ./*.py ./ | ||
|
|
||
| FROM test_dependencies AS test | ||
| COPY --from=code /app . | ||
| COPY ./tests . |
There was a problem hiding this comment.
Trying to write a diagram so that I understand the layers here:
base
\- dependencies
|- test_dependencies
| \- test (technically "test" is a child of "test_dependencies"
| _and_ a child of "code", because of "COPY --from")
\- code
\- production
What are each of the layers for? Why do we need to copy data sideways between different branches of the tree?
I like the simplicity of
base (includes prod-dependencies and prod-code)
|- test (adds test-dependencies and test-code on top)
\- production (no code changes)
| $button_save.dataset.file = file | ||
| $file_title.textContent = file | ||
| $file_content.value = `Loading ${file}` | ||
| fetch(`/file/${file}`) |
There was a problem hiding this comment.
(An example of code which works when run stand-alone, but breaks in production, because in prod the url is /api/edit/file/...)
There was a problem hiding this comment.
Yes!💯
I will make this relative
Using uv and litestar api_edit now lists files Untangled litestar with file model Tweaked tools title Edit service now functions in a basic way Added `source`filename to `tracks.json` api_edit as part of stack api_edit: Added backup file rotation Error state in client Some dockerignore tweaks for completleness logger fix Added api_edit build action Corrected api_queue build path api_edit now shows track being edited Attempt to move api_queue to uv (unfinshed) Updated docs to include api_edit un-ignored `uv.lock` (we already use package-lock.json so why not go all in with python - modern dev tools and their bloat) Trying to get api_queue to run tests again Moved api_queue to uv Had to disable tests on docker build, but the test work when shelling into the container fix tests in docker api_edit tests (still need to fix the compose tests - can't access the container)
* paths are absolute from top level `/api/edit/` * Deprecated file_rotation backup and moving towards `git commit xxx` * Queue Dockerfile can now run tests (no idea what jibble was messin with it 2 months ago) * api_edit now gives warning on startup for no files (and skips folders with '.' it was crawling all of .venv * trying to pannelbeat the tests into working again
Working towards git backup
2 participants
Added the new
/edit/service to edit.txtand.srtfiles on the live server without contributors having to download all of/media/sourceFings wot goin dan in diz PR:
/api_edit/NEW!litestaranduvmodel.pyan abstraction to list, read, write files in/media/sourceORIGINAL_FILENAME.1.oldand cycles/overwrites these based on mtimeindex.htmlzero dependency vanilla js mini POS frontend (but it works).api_queueupload! Gah!RUN. Sanic has some weirdness. I've disabled the tests on build (which is not ideal)uv(in prep for litestar move)uv.lockrather thanpyproject.toml(We can still lock major versions in pyproject if needed, but most dependencies can be fluid andprocessmedia3sourcetotracks.json.tracks.jsonis. My assumption is that it will compress very very well.caddycorrectly. This will need checking as I was unable to run it locally (some cert issue? I will ask for some gudience)/tools/index.htmlProbably worth squashing the commits on merge because the commit messages are a mess.