Skip to content
This repository was archived by the owner on Apr 10, 2026. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 3 additions & 38 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
Expand Down Expand Up @@ -49,7 +50,6 @@ coverage.xml
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
Expand All @@ -72,7 +72,6 @@ instance/
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
Expand All @@ -83,9 +82,7 @@ profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
Expand All @@ -94,22 +91,7 @@ ipython_config.py
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
Expand Down Expand Up @@ -145,20 +127,3 @@ dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

# project-specific
/pylon/requirements/*
!/pylon/requirements/.gitkeep
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# VirtualEnv
Use venv from .venv
78 changes: 76 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,76 @@
# test_project
Test task repository (temporary)
# Arbiter
Distributed tasks queue use Redis as broker. Consists of arbiter and minion.

## Installation

Clone git repo
```bash
git clone https://github.com/carrier-io/arbiter.git
```
and install by running
```bash
cd arbiter
python setup.py install
```

## Basic scenario

Launch Redis, as it required for everything to work
```bash
docker run -d --rm --hostname arbiter-redis --name arbiter-redis \
-p 6379:6379 redis:alpine redis-server
```

### Create simple task
You need to initiate minion and provide connection details:
```python
from arbiter import RedisEventNode
from arbiter import Minion

event_node = RedisEventNode(host="localhost", port=6379, password="", event_queue="tasks")
app = Minion(event_node, queue="default")
```
then you can declare tasks by decorating callable with `@app.task`
```python
@app.task(name="simple_add")
def adds(x, y):
return x + y
```
Every task need to have a name, which it will be referred by when initiated from arbiter.
this is pretty much it to create first task

Now we need to create execution point
```python
if __name__ == "__main__":
app.run(workers=3)
```
where `workers` is a quantity of worker slots to do the job(s)

Run created script. Minion is ready to accept work orders.

### Call created task from arbiter
Arbiter is job initiator, it maintain the state of all jobs it created and can retrieve results.

Each arbiter have it's own communication channel, so job results won't mess between two different arbiters

Declaring the arbiter
```python
from arbiter import RedisEventNode
from arbiter import Arbiter

event_node = RedisEventNode(host="localhost", port=6379, password="", event_queue="tasks")
arbiter = Arbiter(event_node)
```
to call the task and track it till it done (tasks are obviously async)
```python
task_keys = arbiter.apply("simple_add", tasks_count=1, task_args=[1, 2]) # will return array of task ids

# while loop with returns results of each task once it done
for message in arbiter.wait_for_tasks(task_keys):
print(message)
```
Alternatively you can get task result by calling
```python
arbiter.status(task_keys[0])
```
it will return `json` where `result` will be one of the keys
27 changes: 27 additions & 0 deletions arbiter/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/python3
# coding=utf-8
# pylint: disable=C0114

# Copyright 2023 getcarrier.io
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


from arbiter.arbiter import Arbiter
from arbiter.minion import Minion
from arbiter.task import Task

from arbiter.eventnode import make_event_node
from arbiter.eventnode import MockEventNode

from arbiter.tasknode import TaskNode
Loading