Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Jekyll build output
_site/
.jekyll-metadata
.jekyll-cache/

# Sass cache
.sass-cache/

# Local configuration overrides (if any)
_config.yml.tmp
_config.yml.dev

# OS-specific files
.DS_Store
Thumbs.db
255 changes: 255 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,255 @@
GEM
remote: https://rubygems.org/
specs:
addressable (2.9.0)
public_suffix (>= 2.0.2, < 8.0)
base64 (0.3.0)
bigdecimal (4.1.2)
colorator (1.1.0)
concurrent-ruby (1.3.7)
csv (3.3.5)
em-websocket (0.5.3)
eventmachine (>= 0.12.9)
http_parser.rb (~> 0)
eventmachine (1.2.7)
ffi (1.17.4)
ffi (1.17.4-aarch64-linux-gnu)
ffi (1.17.4-aarch64-linux-musl)
ffi (1.17.4-arm-linux-gnu)
ffi (1.17.4-arm-linux-musl)
ffi (1.17.4-arm64-darwin)
ffi (1.17.4-x86-linux-gnu)
ffi (1.17.4-x86-linux-musl)
ffi (1.17.4-x86_64-darwin)
ffi (1.17.4-x86_64-linux-gnu)
ffi (1.17.4-x86_64-linux-musl)
forwardable-extended (2.6.0)
google-protobuf (4.35.1)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-aarch64-linux-gnu)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-aarch64-linux-musl)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-arm64-darwin)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-x86-linux-gnu)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-x86-linux-musl)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-x86_64-darwin)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-x86_64-linux-gnu)
bigdecimal
rake (~> 13.3)
google-protobuf (4.35.1-x86_64-linux-musl)
bigdecimal
rake (~> 13.3)
http_parser.rb (0.8.1)
i18n (1.15.2)
concurrent-ruby (~> 1.0)
jekyll (4.4.1)
addressable (~> 2.4)
base64 (~> 0.2)
colorator (~> 1.0)
csv (~> 3.0)
em-websocket (~> 0.5)
i18n (~> 1.0)
jekyll-sass-converter (>= 2.0, < 4.0)
jekyll-watch (~> 2.0)
json (~> 2.6)
kramdown (~> 2.3, >= 2.3.1)
kramdown-parser-gfm (~> 1.0)
liquid (~> 4.0)
mercenary (~> 0.3, >= 0.3.6)
pathutil (~> 0.9)
rouge (>= 3.0, < 5.0)
safe_yaml (~> 1.0)
terminal-table (>= 1.8, < 4.0)
webrick (~> 1.7)
jekyll-default-layout (0.1.5)
jekyll (>= 3.0, < 5.0)
jekyll-feed (0.17.0)
jekyll (>= 3.7, < 5.0)
jekyll-include-cache (0.2.1)
jekyll (>= 3.7, < 5.0)
jekyll-sass-converter (3.1.0)
sass-embedded (~> 1.75)
jekyll-seo-tag (2.9.0)
jekyll (>= 3.8, < 5.0)
jekyll-watch (2.2.1)
listen (~> 3.0)
json (2.20.0)
just-the-docs (0.12.0)
jekyll (>= 3.8.5)
jekyll-include-cache
jekyll-seo-tag (>= 2.0)
rake (>= 12.3.1)
kramdown (2.5.2)
rexml (>= 3.4.4)
kramdown-parser-gfm (1.1.0)
kramdown (~> 2.0)
liquid (4.0.4)
listen (3.10.0)
logger
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
logger (1.7.0)
mercenary (0.4.0)
pathutil (0.16.2)
forwardable-extended (~> 2.6)
public_suffix (7.0.5)
rake (13.4.2)
rb-fsevent (0.11.2)
rb-inotify (0.11.1)
ffi (~> 1.0)
rexml (3.4.4)
rouge (4.7.0)
safe_yaml (1.0.5)
sass-embedded (1.101.0)
google-protobuf (~> 4.31)
rake (>= 13)
sass-embedded (1.101.0-aarch64-linux-android)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-aarch64-linux-gnu)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-aarch64-linux-musl)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-arm-linux-androideabi)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-arm-linux-gnueabihf)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-arm-linux-musleabihf)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-arm64-darwin)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-riscv64-linux-android)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-riscv64-linux-gnu)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-riscv64-linux-musl)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-x86_64-darwin)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-x86_64-linux-android)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-x86_64-linux-gnu)
google-protobuf (~> 4.31)
sass-embedded (1.101.0-x86_64-linux-musl)
google-protobuf (~> 4.31)
terminal-table (3.0.2)
unicode-display_width (>= 1.1.1, < 3)
unicode-display_width (2.6.0)
webrick (1.9.2)

PLATFORMS
aarch64-linux-android
aarch64-linux-gnu
aarch64-linux-musl
arm-linux-androideabi
arm-linux-gnu
arm-linux-gnueabihf
arm-linux-musl
arm-linux-musleabihf
arm64-darwin
riscv64-linux-android
riscv64-linux-gnu
riscv64-linux-musl
ruby
x86-linux-gnu
x86-linux-musl
x86_64-darwin
x86_64-linux-android
x86_64-linux-gnu
x86_64-linux-musl

DEPENDENCIES
jekyll
jekyll-default-layout
jekyll-feed
jekyll-seo-tag
just-the-docs

CHECKSUMS
addressable (2.9.0) sha256=7fdf6ac3660f7f4e867a0838be3f6cf722ace541dd97767fa42bc6cfa980c7af
base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b
bigdecimal (4.1.2) sha256=53d217666027eab4280346fba98e7d5b66baaae1b9c3c1c0ffe89d48188a3fbd
bundler (4.0.15) sha256=a4ceb882fe94a0e0ac63cd0813932bbfd631a14e5ac0b7975189b19a4d28d9e7
colorator (1.1.0) sha256=e2f85daf57af47d740db2a32191d1bdfb0f6503a0dfbc8327d0c9154d5ddfc38
concurrent-ruby (1.3.7) sha256=4412caec3a5ea2e5fdc52076724c071a81f2c0593d83b2ac8cbb8ca63b3151b0
csv (3.3.5) sha256=6e5134ac3383ef728b7f02725d9872934f523cb40b961479f69cf3afa6c8e73f
em-websocket (0.5.3) sha256=f56a92bde4e6cb879256d58ee31f124181f68f8887bd14d53d5d9a292758c6a8
eventmachine (1.2.7) sha256=994016e42aa041477ba9cff45cbe50de2047f25dd418eba003e84f0d16560972
ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
ffi (1.17.4-aarch64-linux-gnu) sha256=b208f06f91ffd8f5e1193da3cae3d2ccfc27fc36fba577baf698d26d91c080df
ffi (1.17.4-aarch64-linux-musl) sha256=9286b7a615f2676245283aef0a0a3b475ae3aae2bb5448baace630bb77b91f39
ffi (1.17.4-arm-linux-gnu) sha256=d6dbddf7cb77bf955411af5f187a65b8cd378cb003c15c05697f5feee1cb1564
ffi (1.17.4-arm-linux-musl) sha256=9d4838ded0465bef6e2426935f6bcc93134b6616785a84ffd2a3d82bc3cf6f95
ffi (1.17.4-arm64-darwin) sha256=19071aaf1419251b0a46852abf960e77330a3b334d13a4ab51d58b31a937001b
ffi (1.17.4-x86-linux-gnu) sha256=38e150df5f4ca555e25beca4090823ae09657bceded154e3c52f8631c1ed72cf
ffi (1.17.4-x86-linux-musl) sha256=fbeec0fc7c795bcf86f623bb18d31ea1820f7bd580e1703a3d3740d527437809
ffi (1.17.4-x86_64-darwin) sha256=aa70390523cf3235096cf64962b709b4cfbd5c082a2cb2ae714eb0fe2ccda496
ffi (1.17.4-x86_64-linux-gnu) sha256=9d3db14c2eae074b382fa9c083fe95aec6e0a1451da249eab096c34002bc752d
ffi (1.17.4-x86_64-linux-musl) sha256=3fdf9888483de005f8ef8d1cf2d3b20d86626af206cbf780f6a6a12439a9c49e
forwardable-extended (2.6.0) sha256=1bec948c469bbddfadeb3bd90eb8c85f6e627a412a3e852acfd7eaedbac3ec97
google-protobuf (4.35.1) sha256=a3a6471331d918f58dfa4d014a8f6286f0af2cf4840216bde52fcf2ea3fe3726
google-protobuf (4.35.1-aarch64-linux-gnu) sha256=50ca44d0eeff3f8475e630a1accdd974256f3510694d574e2c9d6119ea8bc9e1
google-protobuf (4.35.1-aarch64-linux-musl) sha256=d5c65cef6bd6498a9e5ed5f88cf6cf7e341c10b0a005e32137d5d1a2b6e8c18a
google-protobuf (4.35.1-arm64-darwin) sha256=d9c957df04fa89c749fa9a72a7b383eb4296efc9b2303dc6fd6fbe39c698ad6b
google-protobuf (4.35.1-x86-linux-gnu) sha256=cc7492566b27ad8b5dfa3a7b6f9b11e905050cc53c2fa8ff22de375a9e7a70fb
google-protobuf (4.35.1-x86-linux-musl) sha256=ab403790b59e4dc588ffbed1eaacf05d4ca2f0d12ac9c13d6c64e69380d8c99e
google-protobuf (4.35.1-x86_64-darwin) sha256=66b62b4df00931018a692806df66393efa960d6d2b7da69735187249f950d3ee
google-protobuf (4.35.1-x86_64-linux-gnu) sha256=c786439087512a3fbd199e9897d265b855f951d4027e218ea55e858d45969edd
google-protobuf (4.35.1-x86_64-linux-musl) sha256=91890eb0002934a339fdb7d77a147c46b7474b6799db27872b747b905837f744
http_parser.rb (0.8.1) sha256=9ae8df145b39aa5398b2f90090d651c67bd8e2ebfe4507c966579f641e11097a
i18n (1.15.2) sha256=00f9eb62412fe593b2a65a97daa75300d37abb8f7202ec748e94b6d46a9dd1b5
jekyll (4.4.1) sha256=4c1144d857a5b2b80d45b8cf5138289579a9f8136aadfa6dd684b31fe2bc18c1
jekyll-default-layout (0.1.5) sha256=c626be4e4a5deafca123539da2cd22ff873be350cafd4da134039efdf24320af
jekyll-feed (0.17.0) sha256=689aab16c877949bb9e7a5c436de6278318a51ecb974792232fd94d8b3acfcc3
jekyll-include-cache (0.2.1) sha256=c7d4b9e551732a27442cb2ce853ba36a2f69c66603694b8c1184c99ab1a1a205
jekyll-sass-converter (3.1.0) sha256=83925d84f1d134410c11d0c6643b0093e82e3a3cf127e90757a85294a3862443
jekyll-seo-tag (2.9.0) sha256=0260015a8e1df9bf195cdfb0c675b7b2883fd8cbf12556e1c1cbe36a831c6852
jekyll-watch (2.2.1) sha256=bc44ed43f5e0a552836245a54dbff3ea7421ecc2856707e8a1ee203a8387a7e1
json (2.20.0) sha256=9362bc6e55a952b056abf9167cf053358181c904cb70cd6eee0808ea830fc32b
just-the-docs (0.12.0) sha256=15f2839ac9082898d60f33b978aa6f8e46fc50ba8fac20ae7a7f0e1fb295523e
kramdown (2.5.2) sha256=1ba542204c66b6f9111ff00dcc26075b95b220b07f2905d8261740c82f7f02fa
kramdown-parser-gfm (1.1.0) sha256=fb39745516427d2988543bf01fc4cf0ab1149476382393e0e9c48592f6581729
liquid (4.0.4) sha256=4fcfebb1a045e47918388dbb7a0925e7c3893e58d2bd6c3b3c73ec17a2d8fdb3
listen (3.10.0) sha256=c6e182db62143aeccc2e1960033bebe7445309c7272061979bb098d03760c9d2
logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
mercenary (0.4.0) sha256=b25a1e4a59adca88665e08e24acf0af30da5b5d859f7d8f38fba52c28f405138
pathutil (0.16.2) sha256=e43b74365631cab4f6d5e4228f812927efc9cb2c71e62976edcb252ee948d589
public_suffix (7.0.5) sha256=1a8bb08f1bbea19228d3bed6e5ed908d1cb4f7c2726d18bd9cadf60bc676f623
rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701
rb-fsevent (0.11.2) sha256=43900b972e7301d6570f64b850a5aa67833ee7d87b458ee92805d56b7318aefe
rb-inotify (0.11.1) sha256=a0a700441239b0ff18eb65e3866236cd78613d6b9f78fea1f9ac47a85e47be6e
rexml (3.4.4) sha256=19e0a2c3425dfbf2d4fc1189747bdb2f849b6c5e74180401b15734bc97b5d142
rouge (4.7.0) sha256=dba5896715c0325c362e895460a6d350803dbf6427454f49a47500f3193ea739
safe_yaml (1.0.5) sha256=a6ac2d64b7eb027bdeeca1851fe7e7af0d668e133e8a88066a0c6f7087d9f848
sass-embedded (1.101.0) sha256=57dbc3409e2c0a2c581a4c9945c2bd72ec88e71ec98017bd02dd1da8a76b22f4
sass-embedded (1.101.0-aarch64-linux-android) sha256=ad0a35c6ff5cdc4e31c5d8261a768ef460c6c7d1458081183e42170e1474c4b5
sass-embedded (1.101.0-aarch64-linux-gnu) sha256=8f6926349e880dbb4fda75fb182086fb60aa821b756bcbfc8e5b2b23b1f269d6
sass-embedded (1.101.0-aarch64-linux-musl) sha256=0a02b4b75db305160db1b0512f040762242128a9b5c01b0fa7a504f2b24557f2
sass-embedded (1.101.0-arm-linux-androideabi) sha256=88e762531f90d4a2eef55ffafeba337585e1bb24b22307ebd7fe5505de9a4476
sass-embedded (1.101.0-arm-linux-gnueabihf) sha256=358df7f98d13ff53739ebc45f5c5acd4dd96833bf87f39c74d11f798081d4158
sass-embedded (1.101.0-arm-linux-musleabihf) sha256=d993a3ad017250d46d1312bc10cfded5ab4585906d37a60a56494bd662182d3a
sass-embedded (1.101.0-arm64-darwin) sha256=9fed684380b49499dfc856aba0d026a0748f924bd78044ffff2bae1537aea73e
sass-embedded (1.101.0-riscv64-linux-android) sha256=1bd35527e1ca24af6aa2ba9db797bc9b72331ebbec9ff6e99ea02b24f83b3da1
sass-embedded (1.101.0-riscv64-linux-gnu) sha256=7805a249ed4c31b58b2f7d054d14f0ed24c61a836f579cd76ed2367088142bcf
sass-embedded (1.101.0-riscv64-linux-musl) sha256=8bd60eb9ebe6f9f2740a4e9ddb792a3db1fbba282e3375103844b4979eb7c731
sass-embedded (1.101.0-x86_64-darwin) sha256=20ff4afd7c052b3f8d7b4abe511e8114985a07dff7616750a43ee1fc2759fb28
sass-embedded (1.101.0-x86_64-linux-android) sha256=472b72114ededa00a22e13553dd31f98d1a92d52bdbe2e738a4dbac2dd50c779
sass-embedded (1.101.0-x86_64-linux-gnu) sha256=ff48452b2351eaaf4e6e02f59de0e9e35f6f163e8456d520db49b4d87cf73c27
sass-embedded (1.101.0-x86_64-linux-musl) sha256=2286feef3c7a8d9bbb075aa1651e1a81942aca1213f7c2d30cc1d6f4eb5b4104
terminal-table (3.0.2) sha256=f951b6af5f3e00203fb290a669e0a85c5dd5b051b3b023392ccfd67ba5abae91
unicode-display_width (2.6.0) sha256=12279874bba6d5e4d2728cef814b19197dbb10d7a7837a869bab65da943b7f5a
webrick (1.9.2) sha256=beb4a15fc474defed24a3bda4ffd88a490d517c9e4e6118c3edce59e45864131

BUNDLED WITH
4.0.15
51 changes: 25 additions & 26 deletions architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,24 @@ nav_order: 3
- TOC
{:toc}

Before we can delve into how MASs connect to the DiSSCo architecture, it is useful to know how the
DiSSCo architecture operates.
Before we can delve into how MASs connect to the DiSSCo Architecture, it is useful to know how the
DiSSCo Architecture operates.

As a platform, DiSSCo sits in between data providers and data consumers. The core architecture adds
value by harmonizing data to openDS, minting unique identifiers for specimens and media objects,
As a platform, DiSSCo sits in-between data providers and data consumers. The Core Architecture adds
value by harmonising data to OpenDS, minting unique identifiers for specimens and media objects,
capturing provenance, and, of course, facilitating annotations. As DiSSCo improves the data, DiSSCo
will send this enhanced data upstream, back to the data providers, and downstream, to data
aggregators (though this functionality is currently in development).

Due to its flexible nature, additional services can be built on top of the DiSSCo core architecture.
Due to its flexible nature, additional services can be built on top of the DiSSCo Core Architecture.
A MAS is one such service. What a MAS does is essentially a "black box" from the DiSSCo
Architecture's perspective. A MAS receives a message as input and produces an annotation as an
Architecture's perspective. A MAS receives a message as input and produces an annotation as
output.

The DiSSCo position in the biodiversity data landscape is illustrated in the following diagram. On
the left, data providers send data to the core infrastructure. In the middle, machine and human
agents rest on top of the
core infrastructure, adding additional value. On the right, the enriched data is sent to data
agents provide additional value, resting on top of the
core infrastructure. On the right, the enriched data is sent to data
consumers.

![DiSSCo's position in the biodiversity data landscape](assets/dissco_arch.png)
Expand All @@ -39,28 +39,25 @@ Other value services of DiSSCo:
- Harmonising data to OpenDS
- Minting Persistent Identifiers (PIDs) for resources
- Versioning objects and maintaining a provenance record
- Extending data and linking
- Extending and linking data
- Exposing data through APIs, including a robust search function

# Machine Annotation Services

A Machine Annotation Service (MAS) is an automated service that annotates a target in DiSSCo. Most
often, MASs are scheduled by a user on a specific target - either a digital specimen or a media
object -- either through the DiSSCover platform or programmatically through the DiSSCo API.
A Machine Annotation Service (MAS) is an automated service that annotates a target in DiSSCo. Most
often, MASs are scheduled by a user on a specific target (a digital specimen or a media
object)—either through the DiSSCover platform or programmatically through the DiSSCo API.

{: .note }

> When an existing service is adapted to work within DiSSCo, there are two components involved:
>
> - **Value Service**: This is the original service being adapted to DiSSCo. It is deployed on
infrastructure separate from the core DiSSCo architecture, and should be accessible through
APIs.
> - **MAS Middleware**: This is a lightweight component containerized and deployed on the DiSSCo
core
architecture.
> - **Value Service**: This is the original service being adapted to DiSSCo. It is deployed on infrastructure separate from the DiSSCo Core Architecture, and should be accessible through APIs.
> - **MAS Middleware**: This is a lightweight component containerised and deployed on the DiSSCo Core Architecture.

## When a machine makes an annotation
## When a Machine Makes an Annotation

The flow of data can be summarised below:
The flow of data is summarised below:

**1. The user requests a job**

Expand All @@ -73,41 +70,43 @@ DiSSCo uses [Apache Kafka](https://kafka.apache.org/) as an asynchronous messagi
machine annotation is requested, a "job" is created. A UUID is generated and associated with this
process to track the state of the job. When it is first created, the job is marked as `SCHEDULED`.

The backend sends a message through Kafka. This message that includes the job ID and the target of
The backend sends a message through Kafka. This message includes the job ID and the target of
the annotation. The message has a topic that identifies the message's destination. Each MAS is given
its own unique topic by DiSSCo.

**3. The job is received by the MAS middleware**

Listening for a Kafka message on its topic, the MAS receives the message.
After it started listening for a Kafka message on its topic, the MAS eventually receives the message.

**3a. (Recommended) The MAS middleware informs the backend the job was received.**
**3a. (Recommended) The MAS middleware informs the backend the job was received**

It is strongly recommended that once the MAS receives the message, it informs the backend. This
improves user experience. The MAS sends the job id to the `/running` endpoint. The backend, in turn,
marks the job as `RUNNING` and informs the user.

**4. The MAS middleware calls the value service**

The MAS middleware extracts relevant information from the target and sends it to the value service,
usually through an API call. The results from the value service are formatted into an annotation
event.

**5. The results are sent to DiSSCo**

The MAS middelware sends the formatted annotation event through the `annotation` Kafka topic.

**6. The annotation is processed**

DiSSCo's annotation processing service picks up the message in the `annotation` topic and processes
it. The annotation is then available through the DiSSCo API and the DiSSCover interface.

**Congratulations! You've annotated an object!**

## MAS Deployment

The MAS middleware application is deployed as a container on the DiSSCo core architecture. The value
The MAS middleware application is deployed as a container on the DiSSCo Core Architecture. The value
service will remain deployed where it was originally.

{: .note}
The MAS middleware is deployed on the DiSSCo network. If your value service requires whitelisting a
specific IP address, let the DiSSCo team know so they can provide your MAS middleware with a static
IP address. That way, the middleware may communicate with the value service.

IP address. That way, the middleware may communicate with the value service.
Loading