Skip to content
Snippets Groups Projects
Verified Commit 1994cff3 authored by Martin Weise's avatar Martin Weise
Browse files

Added docs

parent 1a283f46
No related branches found
No related tags found
3 merge requests!277Dev,!275Dev,!270Dev
Show whitespace changes
Inline Side-by-side
Showing
with 628 additions and 368 deletions
.docs/concepts/analysis.md → .docs/api/analyse-service.md
+ 0
− 2
View file @ 1994cff3
......@@ -2,8 +2,6 @@
author: Martin Weise
---
# Analyse Service
## tl;dr
!!! debug "Debug Information"
......
......
.docs/api/auth-service.md 0 → 100644
+ 212
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## tl;dr
!!! debug "Debug Information"
Image: [`dbrepo/authentication-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/authentication-service)
* Ports: 8080/tcp
* UI: `http://<hostname>/api/auth/admin/`
## Overview
By default, users are created using the [User Interface](../system-other-ui) and the sign-up page in the User Interface.
This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then
managed by the
Authentication Service.
## Groups
The authorization scheme follows a group-based access control (GBAC). Users are organized in three distinct
(non-overlapping) groups:
1. Researchers (*default*)
2. Developers
3. Data Stewards
Based on the membership in one of these groups, the user is assigned a set of roles that authorize specific actions. By
default, all users are assigned to the `researchers` group.
## Roles
We organize the roles into default- and escalated composite roles. There are three composite roles, one for each group.
Each of the composite role has a set of other associated composite roles.
<figure markdown>
![Grouped Roles](../images/groups-roles.png)
<figcaption>Three groups (Researchers, Developers, Data Stewards) and their composite roles associated.</figcaption>
</figure>
There is one role for one specific action in the services. For example: the `create-database` role authorizes a user to
create a database in a Docker container. Therefore,
the [`DatabaseEndpoint.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/a5bdd1e2169bae6497e2f7eee82dad8b9b059850/fda-database-service/rest-service/src/main/java/at/tuwien/endpoints/DatabaseEndpoint.java#L78)
endpoint requires a JWT access token with this authority.
```java
@PostMapping
@PreAuthorize("hasAuthority('create-database')")
public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
@Valid @RequestBody DatabaseCreateDto createDto,
@NotNull Principal principal) {
...
}
```
### Default Container Handling
| Name | Description |
|-------------------|-------------------------------|
| `find-container` | Can find a specific container |
| `list-containers` | Can list all containers |
### Default Database Handling
| Name | Description |
|------------------------------|------------------------------------------------------|
| `check-database-access` | Can check the access to a database of a user |
| `create-database` | Can create a database |
| `create-database-access` | Can give a new access to a database of a user |
| `delete-database-access` | Can delete the access to a database of a user |
| `find-database` | Can find a specific database in a container |
| `list-databases` | Can list all databases in a container |
| `modify-database-image` | Can update the database image |
| `modify-database-owner` | Can modify the database owner |
| `modify-database-visibility` | Can modify the database visibility (public, private) |
| `update-database-access` | Can update the access to a database of a user |
### Default Table Handling
| Name | Description |
|---------------------------------|------------------------------------------------------|
| `create-table` | Can create a table |
| `find-tables` | Can list a specific table in a database |
| `list-tables` | Can list all tables |
| `modify-table-column-semantics` | Can modify the column semantics of a specific column |
| `delete-table` | Can delete tables owned by the user in a database |
### Default Query Handling
| Name | Description |
|---------------------------|-----------------------------------------------|
| `create-database-view` | Can create a view in a database |
| `delete-database-view` | Can delete a view in a database |
| `delete-table-data` | Can delete data in a table |
| `execute-query` | Can execute a query statement |
| `export-query-data` | Can export the data that a query has produced |
| `export-table-data` | Can export the data stored in a table |
| `find-database-view` | Can find a specific database view |
| `find-query` | Can find a specific query in the query store |
| `insert-table-data` | Can insert data into a table |
| `list-database-views` | Can list all database views |
| `list-queries` | Can list all queries in the query store |
| `persist-query` | Can persist a query in the query store |
| `re-execute-query` | Can re-execute a query to reproduce a result |
| `view-database-view-data` | Can view the data produced by a database view |
| `view-table-data` | Can view the data in a table |
| `view-table-history` | Can view the data history of a table |
### Default Identifier Handling
| Name | Description |
|---------------------|---------------------------------------------|
| `create-identifier` | Can create an identifier (subset, database) |
| `find-identifier` | Can find a specific identifier |
| `list-identifier` | Can list all identifiers |
### Default User Handling
| Name | Description |
|---------------------------|-----------------------------------------|
| `modify-user-theme` | Can modify the user theme (light, dark) |
| `modify-user-information` | Can modify the user information |
### Default Maintenance Handling
| Name | Description |
|------------------------------|------------------------------------------|
| `create-maintenance-message` | Can create a maintenance message banner |
| `delete-maintenance-message` | Can delete a maintenance message banner |
| `find-maintenance-message` | Can find a maintenance message banner |
| `list-maintenance-messages` | Can list all maintenance message banners |
| `update-maintenance-message` | Can update a maintenance message banner |
### Default Semantics Handling
| Name | Description |
|---------------------------|-----------------------------------------------------------------|
| `create-semantic-unit` | Can save a previously unknown unit for a table column |
| `create-semantic-concept` | Can save a previously unknown concept for a table column |
| `execute-semantic-query` | Can query remote SPARQL endpoints to get labels and description |
| `table-semantic-analyse` | Can automatically suggest units and concepts for a table |
### Escalated User Handling
| Name | Description |
|-------------|-----------------------------------------------|
| `find-user` | Can list user information for a specific user |
### Escalated Container Handling
| Name | Description |
|--------------------|--------------------------|
| `create-container` | Can create a container |
| `delete-container` | Can delete any container |
### Escalated Database Handling
| Name | Description |
|-------------------|------------------------------------------|
| `delete-database` | Can delete any database in any container |
### Escalated Table Handling
| Name | Description |
|------------------------|--------------------------------------|
| `delete-foreign-table` | Can delete any table in any database |
### Escalated Query Handling
| Name | Description |
|------|-------------|
| / | |
### Escalated Identifier Handling
| Name | Description |
|------------------------------|---------------------------------------------------|
| `create-foreign-identifier` | Can create an identifier to any database or query |
| `delete-identifier` | Can delete any identifier |
| `modify-identifier-metadata` | Can modify any identifier metadata |
### Escalated Semantics Handling
| Name | Description |
|-----------------------------------------|----------------------------------------------|
| `create-ontology` | Can register a new ontology |
| `delete-ontology` | Can unregister an ontology |
| `list-ontologies` | Can list all ontologies |
| `modify-foreign-table-column-semantics` | Can modify any table column concept and unit |
| `update-ontology` | Can update ontology metadata |
| `update-semantic-concept` | Can update own table column concept |
| `update-semantic-unit` | Can update own table column unit |
## Limitations
* No support for sending e-mails through Keycloak by default.
* No support for temporary passwords.
* No support for adding identifies in Keycloak directly.
* No support for multi-factor authentication.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
1. Mount your TLS certificate / private key pair into `/app/tls.crt` and `/app/tls.key` and
set `KC_HTTPS_CERTIFICATE_FILE=/app/tls.crt` and set `KC_HTTPS_CERTIFICATE_KEY_FILE=/app/tls.key`.
.docs/concepts/message-broker.md → .docs/api/broker-service.md
+ 2
− 4
View file @ 1994cff3
......@@ -2,8 +2,6 @@
author: Martin Weise
---
# Broker Service
## tl;dr
!!! debug "Debug Information"
......@@ -40,14 +38,14 @@ queue `dbrepo`, connected with a binding of `dbrepo.#` which routes all tuples w
the dot!) to this queue.
<figure markdown>
![Data ingest](images/queue-quorum.png)
![Data ingest](../images/queue-quorum.png)
<figcaption>Replicated quorum queue dbrepo in a cluster with three nodes</figcaption>
</figure>
The consumer takes care of writing it to the correct table in the [Data Service](../system-services-data).
<figure markdown>
![Data ingest](images/exchange-binding.png)
![Data ingest](../images/exchange-binding.png)
<figcaption>Architecture Broker Service</figcaption>
</figure>
......
......
.docs/concepts/relational-database.md → .docs/api/data-db.md
+ 1
− 1
View file @ 1994cff3
......@@ -60,7 +60,7 @@ the [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
natively.
<figure markdown>
![Sidecar architecture detailed](images/architecture-data-db.svg)
![Sidecar architecture detailed](../images/architecture-data-db.svg)
<figcaption>Sidecar that handles the CSV-file upload/download.</figcaption>
</figure>
......
......
.docs/concepts/data.md → .docs/api/data-service.md
+ 0
− 4
View file @ 1994cff3
......@@ -2,8 +2,6 @@
author: Martin Weise
---
# Data Service
## tl;dr
!!! debug "Debug Information"
......@@ -26,8 +24,6 @@ Data Service up.
## Limitations
* No clear differentiation for data-handling endpoints from the [Metadata Service](../system-services-metadata) yet.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
......
......
.docs/concepts/gateway.md → .docs/api/gateway-service.md
+ 0
− 2
View file @ 1994cff3
......@@ -2,8 +2,6 @@
author: Martin Weise
---
# Gateway Service
## tl;dr
!!! debug "Debug Information"
......
......
.docs/usage-overview.md → .docs/api/index.md
+ 0
− 0
View file @ 1994cff3
File moved
.docs/api/metadata-db.md 0 → 100644
+ 5
− 0
View file @ 1994cff3
---
author: Martin Weise
---
TBD
\ No newline at end of file
.docs/api/metadata-service.md 0 → 100644
+ 144
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## tl;dr
!!! debug "Debug Information"
Image: [`dbrepo/metadata-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/metadata-service)
* Ports: 9099/tcp
* Info: `http://<hostname>:9099/actuator/info`
* Health: `http://<hostname>:9099/actuator/health`
- Readiness: `http://<hostname>:9099/actuator/health/readiness`
- Liveness: `http://<hostname>:9099/actuator/health/liveness`
* Prometheus: `http://<hostname>:9099/actuator/prometheus`
* Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
## Overview
This service manages the following topics:
* Databases
* Identifiers (DataCite, OAI-PMH)
* Queries
* Semantics (Ontologies)
* Tables
* Users
* Views
### Databases
The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and
data ingest operations.
### Identifiers
The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a database, subset,
table or view to obtain the metadata attached to it and allow reproduction of the exact same result.
This service also provides an OAI-PMH endpoint for metadata aggregators
(e.g. [OpenAIRE Graph](https://graph.openaire.eu/)). Through the User Interface, it also exposes metadata through
JSON-LD to metadata aggregators (e.g. [Google Datasets](https://datasetsearch.research.google.com/)). PID metadata
is always exposed, even for private databases.
The service generates internal PIDs, essentially representing internal URIs in
the [DataCite Metadata Schema 4.4](https://doi.org/10.14454/3w3z-sa82). This can be enhanced with activating the
external DataCite Fabrica system to generate DOIs, this is disabled by default.
To activate DOI minting, pass your DataCite Fabrica credentials in the environment variables:
```yaml title="docker-compose.yml"
services:
dbrepo-metadata-service:
image: docker.io/dbrepo/metadata-service:1.4.0
environment:
spring_profiles_active: doi
DATACITE_URL: https://api.datacite.org
DATACITE_PREFIX: 10.12345
DATACITE_USERNAME: username
DATACITE_PASSWORD: password
...
```
### Queries
It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query
execution to the raw data. Any stale queries (query that have been executed by users in DBRepo but were not saved) are
periodically being deleted from the query store based on the `DELETE_STALE_QUERIES_RATE` environment variable (defaults
to 60 seconds).
Executing SQL queries through the Query Endpoint must fulfill some restrictions:
* The SQL query does not contain at semicolon `;`
### Semantics
The service provides metadata to the table columns in the [Metadata Database](../system-databases-metadata) from
registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of
Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a
Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc.
### Tables
The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables
in the [Metadata Database](../system-databases-metadata). Any tables that are created outside of DBRepo (e.g. directly via the JDBC API) are
periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable, default interval is 60
seconds).
### Users
The service manages users in the [Data Database](../system-databases-data)
and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker)
and the [Authentication Service](../system-services-authentication).
The default configuration grants the users only very basic permissions on the databases:
* `SELECT`
* `CREATE`
* `CREATE VIEW`
* `CREATE ROUTINE`
* `CREATE TEMPORARY TABLES`
* `LOCK TABLES`
* `INDEX`
* `TRIGGER`
* `INSERT`
* `UPDATE`
* `DELETE`
This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You
can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create
temporary tables:
```yaml title="docker-compose.yml"
services:
dbrepo-metadata-service:
environment:
GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES
...
```
A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/)
### Views
The service manages views in the [Data Database](../system-databases-data)
and [Metadata Database](../system-databases-metadata). Any views that are created outside of DBRepo (e.g. directly via
the JDBC API) are periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable,
default interval is 60 seconds).
## Limitations
* No support for other databases than [MariaDB](https://mariadb.org/) because of system-versioning capabilities missing
in other database engines.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
(none)
.docs/api/open-api.md 0 → 100644
+ 7
− 0
View file @ 1994cff3
---
author: Martin Weise
---
All services are documented using the
[![OpenAPI 3.0](https://img.shields.io/badge/OpenAPI-3.0-leaf?logo=swagger&logoColor=white)](https://www.openapis.org/){ tabindex=-1 }
documentation standard.
.docs/usage-python.md → .docs/api/python.md
+ 2
− 5
View file @ 1994cff3
......@@ -6,11 +6,8 @@ author: Martin Weise
## tl;dr
!!! debug "Debug Information"
PyPI: [`dbrepo`](https://pypi.org/project/dbrepo/)
* Full module documentation <a href="../sphinx" target="_blank">:fontawesome-solid-square-up-right: view online</a>
[:fontawesome-solid-cube: &nbsp;View Docs](../sphinx){ .md-button .md-button--primary }
[:fontawesome-brands-python: &nbsp;PyPI](https://pypi.org/project/dbrepo/){ .md-button .md-button--secondary }
## Installing
......
......
.docs/api/search-service.md 0 → 100644
+ 89
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## tl;dr
!!! debug "Debug Information"
Image: [`dbrepo/search-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/search-service)
* Ports: 4000/tcp
* Health: `http://<hostname>:4000/api/search/health`
* Prometheus: `http://<hostname>:4000/metrics`
* Swagger UI: `http://<hostname>:4000/swagger-ui/` <a href="../swagger/search" target="_blank">:fontawesome-solid-square-up-right: view online</a>
## Overview
This service communicates between the [Search Database](../system-databases-search) and
the [User Interface](../system-other-ui) to allow structured search of databases, tables, columns, users, identifiers,
views, semantic concepts &amp; units of measurements used in databases.
<figure markdown>
![Built-in search](../images/screenshots/feature-search.png){ .img-border }
<figcaption>Figure 1: Faceted browsing</figcaption>
</figure>
## Index
There is only one
index [`database`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-search-db/init/indices/database.json)
that holds all the metadata information which is mirrored from the [Metadata Database](../system-databases-metadata).
<figure markdown>
![Mirroring statistical properties in Metadata Database and Search Database](../images/statistics-mirror.png)
<figcaption>Figure 2: Statistical properties in Metadata Database and Search Database</figcaption>
</figure>
## Faceted Browsing
This service enables the frontend to search the `database` index with eight different *types* of desired results
(database, table, column, view, identifier, user, concept, unit) and their *facets*.
For example, the [User Interface](../system-other-ui) allows for the search of databases that contain a certain
semantic concept (provided as URI, e.g.
temperature [http://www.wikidata.org/entity/Q11466](http://www.wikidata.org/entity/Q11466)) and unit of measurement
(provided as URI, e.g. degree
Celsius [http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius](http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius)).
An example on faceted browsing is found in the [usage examples](../usage-search).
## Unit Independent Search
Since the repository automatically collects statistical properties (min, max, mean, median, std.dev) in both the
[Metadata Database](../system-databases-metadata) and the [Search Database](../system-databases-search), a special
search can be performed when at least two columns have the same semantic concept (e.g. temperature) annotated and
the units of measurements can be transformed.
<figure markdown>
![Two tables with compatible semantic concepts (Temperature) and units of measurement (left is in degree Celsius, right is in degree Fahrenheit)](../images/statistics-example.png)
<figcaption>Figure 3: Two tables with compatible semantic concepts and units of measurement</figcaption>
</figure>
In short, the search service transforms the statistical properties not in the target unit of measurements is transformed
by using the [`omlib`](https://github.com/dieudonneWillems/OMLib) package.
For example: a user wants to find datasets that contain *"temperature measurements between 0 - 10 &deg;C"*. Then the
search service transforms the query to the dataset on the right from &deg;F to contain *"temperature measurements
between 32 - 50 &deg;F"* instead.
<figure markdown>
![Unit independent search query transformation](../images/statistics-example-unit-independent-search.png)
<figcaption>Figure 4: Unit independent search query transformation</figcaption>
</figure>
## Examples
View [usage examples](../usage-search/).
## Limitations
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
(nothing)
.docs/api/storage-service.md 0 → 100644
+ 44
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## tl;dr
!!! debug "Debug Information"
Image: [`chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs)
* Ports: 9000/tcp
* Prometheus: `http://<hostname>:9091/metrics`
## Overview
We use [SeaweedFS](https://seaweedfs.github.io/) as a high-performance, S3 compatible object store for easy, cloud-ready
deployments that by default support replication and monitoring. No graphical user interface is provided out-of-the-box,
administrators can access the S3 storage via S3-compatible clients
e.g. [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/) (see below).
### Users
The default configuration creates one user `seaweedfsadmin` with password `seaweedfsadmin`.
### Buckets
The default configuration creates two buckets `dbrepo-upload`, `dbrepo-download`:
* `dbrepo-upload` for CSV-file upload (for import of data, analysis, etc.) from the User Interface
* `dbrepo-download` for CSV-file download (exporting data, metadata, etc.)
## Limitations
* No support for multiple regions.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
1. For public deployments, change the default credentials.
.docs/api/ui.md 0 → 100644
+ 69
− 0
View file @ 1994cff3
---
author: Martin Weise
---
The User Interface is configured in the `runtimeConfig` section of the `nuxt.config.ts` file during build time. For the
runtime, you need to override those values through environment variables or by mounting a `.env` file. As a small
example, you can configure the logo :material-numeric-1-circle-outline: in Figure 2. Make sure you mount the logo as
image as well, in this example we want to mount a custom logo `my_logo.png` into the container and specify the name.
<figure markdown>
![Architecture of the UI microservice](../images/screenshots/ui-config-step-1.png){ .img-border }
<figcaption>Figure 2: Architecture of the UI microservice</figcaption>
</figure>
Text values like the version :material-numeric-2-circle-outline: and title :material-numeric-3-circle-outline: can be
configured as well via the Nuxt runtime configuration through single environment variables or `.env` files
```yaml title=".env"
NUXT_PUBLIC_TITLE="My overriden title"
NUXT_PUBLIC_LOGO="/app/.output/public/my_logo.png"
...
```
To work, you need to mount the `my_logo.png` file into the `dbrepo-ui` container via the `docker-compose.yml` file (or
if you use a Kubernetes deployment via ConfigMap and Volumes).
```yaml title="docker-compose.yml"
services:
dbrepo-ui:
image: docker.io/dbrepo/ui:__APPVERSION__
volumes:
- ./my_logo.png:/app/.output/public/my_logo.png
...
```
### Architecture
The server-client architecture of the User Interface is shown in [Figure 3](#fig3), it is supposed to help debug the
User Interface on development.
<figure id="fig3" markdown>
![Architecture of the UI microservice](../images/architecture-ui.svg)
<figcaption>Figure 3: Architecture of the User Interface</figcaption>
</figure>
* Runtime: [Bun 1+](https://bun.sh/) (preferred), *alternatively* Node.js 18+
* Builder: [Vite](https://vitejs.dev/)
* Server: [Nuxt.js 3+](https://nuxt.com/)
* Components: [Vue.js 3+](https://vuejs.org/)
* Frontend: [Vuetify 3+](https://vuetifyjs.com/en/)
* State: [Pinia](https://pinia.vuejs.org/)
### Example
See the [Usage Overview](../usage-overview/) page for detailed examples.
## Limitations
(none)
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
(none)
.docs/concepts/upload.md → .docs/api/upload-service.md
+ 1
− 3
View file @ 1994cff3
......@@ -2,8 +2,6 @@
author: Martin Weise
---
# Upload Service
## tl;dr
!!! debug "Debug Information"
......@@ -44,7 +42,7 @@ If your deployment is secured with SSL/TLS (recommended) set the `useSsl` variab
The Upload Service communicates internally with the [Storage Service](../system-services-storage) (c.f. [Figure 1](#fig1)).
<figure id="fig1" markdown>
![Architecture of the Upload Service](images/architecture-upload-service.svg)
![Architecture of the Upload Service](../images/architecture-upload-service.svg)
<figcaption>Figure 1: Architecture of the Upload Service</figcaption>
</figure>
......
......
.docs/concepts/authentication.md
+ 6
− 205
View file @ 1994cff3
......@@ -2,213 +2,14 @@
author: Martin Weise
---
# Authentication Service
## Bearer Authentication
## tl;dr
TBD
!!! debug "Debug Information"
## Basic Authentication
Image: [`dbrepo/authentication-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/authentication-service)
TBD
* Ports: 8080/tcp
* UI: `http://<hostname>/api/auth/admin/`
## OpenID Connect (OIDC)
## Overview
By default, users are created using the [User Interface](../system-other-ui) and the sign-up page in the User Interface.
This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then
managed by the
Authentication Service.
## Groups
The authorization scheme follows a group-based access control (GBAC). Users are organized in three distinct
(non-overlapping) groups:
1. Researchers (*default*)
2. Developers
3. Data Stewards
Based on the membership in one of these groups, the user is assigned a set of roles that authorize specific actions. By
default, all users are assigned to the `researchers` group.
## Roles
We organize the roles into default- and escalated composite roles. There are three composite roles, one for each group.
Each of the composite role has a set of other associated composite roles.
<figure markdown>
![](images/groups-roles.png)
<figcaption>Three groups (Researchers, Developers, Data Stewards) and their composite roles associated.</figcaption>
</figure>
There is one role for one specific action in the services. For example: the `create-database` role authorizes a user to
create a database in a Docker container. Therefore,
the [`DatabaseEndpoint.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/a5bdd1e2169bae6497e2f7eee82dad8b9b059850/fda-database-service/rest-service/src/main/java/at/tuwien/endpoints/DatabaseEndpoint.java#L78)
endpoint requires a JWT access token with this authority.
```java
@PostMapping
@PreAuthorize("hasAuthority('create-database')")
public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
@Valid @RequestBody DatabaseCreateDto createDto,
@NotNull Principal principal) {
...
}
```
### Default Container Handling
| Name | Description |
|-------------------|-------------------------------|
| `find-container` | Can find a specific container |
| `list-containers` | Can list all containers |
### Default Database Handling
| Name | Description |
|------------------------------|------------------------------------------------------|
| `check-database-access` | Can check the access to a database of a user |
| `create-database` | Can create a database |
| `create-database-access` | Can give a new access to a database of a user |
| `delete-database-access` | Can delete the access to a database of a user |
| `find-database` | Can find a specific database in a container |
| `list-databases` | Can list all databases in a container |
| `modify-database-image` | Can update the database image |
| `modify-database-owner` | Can modify the database owner |
| `modify-database-visibility` | Can modify the database visibility (public, private) |
| `update-database-access` | Can update the access to a database of a user |
### Default Table Handling
| Name | Description |
|---------------------------------|------------------------------------------------------|
| `create-table` | Can create a table |
| `find-tables` | Can list a specific table in a database |
| `list-tables` | Can list all tables |
| `modify-table-column-semantics` | Can modify the column semantics of a specific column |
| `delete-table` | Can delete tables owned by the user in a database |
### Default Query Handling
| Name | Description |
|---------------------------|-----------------------------------------------|
| `create-database-view` | Can create a view in a database |
| `delete-database-view` | Can delete a view in a database |
| `delete-table-data` | Can delete data in a table |
| `execute-query` | Can execute a query statement |
| `export-query-data` | Can export the data that a query has produced |
| `export-table-data` | Can export the data stored in a table |
| `find-database-view` | Can find a specific database view |
| `find-query` | Can find a specific query in the query store |
| `insert-table-data` | Can insert data into a table |
| `list-database-views` | Can list all database views |
| `list-queries` | Can list all queries in the query store |
| `persist-query` | Can persist a query in the query store |
| `re-execute-query` | Can re-execute a query to reproduce a result |
| `view-database-view-data` | Can view the data produced by a database view |
| `view-table-data` | Can view the data in a table |
| `view-table-history` | Can view the data history of a table |
### Default Identifier Handling
| Name | Description |
|---------------------|---------------------------------------------|
| `create-identifier` | Can create an identifier (subset, database) |
| `find-identifier` | Can find a specific identifier |
| `list-identifier` | Can list all identifiers |
### Default User Handling
| Name | Description |
|---------------------------|-----------------------------------------|
| `modify-user-theme` | Can modify the user theme (light, dark) |
| `modify-user-information` | Can modify the user information |
### Default Maintenance Handling
| Name | Description |
|------------------------------|------------------------------------------|
| `create-maintenance-message` | Can create a maintenance message banner |
| `delete-maintenance-message` | Can delete a maintenance message banner |
| `find-maintenance-message` | Can find a maintenance message banner |
| `list-maintenance-messages` | Can list all maintenance message banners |
| `update-maintenance-message` | Can update a maintenance message banner |
### Default Semantics Handling
| Name | Description |
|---------------------------|-----------------------------------------------------------------|
| `create-semantic-unit` | Can save a previously unknown unit for a table column |
| `create-semantic-concept` | Can save a previously unknown concept for a table column |
| `execute-semantic-query` | Can query remote SPARQL endpoints to get labels and description |
| `table-semantic-analyse` | Can automatically suggest units and concepts for a table |
### Escalated User Handling
| Name | Description |
|-------------|-----------------------------------------------|
| `find-user` | Can list user information for a specific user |
### Escalated Container Handling
| Name | Description |
|--------------------|--------------------------|
| `create-container` | Can create a container |
| `delete-container` | Can delete any container |
### Escalated Database Handling
| Name | Description |
|-------------------|------------------------------------------|
| `delete-database` | Can delete any database in any container |
### Escalated Table Handling
| Name | Description |
|------------------------|--------------------------------------|
| `delete-foreign-table` | Can delete any table in any database |
### Escalated Query Handling
| Name | Description |
|------|-------------|
| / | |
### Escalated Identifier Handling
| Name | Description |
|------------------------------|---------------------------------------------------|
| `create-foreign-identifier` | Can create an identifier to any database or query |
| `delete-identifier` | Can delete any identifier |
| `modify-identifier-metadata` | Can modify any identifier metadata |
### Escalated Semantics Handling
| Name | Description |
|-----------------------------------------|----------------------------------------------|
| `create-ontology` | Can register a new ontology |
| `delete-ontology` | Can unregister an ontology |
| `list-ontologies` | Can list all ontologies |
| `modify-foreign-table-column-semantics` | Can modify any table column concept and unit |
| `update-ontology` | Can update ontology metadata |
| `update-semantic-concept` | Can update own table column concept |
| `update-semantic-unit` | Can update own table column unit |
## Limitations
* No support for sending e-mails through Keycloak by default.
* No support for temporary passwords.
* No support for adding identifies in Keycloak directly.
* No support for multi-factor authentication.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
1. Mount your TLS certificate / private key pair into `/app/tls.crt` and `/app/tls.key` and
set `KC_HTTPS_CERTIFICATE_FILE=/app/tls.crt` and set `KC_HTTPS_CERTIFICATE_KEY_FILE=/app/tls.key`.
TBD
\ No newline at end of file
.docs/concepts/dbms.md 0 → 100644
+ 15
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## Relational Database
TBD
## Query
TBD
## System Versioning
TBD
\ No newline at end of file
.docs/concepts/index.md 0 → 100644
+ 15
− 0
View file @ 1994cff3
---
author: Martin Weise
---
This is the full system description from a technical/developer view and continuously being updated as the development
progresses.
## Data Ingest
<figure markdown>
![Data ingest](../images/data-ingest.svg)
<figcaption>Figure 1: Modes of data ingest</figcaption>
</figure>
More [usage examples](../usage-overview/) include how to ingest datasets, data dumps, live data, etc.
.docs/concepts/messaging.md 0 → 100644
+ 15
− 0
View file @ 1994cff3
---
author: Martin Weise
---
## Tuple
TBD
## AMQP
TBD
## MQTT
TBD
\ No newline at end of file
.docs/concepts/metadata.md
+ 1
− 142
View file @ 1994cff3
......@@ -2,145 +2,4 @@
author: Martin Weise
---
# Metadata Service
## tl;dr
!!! debug "Debug Information"
Image: [`dbrepo/metadata-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/metadata-service)
* Ports: 9099/tcp
* Info: `http://<hostname>:9099/actuator/info`
* Health: `http://<hostname>:9099/actuator/health`
- Readiness: `http://<hostname>:9099/actuator/health/readiness`
- Liveness: `http://<hostname>:9099/actuator/health/liveness`
* Prometheus: `http://<hostname>:9099/actuator/prometheus`
* Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
## Overview
This service manages the following topics:
* Databases
* Identifiers (DataCite, OAI-PMH)
* Queries
* Semantics (Ontologies)
* Tables
* Users
* Views
### Databases
The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and
data ingest operations.
### Identifiers
The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a database, subset,
table or view to obtain the metadata attached to it and allow reproduction of the exact same result.
This service also provides an OAI-PMH endpoint for metadata aggregators
(e.g. [OpenAIRE Graph](https://graph.openaire.eu/)). Through the User Interface, it also exposes metadata through
JSON-LD to metadata aggregators (e.g. [Google Datasets](https://datasetsearch.research.google.com/)). PID metadata
is always exposed, even for private databases.
The service generates internal PIDs, essentially representing internal URIs in
the [DataCite Metadata Schema 4.4](https://doi.org/10.14454/3w3z-sa82). This can be enhanced with activating the
external DataCite Fabrica system to generate DOIs, this is disabled by default.
To activate DOI minting, pass your DataCite Fabrica credentials in the environment variables:
```yaml title="docker-compose.yml"
services:
dbrepo-metadata-service:
image: docker.io/dbrepo/metadata-service:1.4.0
environment:
spring_profiles_active: doi
DATACITE_URL: https://api.datacite.org
DATACITE_PREFIX: 10.12345
DATACITE_USERNAME: username
DATACITE_PASSWORD: password
...
```
### Queries
It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query
execution to the raw data. Any stale queries (query that have been executed by users in DBRepo but were not saved) are
periodically being deleted from the query store based on the `DELETE_STALE_QUERIES_RATE` environment variable (defaults
to 60 seconds).
Executing SQL queries through the Query Endpoint must fulfill some restrictions:
* The SQL query does not contain at semicolon `;`
### Semantics
The service provides metadata to the table columns in the [Metadata Database](../system-databases-metadata) from
registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of
Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a
Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc.
### Tables
The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables
in the [Metadata Database](../system-databases-metadata). Any tables that are created outside of DBRepo (e.g. directly via the JDBC API) are
periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable, default interval is 60
seconds).
### Users
The service manages users in the [Data Database](../system-databases-data)
and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker)
and the [Authentication Service](../system-services-authentication).
The default configuration grants the users only very basic permissions on the databases:
* `SELECT`
* `CREATE`
* `CREATE VIEW`
* `CREATE ROUTINE`
* `CREATE TEMPORARY TABLES`
* `LOCK TABLES`
* `INDEX`
* `TRIGGER`
* `INSERT`
* `UPDATE`
* `DELETE`
This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You
can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create
temporary tables:
```yaml title="docker-compose.yml"
services:
dbrepo-metadata-service:
environment:
GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES
...
```
A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/)
### Views
The service manages views in the [Data Database](../system-databases-data)
and [Metadata Database](../system-databases-metadata). Any views that are created outside of DBRepo (e.g. directly via
the JDBC API) are periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable,
default interval is 60 seconds).
## Limitations
* No support for other databases than [MariaDB](https://mariadb.org/) because of system-versioning capabilities missing
in other database engines.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
experience!
## Security
(none)
TBD
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment