diff --git a/.docs/concepts/analysis.md b/.docs/api/analyse-service.md
similarity index 99%
rename from .docs/concepts/analysis.md
rename to .docs/api/analyse-service.md
index 8d940aba68301a693739358db355e8e631afe1fb..be5efdbf5c8c58dd0c0825e7c1328188d01b1aba 100644
--- a/.docs/concepts/analysis.md
+++ b/.docs/api/analyse-service.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-# Analyse Service
-
## tl;dr
!!! debug "Debug Information"
diff --git a/.docs/api/auth-service.md b/.docs/api/auth-service.md
new file mode 100644
index 0000000000000000000000000000000000000000..5d3e0f42b2bb19b28451c8a8c8e40d937ffe9fab
--- /dev/null
+++ b/.docs/api/auth-service.md
@@ -0,0 +1,212 @@
+---
+author: Martin Weise
+---
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+ Image: [`dbrepo/authentication-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/authentication-service)
+
+ * Ports: 8080/tcp
+ * UI: `http://<hostname>/api/auth/admin/`
+
+## Overview
+
+By default, users are created using the [User Interface](../system-other-ui) and the sign-up page in the User Interface.
+This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then
+managed by the
+Authentication Service.
+
+## Groups
+
+The authorization scheme follows a group-based access control (GBAC). Users are organized in three distinct
+(non-overlapping) groups:
+
+1. Researchers (*default*)
+2. Developers
+3. Data Stewards
+
+Based on the membership in one of these groups, the user is assigned a set of roles that authorize specific actions. By
+default, all users are assigned to the `researchers` group.
+
+## Roles
+
+We organize the roles into default- and escalated composite roles. There are three composite roles, one for each group.
+Each of the composite role has a set of other associated composite roles.
+
+<figure markdown>
+
+<figcaption>Three groups (Researchers, Developers, Data Stewards) and their composite roles associated.</figcaption>
+</figure>
+
+There is one role for one specific action in the services. For example: the `create-database` role authorizes a user to
+create a database in a Docker container. Therefore,
+the [`DatabaseEndpoint.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/a5bdd1e2169bae6497e2f7eee82dad8b9b059850/fda-database-service/rest-service/src/main/java/at/tuwien/endpoints/DatabaseEndpoint.java#L78)
+endpoint requires a JWT access token with this authority.
+
+```java
+@PostMapping
+@PreAuthorize("hasAuthority('create-database')")
+public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
+ @Valid @RequestBody DatabaseCreateDto createDto,
+ @NotNull Principal principal) {
+...
+}
+```
+
+### Default Container Handling
+
+| Name | Description |
+|-------------------|-------------------------------|
+| `find-container` | Can find a specific container |
+| `list-containers` | Can list all containers |
+
+### Default Database Handling
+
+| Name | Description |
+|------------------------------|------------------------------------------------------|
+| `check-database-access` | Can check the access to a database of a user |
+| `create-database` | Can create a database |
+| `create-database-access` | Can give a new access to a database of a user |
+| `delete-database-access` | Can delete the access to a database of a user |
+| `find-database` | Can find a specific database in a container |
+| `list-databases` | Can list all databases in a container |
+| `modify-database-image` | Can update the database image |
+| `modify-database-owner` | Can modify the database owner |
+| `modify-database-visibility` | Can modify the database visibility (public, private) |
+| `update-database-access` | Can update the access to a database of a user |
+
+### Default Table Handling
+
+| Name | Description |
+|---------------------------------|------------------------------------------------------|
+| `create-table` | Can create a table |
+| `find-tables` | Can list a specific table in a database |
+| `list-tables` | Can list all tables |
+| `modify-table-column-semantics` | Can modify the column semantics of a specific column |
+| `delete-table` | Can delete tables owned by the user in a database |
+
+### Default Query Handling
+
+| Name | Description |
+|---------------------------|-----------------------------------------------|
+| `create-database-view` | Can create a view in a database |
+| `delete-database-view` | Can delete a view in a database |
+| `delete-table-data` | Can delete data in a table |
+| `execute-query` | Can execute a query statement |
+| `export-query-data` | Can export the data that a query has produced |
+| `export-table-data` | Can export the data stored in a table |
+| `find-database-view` | Can find a specific database view |
+| `find-query` | Can find a specific query in the query store |
+| `insert-table-data` | Can insert data into a table |
+| `list-database-views` | Can list all database views |
+| `list-queries` | Can list all queries in the query store |
+| `persist-query` | Can persist a query in the query store |
+| `re-execute-query` | Can re-execute a query to reproduce a result |
+| `view-database-view-data` | Can view the data produced by a database view |
+| `view-table-data` | Can view the data in a table |
+| `view-table-history` | Can view the data history of a table |
+
+### Default Identifier Handling
+
+| Name | Description |
+|---------------------|---------------------------------------------|
+| `create-identifier` | Can create an identifier (subset, database) |
+| `find-identifier` | Can find a specific identifier |
+| `list-identifier` | Can list all identifiers |
+
+### Default User Handling
+
+| Name | Description |
+|---------------------------|-----------------------------------------|
+| `modify-user-theme` | Can modify the user theme (light, dark) |
+| `modify-user-information` | Can modify the user information |
+
+### Default Maintenance Handling
+
+| Name | Description |
+|------------------------------|------------------------------------------|
+| `create-maintenance-message` | Can create a maintenance message banner |
+| `delete-maintenance-message` | Can delete a maintenance message banner |
+| `find-maintenance-message` | Can find a maintenance message banner |
+| `list-maintenance-messages` | Can list all maintenance message banners |
+| `update-maintenance-message` | Can update a maintenance message banner |
+
+### Default Semantics Handling
+
+| Name | Description |
+|---------------------------|-----------------------------------------------------------------|
+| `create-semantic-unit` | Can save a previously unknown unit for a table column |
+| `create-semantic-concept` | Can save a previously unknown concept for a table column |
+| `execute-semantic-query` | Can query remote SPARQL endpoints to get labels and description |
+| `table-semantic-analyse` | Can automatically suggest units and concepts for a table |
+
+### Escalated User Handling
+
+| Name | Description |
+|-------------|-----------------------------------------------|
+| `find-user` | Can list user information for a specific user |
+
+### Escalated Container Handling
+
+| Name | Description |
+|--------------------|--------------------------|
+| `create-container` | Can create a container |
+| `delete-container` | Can delete any container |
+
+### Escalated Database Handling
+
+| Name | Description |
+|-------------------|------------------------------------------|
+| `delete-database` | Can delete any database in any container |
+
+### Escalated Table Handling
+
+| Name | Description |
+|------------------------|--------------------------------------|
+| `delete-foreign-table` | Can delete any table in any database |
+
+### Escalated Query Handling
+
+| Name | Description |
+|------|-------------|
+| / | |
+
+### Escalated Identifier Handling
+
+| Name | Description |
+|------------------------------|---------------------------------------------------|
+| `create-foreign-identifier` | Can create an identifier to any database or query |
+| `delete-identifier` | Can delete any identifier |
+| `modify-identifier-metadata` | Can modify any identifier metadata |
+
+### Escalated Semantics Handling
+
+| Name | Description |
+|-----------------------------------------|----------------------------------------------|
+| `create-ontology` | Can register a new ontology |
+| `delete-ontology` | Can unregister an ontology |
+| `list-ontologies` | Can list all ontologies |
+| `modify-foreign-table-column-semantics` | Can modify any table column concept and unit |
+| `update-ontology` | Can update ontology metadata |
+| `update-semantic-concept` | Can update own table column concept |
+| `update-semantic-unit` | Can update own table column unit |
+
+## Limitations
+
+* No support for sending e-mails through Keycloak by default.
+* No support for temporary passwords.
+* No support for adding identifies in Keycloak directly.
+* No support for multi-factor authentication.
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+ We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+ in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
+ experience!
+
+## Security
+
+1. Mount your TLS certificate / private key pair into `/app/tls.crt` and `/app/tls.key` and
+ set `KC_HTTPS_CERTIFICATE_FILE=/app/tls.crt` and set `KC_HTTPS_CERTIFICATE_KEY_FILE=/app/tls.key`.
diff --git a/.docs/concepts/message-broker.md b/.docs/api/broker-service.md
similarity index 96%
rename from .docs/concepts/message-broker.md
rename to .docs/api/broker-service.md
index 7ed0c3333f796180f11d0581e248065a979e1238..0345271bf1c23545e029b51f2de70311235ade5a 100644
--- a/.docs/concepts/message-broker.md
+++ b/.docs/api/broker-service.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-# Broker Service
-
## tl;dr
!!! debug "Debug Information"
@@ -40,14 +38,14 @@ queue `dbrepo`, connected with a binding of `dbrepo.#` which routes all tuples w
the dot!) to this queue.
<figure markdown>
- 
+ 
<figcaption>Replicated quorum queue dbrepo in a cluster with three nodes</figcaption>
</figure>
The consumer takes care of writing it to the correct table in the [Data Service](../system-services-data).
<figure markdown>
- 
+ 
<figcaption>Architecture Broker Service</figcaption>
</figure>
diff --git a/.docs/concepts/relational-database.md b/.docs/api/data-db.md
similarity index 97%
rename from .docs/concepts/relational-database.md
rename to .docs/api/data-db.md
index 468b97ac07fa5a5948730abe9fd857539b5b3a11..859264a53a08da57536c221dfd76899a833b3a6d 100644
--- a/.docs/concepts/relational-database.md
+++ b/.docs/api/data-db.md
@@ -60,7 +60,7 @@ the [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
natively.
<figure markdown>
-
+
<figcaption>Sidecar that handles the CSV-file upload/download.</figcaption>
</figure>
diff --git a/.docs/concepts/data.md b/.docs/api/data-service.md
similarity index 90%
rename from .docs/concepts/data.md
rename to .docs/api/data-service.md
index b8a1fe6263da6f236735206b7477c65adda84128..df6b87e2f06bd59e90ce6116a240d538cd8185c4 100644
--- a/.docs/concepts/data.md
+++ b/.docs/api/data-service.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-# Data Service
-
## tl;dr
!!! debug "Debug Information"
@@ -26,8 +24,6 @@ Data Service up.
## Limitations
-* No clear differentiation for data-handling endpoints from the [Metadata Service](../system-services-metadata) yet.
-
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
diff --git a/.docs/concepts/gateway.md b/.docs/api/gateway-service.md
similarity index 99%
rename from .docs/concepts/gateway.md
rename to .docs/api/gateway-service.md
index 9950110c71c6a86088e3e530f59a5d5bca6935f7..172892e3bdb13411c658093cbc7adf76fbdd6de7 100644
--- a/.docs/concepts/gateway.md
+++ b/.docs/api/gateway-service.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-# Gateway Service
-
## tl;dr
!!! debug "Debug Information"
diff --git a/.docs/usage-overview.md b/.docs/api/index.md
similarity index 100%
rename from .docs/usage-overview.md
rename to .docs/api/index.md
diff --git a/.docs/api/metadata-db.md b/.docs/api/metadata-db.md
new file mode 100644
index 0000000000000000000000000000000000000000..e56b88ab8b99fb4cdffc9d68c5db28703f4fdaa8
--- /dev/null
+++ b/.docs/api/metadata-db.md
@@ -0,0 +1,5 @@
+---
+author: Martin Weise
+---
+
+TBD
\ No newline at end of file
diff --git a/.docs/api/metadata-service.md b/.docs/api/metadata-service.md
new file mode 100644
index 0000000000000000000000000000000000000000..33f3db3bfe5be3d473420599ff90b6582a9e4443
--- /dev/null
+++ b/.docs/api/metadata-service.md
@@ -0,0 +1,144 @@
+---
+author: Martin Weise
+---
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+ Image: [`dbrepo/metadata-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/metadata-service)
+
+ * Ports: 9099/tcp
+ * Info: `http://<hostname>:9099/actuator/info`
+ * Health: `http://<hostname>:9099/actuator/health`
+ - Readiness: `http://<hostname>:9099/actuator/health/readiness`
+ - Liveness: `http://<hostname>:9099/actuator/health/liveness`
+ * Prometheus: `http://<hostname>:9099/actuator/prometheus`
+ * Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+## Overview
+
+This service manages the following topics:
+
+* Databases
+* Identifiers (DataCite, OAI-PMH)
+* Queries
+* Semantics (Ontologies)
+* Tables
+* Users
+* Views
+
+### Databases
+
+The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and
+data ingest operations.
+
+### Identifiers
+
+The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a database, subset,
+table or view to obtain the metadata attached to it and allow reproduction of the exact same result.
+
+This service also provides an OAI-PMH endpoint for metadata aggregators
+(e.g. [OpenAIRE Graph](https://graph.openaire.eu/)). Through the User Interface, it also exposes metadata through
+JSON-LD to metadata aggregators (e.g. [Google Datasets](https://datasetsearch.research.google.com/)). PID metadata
+is always exposed, even for private databases.
+
+The service generates internal PIDs, essentially representing internal URIs in
+the [DataCite Metadata Schema 4.4](https://doi.org/10.14454/3w3z-sa82). This can be enhanced with activating the
+external DataCite Fabrica system to generate DOIs, this is disabled by default.
+
+To activate DOI minting, pass your DataCite Fabrica credentials in the environment variables:
+
+```yaml title="docker-compose.yml"
+services:
+ dbrepo-metadata-service:
+ image: docker.io/dbrepo/metadata-service:1.4.0
+ environment:
+ spring_profiles_active: doi
+ DATACITE_URL: https://api.datacite.org
+ DATACITE_PREFIX: 10.12345
+ DATACITE_USERNAME: username
+ DATACITE_PASSWORD: password
+ ...
+```
+
+### Queries
+
+It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query
+execution to the raw data. Any stale queries (query that have been executed by users in DBRepo but were not saved) are
+periodically being deleted from the query store based on the `DELETE_STALE_QUERIES_RATE` environment variable (defaults
+to 60 seconds).
+
+Executing SQL queries through the Query Endpoint must fulfill some restrictions:
+
+* The SQL query does not contain at semicolon `;`
+
+### Semantics
+
+The service provides metadata to the table columns in the [Metadata Database](../system-databases-metadata) from
+registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of
+Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a
+Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc.
+
+### Tables
+
+The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables
+in the [Metadata Database](../system-databases-metadata). Any tables that are created outside of DBRepo (e.g. directly via the JDBC API) are
+periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable, default interval is 60
+seconds).
+
+### Users
+
+The service manages users in the [Data Database](../system-databases-data)
+and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker)
+and the [Authentication Service](../system-services-authentication).
+
+The default configuration grants the users only very basic permissions on the databases:
+
+* `SELECT`
+* `CREATE`
+* `CREATE VIEW`
+* `CREATE ROUTINE`
+* `CREATE TEMPORARY TABLES`
+* `LOCK TABLES`
+* `INDEX`
+* `TRIGGER`
+* `INSERT`
+* `UPDATE`
+* `DELETE`
+
+This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You
+can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create
+temporary tables:
+
+```yaml title="docker-compose.yml"
+services:
+ dbrepo-metadata-service:
+ environment:
+ GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES
+ ...
+```
+
+A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/)
+
+### Views
+
+The service manages views in the [Data Database](../system-databases-data)
+and [Metadata Database](../system-databases-metadata). Any views that are created outside of DBRepo (e.g. directly via
+the JDBC API) are periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable,
+default interval is 60 seconds).
+
+## Limitations
+
+* No support for other databases than [MariaDB](https://mariadb.org/) because of system-versioning capabilities missing
+ in other database engines.
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+ We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+ in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
+ experience!
+
+## Security
+
+(none)
diff --git a/.docs/api/open-api.md b/.docs/api/open-api.md
new file mode 100644
index 0000000000000000000000000000000000000000..f06ec67a0daec31d77ddd0b62716276183c4b728
--- /dev/null
+++ b/.docs/api/open-api.md
@@ -0,0 +1,7 @@
+---
+author: Martin Weise
+---
+
+All services are documented using the
+[](https://www.openapis.org/){ tabindex=-1 }
+documentation standard.
diff --git a/.docs/usage-python.md b/.docs/api/python.md
similarity index 95%
rename from .docs/usage-python.md
rename to .docs/api/python.md
index ce83d6304f0d3f54e8575daac13cc63fa470a07b..bc7469d680b14f6a1b770ac508c1b35a98033db7 100644
--- a/.docs/usage-python.md
+++ b/.docs/api/python.md
@@ -6,11 +6,8 @@ author: Martin Weise
## tl;dr
-!!! debug "Debug Information"
-
- PyPI: [`dbrepo`](https://pypi.org/project/dbrepo/)
-
- * Full module documentation <a href="../sphinx" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+[:fontawesome-solid-cube: View Docs](../sphinx){ .md-button .md-button--primary }
+[:fontawesome-brands-python: PyPI](https://pypi.org/project/dbrepo/){ .md-button .md-button--secondary }
## Installing
diff --git a/.docs/api/search-service.md b/.docs/api/search-service.md
new file mode 100644
index 0000000000000000000000000000000000000000..fff317d6f8adc093cdf6f725bab31f5bbdb424e7
--- /dev/null
+++ b/.docs/api/search-service.md
@@ -0,0 +1,89 @@
+---
+author: Martin Weise
+---
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+ Image: [`dbrepo/search-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/search-service)
+
+ * Ports: 4000/tcp
+ * Health: `http://<hostname>:4000/api/search/health`
+ * Prometheus: `http://<hostname>:4000/metrics`
+ * Swagger UI: `http://<hostname>:4000/swagger-ui/` <a href="../swagger/search" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+## Overview
+
+This service communicates between the [Search Database](../system-databases-search) and
+the [User Interface](../system-other-ui) to allow structured search of databases, tables, columns, users, identifiers,
+views, semantic concepts & units of measurements used in databases.
+
+<figure markdown>
+{ .img-border }
+<figcaption>Figure 1: Faceted browsing</figcaption>
+</figure>
+
+## Index
+
+There is only one
+index [`database`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-search-db/init/indices/database.json)
+that holds all the metadata information which is mirrored from the [Metadata Database](../system-databases-metadata).
+
+<figure markdown>
+
+<figcaption>Figure 2: Statistical properties in Metadata Database and Search Database</figcaption>
+</figure>
+
+## Faceted Browsing
+
+This service enables the frontend to search the `database` index with eight different *types* of desired results
+(database, table, column, view, identifier, user, concept, unit) and their *facets*.
+
+For example, the [User Interface](../system-other-ui) allows for the search of databases that contain a certain
+semantic concept (provided as URI, e.g.
+temperature [http://www.wikidata.org/entity/Q11466](http://www.wikidata.org/entity/Q11466)) and unit of measurement
+(provided as URI, e.g. degree
+Celsius [http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius](http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius)).
+
+An example on faceted browsing is found in the [usage examples](../usage-search).
+
+## Unit Independent Search
+
+Since the repository automatically collects statistical properties (min, max, mean, median, std.dev) in both the
+[Metadata Database](../system-databases-metadata) and the [Search Database](../system-databases-search), a special
+search can be performed when at least two columns have the same semantic concept (e.g. temperature) annotated and
+the units of measurements can be transformed.
+
+<figure markdown>
+
+<figcaption>Figure 3: Two tables with compatible semantic concepts and units of measurement</figcaption>
+</figure>
+
+In short, the search service transforms the statistical properties not in the target unit of measurements is transformed
+by using the [`omlib`](https://github.com/dieudonneWillems/OMLib) package.
+
+For example: a user wants to find datasets that contain *"temperature measurements between 0 - 10 °C"*. Then the
+search service transforms the query to the dataset on the right from °F to contain *"temperature measurements
+between 32 - 50 °F"* instead.
+
+<figure markdown>
+
+<figcaption>Figure 4: Unit independent search query transformation</figcaption>
+</figure>
+
+## Examples
+
+View [usage examples](../usage-search/).
+
+## Limitations
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+ We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+ in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
+ experience!
+
+## Security
+
+(nothing)
diff --git a/.docs/api/storage-service.md b/.docs/api/storage-service.md
new file mode 100644
index 0000000000000000000000000000000000000000..2219c5fa5759dce36cbb0276594b1f0c5d32ee64
--- /dev/null
+++ b/.docs/api/storage-service.md
@@ -0,0 +1,44 @@
+---
+author: Martin Weise
+---
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+ Image: [`chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs)
+
+ * Ports: 9000/tcp
+ * Prometheus: `http://<hostname>:9091/metrics`
+
+## Overview
+
+We use [SeaweedFS](https://seaweedfs.github.io/) as a high-performance, S3 compatible object store for easy, cloud-ready
+deployments that by default support replication and monitoring. No graphical user interface is provided out-of-the-box,
+administrators can access the S3 storage via S3-compatible clients
+e.g. [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/) (see below).
+
+### Users
+
+The default configuration creates one user `seaweedfsadmin` with password `seaweedfsadmin`.
+
+### Buckets
+
+The default configuration creates two buckets `dbrepo-upload`, `dbrepo-download`:
+
+* `dbrepo-upload` for CSV-file upload (for import of data, analysis, etc.) from the User Interface
+* `dbrepo-download` for CSV-file download (exporting data, metadata, etc.)
+
+## Limitations
+
+* No support for multiple regions.
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+ We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+ in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
+ experience!
+
+## Security
+
+1. For public deployments, change the default credentials.
diff --git a/.docs/api/ui.md b/.docs/api/ui.md
new file mode 100644
index 0000000000000000000000000000000000000000..659b21ea90137034872c4312aaeeca4caf5bd346
--- /dev/null
+++ b/.docs/api/ui.md
@@ -0,0 +1,69 @@
+---
+author: Martin Weise
+---
+
+The User Interface is configured in the `runtimeConfig` section of the `nuxt.config.ts` file during build time. For the
+runtime, you need to override those values through environment variables or by mounting a `.env` file. As a small
+example, you can configure the logo :material-numeric-1-circle-outline: in Figure 2. Make sure you mount the logo as
+image as well, in this example we want to mount a custom logo `my_logo.png` into the container and specify the name.
+
+<figure markdown>
+{ .img-border }
+<figcaption>Figure 2: Architecture of the UI microservice</figcaption>
+</figure>
+
+Text values like the version :material-numeric-2-circle-outline: and title :material-numeric-3-circle-outline: can be
+configured as well via the Nuxt runtime configuration through single environment variables or `.env` files
+
+```yaml title=".env"
+NUXT_PUBLIC_TITLE="My overriden title"
+NUXT_PUBLIC_LOGO="/app/.output/public/my_logo.png"
+...
+```
+
+To work, you need to mount the `my_logo.png` file into the `dbrepo-ui` container via the `docker-compose.yml` file (or
+if you use a Kubernetes deployment via ConfigMap and Volumes).
+
+```yaml title="docker-compose.yml"
+services:
+ dbrepo-ui:
+ image: docker.io/dbrepo/ui:__APPVERSION__
+ volumes:
+ - ./my_logo.png:/app/.output/public/my_logo.png
+ ...
+```
+
+### Architecture
+
+The server-client architecture of the User Interface is shown in [Figure 3](#fig3), it is supposed to help debug the
+User Interface on development.
+
+<figure id="fig3" markdown>
+
+<figcaption>Figure 3: Architecture of the User Interface</figcaption>
+</figure>
+
+* Runtime: [Bun 1+](https://bun.sh/) (preferred), *alternatively* Node.js 18+
+* Builder: [Vite](https://vitejs.dev/)
+* Server: [Nuxt.js 3+](https://nuxt.com/)
+* Components: [Vue.js 3+](https://vuejs.org/)
+* Frontend: [Vuetify 3+](https://vuetifyjs.com/en/)
+* State: [Pinia](https://pinia.vuejs.org/)
+
+### Example
+
+See the [Usage Overview](../usage-overview/) page for detailed examples.
+
+## Limitations
+
+(none)
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+ We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+ in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
+ experience!
+
+## Security
+
+(none)
diff --git a/.docs/concepts/upload.md b/.docs/api/upload-service.md
similarity index 95%
rename from .docs/concepts/upload.md
rename to .docs/api/upload-service.md
index 56cfb27ddc3a0d96d032d214aa4efa71101bf81b..88812d308ba856a1d7c77e65a1bf97298cb2e968 100644
--- a/.docs/concepts/upload.md
+++ b/.docs/api/upload-service.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-# Upload Service
-
## tl;dr
!!! debug "Debug Information"
@@ -44,7 +42,7 @@ If your deployment is secured with SSL/TLS (recommended) set the `useSsl` variab
The Upload Service communicates internally with the [Storage Service](../system-services-storage) (c.f. [Figure 1](#fig1)).
<figure id="fig1" markdown>
-
+
<figcaption>Figure 1: Architecture of the Upload Service</figcaption>
</figure>
diff --git a/.docs/concepts/authentication.md b/.docs/concepts/authentication.md
index 0ec89e11c197e8c1b5c1ede7c1cf7f1f0167ccb1..5919b257239af3cb464bc54271037b179d418727 100644
--- a/.docs/concepts/authentication.md
+++ b/.docs/concepts/authentication.md
@@ -2,213 +2,14 @@
author: Martin Weise
---
-# Authentication Service
+## Bearer Authentication
-## tl;dr
+TBD
-!!! debug "Debug Information"
+## Basic Authentication
- Image: [`dbrepo/authentication-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/authentication-service)
+TBD
- * Ports: 8080/tcp
- * UI: `http://<hostname>/api/auth/admin/`
+## OpenID Connect (OIDC)
-## Overview
-
-By default, users are created using the [User Interface](../system-other-ui) and the sign-up page in the User Interface.
-This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then
-managed by the
-Authentication Service.
-
-## Groups
-
-The authorization scheme follows a group-based access control (GBAC). Users are organized in three distinct
-(non-overlapping) groups:
-
-1. Researchers (*default*)
-2. Developers
-3. Data Stewards
-
-Based on the membership in one of these groups, the user is assigned a set of roles that authorize specific actions. By
-default, all users are assigned to the `researchers` group.
-
-## Roles
-
-We organize the roles into default- and escalated composite roles. There are three composite roles, one for each group.
-Each of the composite role has a set of other associated composite roles.
-
-<figure markdown>
-
-<figcaption>Three groups (Researchers, Developers, Data Stewards) and their composite roles associated.</figcaption>
-</figure>
-
-There is one role for one specific action in the services. For example: the `create-database` role authorizes a user to
-create a database in a Docker container. Therefore,
-the [`DatabaseEndpoint.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/a5bdd1e2169bae6497e2f7eee82dad8b9b059850/fda-database-service/rest-service/src/main/java/at/tuwien/endpoints/DatabaseEndpoint.java#L78)
-endpoint requires a JWT access token with this authority.
-
-```java
-@PostMapping
-@PreAuthorize("hasAuthority('create-database')")
-public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
- @Valid @RequestBody DatabaseCreateDto createDto,
- @NotNull Principal principal) {
-...
-}
-```
-
-### Default Container Handling
-
-| Name | Description |
-|-------------------|-------------------------------|
-| `find-container` | Can find a specific container |
-| `list-containers` | Can list all containers |
-
-### Default Database Handling
-
-| Name | Description |
-|------------------------------|------------------------------------------------------|
-| `check-database-access` | Can check the access to a database of a user |
-| `create-database` | Can create a database |
-| `create-database-access` | Can give a new access to a database of a user |
-| `delete-database-access` | Can delete the access to a database of a user |
-| `find-database` | Can find a specific database in a container |
-| `list-databases` | Can list all databases in a container |
-| `modify-database-image` | Can update the database image |
-| `modify-database-owner` | Can modify the database owner |
-| `modify-database-visibility` | Can modify the database visibility (public, private) |
-| `update-database-access` | Can update the access to a database of a user |
-
-### Default Table Handling
-
-| Name | Description |
-|---------------------------------|------------------------------------------------------|
-| `create-table` | Can create a table |
-| `find-tables` | Can list a specific table in a database |
-| `list-tables` | Can list all tables |
-| `modify-table-column-semantics` | Can modify the column semantics of a specific column |
-| `delete-table` | Can delete tables owned by the user in a database |
-
-### Default Query Handling
-
-| Name | Description |
-|---------------------------|-----------------------------------------------|
-| `create-database-view` | Can create a view in a database |
-| `delete-database-view` | Can delete a view in a database |
-| `delete-table-data` | Can delete data in a table |
-| `execute-query` | Can execute a query statement |
-| `export-query-data` | Can export the data that a query has produced |
-| `export-table-data` | Can export the data stored in a table |
-| `find-database-view` | Can find a specific database view |
-| `find-query` | Can find a specific query in the query store |
-| `insert-table-data` | Can insert data into a table |
-| `list-database-views` | Can list all database views |
-| `list-queries` | Can list all queries in the query store |
-| `persist-query` | Can persist a query in the query store |
-| `re-execute-query` | Can re-execute a query to reproduce a result |
-| `view-database-view-data` | Can view the data produced by a database view |
-| `view-table-data` | Can view the data in a table |
-| `view-table-history` | Can view the data history of a table |
-
-### Default Identifier Handling
-
-| Name | Description |
-|---------------------|---------------------------------------------|
-| `create-identifier` | Can create an identifier (subset, database) |
-| `find-identifier` | Can find a specific identifier |
-| `list-identifier` | Can list all identifiers |
-
-### Default User Handling
-
-| Name | Description |
-|---------------------------|-----------------------------------------|
-| `modify-user-theme` | Can modify the user theme (light, dark) |
-| `modify-user-information` | Can modify the user information |
-
-### Default Maintenance Handling
-
-| Name | Description |
-|------------------------------|------------------------------------------|
-| `create-maintenance-message` | Can create a maintenance message banner |
-| `delete-maintenance-message` | Can delete a maintenance message banner |
-| `find-maintenance-message` | Can find a maintenance message banner |
-| `list-maintenance-messages` | Can list all maintenance message banners |
-| `update-maintenance-message` | Can update a maintenance message banner |
-
-### Default Semantics Handling
-
-| Name | Description |
-|---------------------------|-----------------------------------------------------------------|
-| `create-semantic-unit` | Can save a previously unknown unit for a table column |
-| `create-semantic-concept` | Can save a previously unknown concept for a table column |
-| `execute-semantic-query` | Can query remote SPARQL endpoints to get labels and description |
-| `table-semantic-analyse` | Can automatically suggest units and concepts for a table |
-
-### Escalated User Handling
-
-| Name | Description |
-|-------------|-----------------------------------------------|
-| `find-user` | Can list user information for a specific user |
-
-### Escalated Container Handling
-
-| Name | Description |
-|--------------------|--------------------------|
-| `create-container` | Can create a container |
-| `delete-container` | Can delete any container |
-
-### Escalated Database Handling
-
-| Name | Description |
-|-------------------|------------------------------------------|
-| `delete-database` | Can delete any database in any container |
-
-### Escalated Table Handling
-
-| Name | Description |
-|------------------------|--------------------------------------|
-| `delete-foreign-table` | Can delete any table in any database |
-
-### Escalated Query Handling
-
-| Name | Description |
-|------|-------------|
-| / | |
-
-### Escalated Identifier Handling
-
-| Name | Description |
-|------------------------------|---------------------------------------------------|
-| `create-foreign-identifier` | Can create an identifier to any database or query |
-| `delete-identifier` | Can delete any identifier |
-| `modify-identifier-metadata` | Can modify any identifier metadata |
-
-### Escalated Semantics Handling
-
-| Name | Description |
-|-----------------------------------------|----------------------------------------------|
-| `create-ontology` | Can register a new ontology |
-| `delete-ontology` | Can unregister an ontology |
-| `list-ontologies` | Can list all ontologies |
-| `modify-foreign-table-column-semantics` | Can modify any table column concept and unit |
-| `update-ontology` | Can update ontology metadata |
-| `update-semantic-concept` | Can update own table column concept |
-| `update-semantic-unit` | Can update own table column unit |
-
-## Limitations
-
-* No support for sending e-mails through Keycloak by default.
-* No support for temporary passwords.
-* No support for adding identifies in Keycloak directly.
-* No support for multi-factor authentication.
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
-
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
-
-## Security
-
-1. Mount your TLS certificate / private key pair into `/app/tls.crt` and `/app/tls.key` and
- set `KC_HTTPS_CERTIFICATE_FILE=/app/tls.crt` and set `KC_HTTPS_CERTIFICATE_KEY_FILE=/app/tls.key`.
+TBD
\ No newline at end of file
diff --git a/.docs/concepts/dbms.md b/.docs/concepts/dbms.md
new file mode 100644
index 0000000000000000000000000000000000000000..5dc2e084357f4cadba08cb11a3b72d9a37d70594
--- /dev/null
+++ b/.docs/concepts/dbms.md
@@ -0,0 +1,15 @@
+---
+author: Martin Weise
+---
+
+## Relational Database
+
+TBD
+
+## Query
+
+TBD
+
+## System Versioning
+
+TBD
\ No newline at end of file
diff --git a/.docs/concepts/index.md b/.docs/concepts/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..a4cc85435ac313d355b8b03b753073502fa5eeff
--- /dev/null
+++ b/.docs/concepts/index.md
@@ -0,0 +1,15 @@
+---
+author: Martin Weise
+---
+
+This is the full system description from a technical/developer view and continuously being updated as the development
+progresses.
+
+## Data Ingest
+
+<figure markdown>
+
+<figcaption>Figure 1: Modes of data ingest</figcaption>
+</figure>
+
+More [usage examples](../usage-overview/) include how to ingest datasets, data dumps, live data, etc.
diff --git a/.docs/concepts/messaging.md b/.docs/concepts/messaging.md
new file mode 100644
index 0000000000000000000000000000000000000000..448c7a3f63071e0ce8fc0597b8d06e061e519eeb
--- /dev/null
+++ b/.docs/concepts/messaging.md
@@ -0,0 +1,15 @@
+---
+author: Martin Weise
+---
+
+## Tuple
+
+TBD
+
+## AMQP
+
+TBD
+
+## MQTT
+
+TBD
\ No newline at end of file
diff --git a/.docs/concepts/metadata.md b/.docs/concepts/metadata.md
index b7fe150c8b90b09826f66feb8a158abf21793f2a..331a4bb98036a5db8769b097b895c73432cd01cc 100644
--- a/.docs/concepts/metadata.md
+++ b/.docs/concepts/metadata.md
@@ -2,145 +2,4 @@
author: Martin Weise
---
-# Metadata Service
-
-## tl;dr
-
-!!! debug "Debug Information"
-
- Image: [`dbrepo/metadata-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/metadata-service)
-
- * Ports: 9099/tcp
- * Info: `http://<hostname>:9099/actuator/info`
- * Health: `http://<hostname>:9099/actuator/health`
- - Readiness: `http://<hostname>:9099/actuator/health/readiness`
- - Liveness: `http://<hostname>:9099/actuator/health/liveness`
- * Prometheus: `http://<hostname>:9099/actuator/prometheus`
- * Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
-
-## Overview
-
-This service manages the following topics:
-
-* Databases
-* Identifiers (DataCite, OAI-PMH)
-* Queries
-* Semantics (Ontologies)
-* Tables
-* Users
-* Views
-
-### Databases
-
-The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and
-data ingest operations.
-
-### Identifiers
-
-The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a database, subset,
-table or view to obtain the metadata attached to it and allow reproduction of the exact same result.
-
-This service also provides an OAI-PMH endpoint for metadata aggregators
-(e.g. [OpenAIRE Graph](https://graph.openaire.eu/)). Through the User Interface, it also exposes metadata through
-JSON-LD to metadata aggregators (e.g. [Google Datasets](https://datasetsearch.research.google.com/)). PID metadata
-is always exposed, even for private databases.
-
-The service generates internal PIDs, essentially representing internal URIs in
-the [DataCite Metadata Schema 4.4](https://doi.org/10.14454/3w3z-sa82). This can be enhanced with activating the
-external DataCite Fabrica system to generate DOIs, this is disabled by default.
-
-To activate DOI minting, pass your DataCite Fabrica credentials in the environment variables:
-
-```yaml title="docker-compose.yml"
-services:
- dbrepo-metadata-service:
- image: docker.io/dbrepo/metadata-service:1.4.0
- environment:
- spring_profiles_active: doi
- DATACITE_URL: https://api.datacite.org
- DATACITE_PREFIX: 10.12345
- DATACITE_USERNAME: username
- DATACITE_PASSWORD: password
- ...
-```
-
-### Queries
-
-It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query
-execution to the raw data. Any stale queries (query that have been executed by users in DBRepo but were not saved) are
-periodically being deleted from the query store based on the `DELETE_STALE_QUERIES_RATE` environment variable (defaults
-to 60 seconds).
-
-Executing SQL queries through the Query Endpoint must fulfill some restrictions:
-
-* The SQL query does not contain at semicolon `;`
-
-### Semantics
-
-The service provides metadata to the table columns in the [Metadata Database](../system-databases-metadata) from
-registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of
-Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a
-Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc.
-
-### Tables
-
-The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables
-in the [Metadata Database](../system-databases-metadata). Any tables that are created outside of DBRepo (e.g. directly via the JDBC API) are
-periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable, default interval is 60
-seconds).
-
-### Users
-
-The service manages users in the [Data Database](../system-databases-data)
-and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker)
-and the [Authentication Service](../system-services-authentication).
-
-The default configuration grants the users only very basic permissions on the databases:
-
-* `SELECT`
-* `CREATE`
-* `CREATE VIEW`
-* `CREATE ROUTINE`
-* `CREATE TEMPORARY TABLES`
-* `LOCK TABLES`
-* `INDEX`
-* `TRIGGER`
-* `INSERT`
-* `UPDATE`
-* `DELETE`
-
-This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You
-can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create
-temporary tables:
-
-```yaml title="docker-compose.yml"
-services:
- dbrepo-metadata-service:
- environment:
- GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES
- ...
-```
-
-A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/)
-
-### Views
-
-The service manages views in the [Data Database](../system-databases-data)
-and [Metadata Database](../system-databases-metadata). Any views that are created outside of DBRepo (e.g. directly via
-the JDBC API) are periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable,
-default interval is 60 seconds).
-
-## Limitations
-
-* No support for other databases than [MariaDB](https://mariadb.org/) because of system-versioning capabilities missing
- in other database engines.
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
-
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
-
-## Security
-
-(none)
+TBD
diff --git a/.docs/concepts/microservice.md b/.docs/concepts/microservice.md
deleted file mode 100644
index 12b6553820e14feae108f51e4ce0649d81227e66..0000000000000000000000000000000000000000
--- a/.docs/concepts/microservice.md
+++ /dev/null
@@ -1,27 +0,0 @@
----
-author: Martin Weise
----
-
-# System
-
-!!! abstract "Abstract"
-
- This is the full system description from a technical/developer view and continously being updated as the development
- progresses.
-
-## Usage
-
-<figure markdown>
-
-<figcaption>Figure 1: Modes of data ingest</figcaption>
-</figure>
-
-More [usage examples](../usage-overview/) include how to ingest datasets, data dumps, live data, etc.
-
-## Limitations
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
-
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
\ No newline at end of file
diff --git a/.docs/concepts/search.md b/.docs/concepts/search.md
index edca2df6fc70ac39f75a61a8ae416ff072ffa0a7..8ae41c80ba2566dce997eed8ac4052b0cfa23631 100644
--- a/.docs/concepts/search.md
+++ b/.docs/concepts/search.md
@@ -2,90 +2,12 @@
author: Martin Weise
---
-# Search Service
-
-## tl;dr
-
-!!! debug "Debug Information"
-
- Image: [`dbrepo/search-service:__APPVERSION__`](https://hub.docker.com/r/dbrepo/search-service)
-
- * Ports: 4000/tcp
- * Health: `http://<hostname>:4000/api/search/health`
- * Prometheus: `http://<hostname>:4000/metrics`
- * Swagger UI: `http://<hostname>:4000/swagger-ui/` <a href="../swagger/search" target="_blank">:fontawesome-solid-square-up-right: view online</a>
-
-## Overview
-
-This service communicates between the [Search Database](../system-databases-search) and
-the [User Interface](../system-other-ui) to allow structured search of databases, tables, columns, users, identifiers,
-views, semantic concepts & units of measurements used in databases.
-
-<figure markdown>
-{ .img-border }
-<figcaption>Figure 1: Faceted browsing</figcaption>
-</figure>
-
## Index
-There is only one
-index [`database`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-search-db/init/indices/database.json)
-that holds all the metadata information which is mirrored from the [Metadata Database](../system-databases-metadata).
-
-<figure markdown>
-
-<figcaption>Figure 2: Statistical properties in Metadata Database and Search Database</figcaption>
-</figure>
-
-## Faceted Browsing
-
-This service enables the frontend to search the `database` index with eight different *types* of desired results
-(database, table, column, view, identifier, user, concept, unit) and their *facets*.
-
-For example, the [User Interface](../system-other-ui) allows for the search of databases that contain a certain
-semantic concept (provided as URI, e.g.
-temperature [http://www.wikidata.org/entity/Q11466](http://www.wikidata.org/entity/Q11466)) and unit of measurement
-(provided as URI, e.g. degree
-Celsius [http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius](http://www.ontology-of-units-of-measure.org/resource/om-2/degreeCelsius)).
-
-An example on faceted browsing is found in the [usage examples](../usage-search).
-
-## Unit Independent Search
-
-Since the repository automatically collects statistical properties (min, max, mean, median, std.dev) in both the
-[Metadata Database](../system-databases-metadata) and the [Search Database](../system-databases-search), a special
-search can be performed when at least two columns have the same semantic concept (e.g. temperature) annotated and
-the units of measurements can be transformed.
-
-<figure markdown>
-
-<figcaption>Figure 3: Two tables with compatible semantic concepts and units of measurement</figcaption>
-</figure>
-
-In short, the search service transforms the statistical properties not in the target unit of measurements is transformed
-by using the [`omlib`](https://github.com/dieudonneWillems/OMLib) package.
-
-For example: a user wants to find datasets that contain *"temperature measurements between 0 - 10 °C"*. Then the
-search service transforms the query to the dataset on the right from °F to contain *"temperature measurements
-between 32 - 50 °F"* instead.
-
-<figure markdown>
-
-<figcaption>Figure 4: Unit independent search query transformation</figcaption>
-</figure>
-
-## Examples
-
-View [usage examples](../usage-search/).
-
-## Limitations
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
+TBD
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
+## Document
-## Security
+TBD
-(nothing)
+## Query
\ No newline at end of file
diff --git a/.docs/concepts/storage.md b/.docs/concepts/storage.md
index f6b9ae28ee48fe5fa8dc319228ffe09ee4e3f030..fcc975fab2f20d4f3b3b6b0ebd69b02168227bc1 100644
--- a/.docs/concepts/storage.md
+++ b/.docs/concepts/storage.md
@@ -2,49 +2,14 @@
author: Martin Weise
---
-# Storage Service
+## S3
-## tl;dr
+TBD
-!!! debug "Debug Information"
+## Sidecar
- Image: [`chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs)
+TBD
- * Ports: 9000/tcp
- * Prometheus: `http://<hostname>:9091/metrics`
+## Upload
-## Overview
-
-We use [SeaweedFS](https://seaweedfs.github.io/) as a high-performance, S3 compatible object store for easy, cloud-ready
-deployments that by default support replication and monitoring. No graphical user interface is provided out-of-the-box,
-administrators can access the S3 storage via S3-compatible clients
-e.g. [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/) (see below).
-
-### Users
-
-The default configuration creates one user `seaweedfsadmin` with password `seaweedfsadmin`.
-
-### Buckets
-
-The default configuration creates two buckets `dbrepo-upload`, `dbrepo-download`:
-
-* `dbrepo-upload` for CSV-file upload (for import of data, analysis, etc.) from the User Interface
-* `dbrepo-download` for CSV-file download (exporting data, metadata, etc.)
-
-### Examples
-
-See the [usage page](../usage-storage).
-
-## Limitations
-
-* No support for multiple regions.
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
-
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
-
-## Security
-
-1. For public deployments, change the default credentials.
+TBD
\ No newline at end of file
diff --git a/.docs/concepts/ui.md b/.docs/concepts/ui.md
index cb8a1331035b91e7345ae98848bc49e73b05fa5c..ab6848b7ba251bde381c81af5fd8fb0b03293910 100644
--- a/.docs/concepts/ui.md
+++ b/.docs/concepts/ui.md
@@ -2,93 +2,20 @@
author: Martin Weise
---
-# User Interface
-
-## tl;dr
-
-!!! debug "Debug Information"
-
- Image: [`dbrepo/ui:__APPVERSION__`](https://hub.docker.com/r/dbrepo/ui)
-
- * Ports: 3000/tcp
- * UI: `http://<hostname>/`
-
-## Overview
-
It provides a graphical interface for a researcher to interact with the API (c.f. Figure 1).
<figure markdown>
-{ .img-border }
+{ .img-border }
<figcaption>Figure 1: User Interface</figcaption>
</figure>
-For examples on how to use the User Interface, visit the [Usage Overview](../usage-overview/) to find out how to create
+For examples on how to use the User Interface, visit the [API](../api/) page to find out how to create
users, databases and how to import your data.
-### Settings
-
-The User Interface is configured in the `runtimeConfig` section of the `nuxt.config.ts` file during build time. For the
-runtime, you need to override those values through environment variables or by mounting a `.env` file. As a small
-example, you can configure the logo :material-numeric-1-circle-outline: in Figure 2. Make sure you mount the logo as
-image as well, in this example we want to mount a custom logo `my_logo.png` into the container and specify the name.
-
-<figure markdown>
-{ .img-border }
-<figcaption>Figure 2: Architecture of the UI microservice</figcaption>
-</figure>
-
-Text values like the version :material-numeric-2-circle-outline: and title :material-numeric-3-circle-outline: can be
-configured as well via the Nuxt runtime configuration through single environment variables or `.env` files
-
-```yaml title=".env"
-NUXT_PUBLIC_TITLE="My overriden title"
-NUXT_PUBLIC_LOGO="/app/.output/public/my_logo.png"
-...
-```
-
-To work, you need to mount the `my_logo.png` file into the `dbrepo-ui` container via the `docker-compose.yml` file (or
-if you use a Kubernetes deployment via ConfigMap and Volumes).
-
-```yaml title="docker-compose.yml"
-services:
- dbrepo-ui:
- image: docker.io/dbrepo/ui:__APPVERSION__
- volumes:
- - ./my_logo.png:/app/.output/public/my_logo.png
- ...
-```
-
-### Architecture
-
-The server-client architecture of the User Interface is shown in [Figure 3](#fig3), it is supposed to help debug the
-User Interface on development.
-
-<figure id="fig3" markdown>
-
-<figcaption>Figure 3: Architecture of the User Interface</figcaption>
-</figure>
-
-* Runtime: [Bun 1+](https://bun.sh/) (preferred), *alternatively* Node.js 18+
-* Builder: [Vite](https://vitejs.dev/)
-* Server: [Nuxt.js 3+](https://nuxt.com/)
-* Components: [Vue.js 3+](https://vuejs.org/)
-* Frontend: [Vuetify 3+](https://vuetifyjs.com/en/)
-* State: [Pinia](https://pinia.vuejs.org/)
-
-### Example
-
-See the [Usage Overview](../usage-overview/) page for detailed examples.
-
-## Limitations
-
-(none)
-
-!!! question "Do you miss functionality? Do these limitations affect you?"
+## Server / Client
- We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
- in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming
- experience!
+TBD
-## Security
+## Cache
-(none)
+TBD
\ No newline at end of file
diff --git a/.docs/contributing.md b/.docs/contributing.md
index 4268b7807faa7be726b3de400407e09ec528ed35..259fcae55b1c3cd91ab9e55ce346db3953709876 100644
--- a/.docs/contributing.md
+++ b/.docs/contributing.md
@@ -2,8 +2,6 @@
author: Martin Weise
---
-## Contributing
-
We welcome contributions to DBRepo!
## Dependencies
diff --git a/.docs/examples/covid19.md b/.docs/examples/covid19.md
new file mode 100644
index 0000000000000000000000000000000000000000..0aedca2e37ba1bd4b45c6a86398e06d0c568fe17
--- /dev/null
+++ b/.docs/examples/covid19.md
@@ -0,0 +1,11 @@
+---
+author: Martin Weise
+---
+
+## Description
+
+TBD
+
+## Solution
+
+TBD
\ No newline at end of file
diff --git a/.docs/examples/influenza.md b/.docs/examples/influenza.md
new file mode 100644
index 0000000000000000000000000000000000000000..0aedca2e37ba1bd4b45c6a86398e06d0c568fe17
--- /dev/null
+++ b/.docs/examples/influenza.md
@@ -0,0 +1,11 @@
+---
+author: Martin Weise
+---
+
+## Description
+
+TBD
+
+## Solution
+
+TBD
\ No newline at end of file
diff --git a/.docs/examples/manufacturing.md b/.docs/examples/manufacturing.md
new file mode 100644
index 0000000000000000000000000000000000000000..0aedca2e37ba1bd4b45c6a86398e06d0c568fe17
--- /dev/null
+++ b/.docs/examples/manufacturing.md
@@ -0,0 +1,11 @@
+---
+author: Martin Weise
+---
+
+## Description
+
+TBD
+
+## Solution
+
+TBD
\ No newline at end of file
diff --git a/.docs/examples/power.md b/.docs/examples/power.md
new file mode 100644
index 0000000000000000000000000000000000000000..0aedca2e37ba1bd4b45c6a86398e06d0c568fe17
--- /dev/null
+++ b/.docs/examples/power.md
@@ -0,0 +1,11 @@
+---
+author: Martin Weise
+---
+
+## Description
+
+TBD
+
+## Solution
+
+TBD
\ No newline at end of file
diff --git a/.docs/examples/transportation.md b/.docs/examples/transportation.md
new file mode 100644
index 0000000000000000000000000000000000000000..0aedca2e37ba1bd4b45c6a86398e06d0c568fe17
--- /dev/null
+++ b/.docs/examples/transportation.md
@@ -0,0 +1,11 @@
+---
+author: Martin Weise
+---
+
+## Description
+
+TBD
+
+## Solution
+
+TBD
\ No newline at end of file
diff --git a/.docs/help.md b/.docs/help.md
index 5dd7d1c58a8cc2a2c76fa280f4b05a5d79bffa72..eaecbe6b0461a18320f9572bb408728e110281d2 100644
--- a/.docs/help.md
+++ b/.docs/help.md
@@ -2,10 +2,10 @@
author: Martin Weise
---
-## Usage Documentation
+## Concepts Documentation
-The [usage documentation](#) is the most complete guide on how to use DBRepo.
+The [concepts documentation](../concepts/) is the most complete guide on how to use DBRepo.
## API Documentation
-The [API documentation](#) present reference docs for all APIs.
+The [API documentation](../api/) present reference docs for all APIs.
diff --git a/.docs/index.md b/.docs/index.md
index 882dff6841cebde7b18874c561bf30349a352ba3..1776ffb83050935e7673ba19c3dff909349d5f22 100644
--- a/.docs/index.md
+++ b/.docs/index.md
@@ -4,9 +4,8 @@ author: Martin Weise
[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
-[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
-[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
-[](https://artifacthub.io/packages/helm/dbrepo/dbrepo){ tabindex=-1 }
+[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
+[](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services){ tabindex=-1 }
## Features
@@ -43,4 +42,4 @@ Galera Cluster with high degree of availability ensuring your data is always acc
We run a small demonstration instance so you can see the latest version of DBRepo in action. The demonstration instance
is updated with new releases and should be considered ephemeral.
-[:fontawesome-solid-flask: Demonstration Instance](https://test.dbrepo.tuwien.ac.at){ .md-button .md-button--primary }
\ No newline at end of file
+[:fontawesome-solid-flask: Demonstration Instance](https://test.dbrepo.tuwien.ac.at){ .md-button .md-button--primary }
\ No newline at end of file
diff --git a/.docs/migration.md b/.docs/migration.md
index fe8052fc8f9f03c416517be5cae7e1fa86e6f947..e56b88ab8b99fb4cdffc9d68c5db28703f4fdaa8 100644
--- a/.docs/migration.md
+++ b/.docs/migration.md
@@ -2,5 +2,4 @@
author: Martin Weise
---
-## Migration Guide
-
+TBD
\ No newline at end of file
diff --git a/.docs/stylesheets/.sass-cache/10990fa183107f4149f38216a4d00fe324a8131e/extra.scssc b/.docs/stylesheets/.sass-cache/10990fa183107f4149f38216a4d00fe324a8131e/extra.scssc
index 381e95eee429698ebbbc5f4a1b6337eb2823178d..c66d9e0bfba4de127388de9e8ad6f4212c1f5d36 100644
Binary files a/.docs/stylesheets/.sass-cache/10990fa183107f4149f38216a4d00fe324a8131e/extra.scssc and b/.docs/stylesheets/.sass-cache/10990fa183107f4149f38216a4d00fe324a8131e/extra.scssc differ
diff --git a/.docs/stylesheets/extra.css b/.docs/stylesheets/extra.css
index 5356c0f2fe1796dc3ebb579744653c0621e5fe55..d46da35109e113a53b127c02144f072752093c20 100644
--- a/.docs/stylesheets/extra.css
+++ b/.docs/stylesheets/extra.css
@@ -18,12 +18,12 @@ img.img-border {
.md-main .md-content a:not(.action-button):not([tabindex]),
.md-main .md-content a:not(.action-button):not([tabindex]) {
- color: var(--md-typeset-color);
+ color: var(--md-primary-fg-color);
border-bottom: 2px solid var(--md-primary-fg-color); }
.md-main .md-content a:not(.action-button):not([tabindex]):focus, .md-main .md-content a:not(.action-button):not([tabindex]):hover,
.md-main .md-content a:not(.action-button):not([tabindex]):focus,
.md-main .md-content a:not(.action-button):not([tabindex]):hover {
- color: var(--md-typeset-color);
+ color: var(--md-primary-fg-color--dark);
border-bottom: 2px solid var(--md-primary-fg-color--dark); }
.md-banner {
diff --git a/.docs/stylesheets/extra.css.map b/.docs/stylesheets/extra.css.map
index 5c5c3a911fcb8265a54e5b27f62e6b92bc1900c1..89ef3df537a6938c967eddd72fbf28bc85756b5e 100644
--- a/.docs/stylesheets/extra.css.map
+++ b/.docs/stylesheets/extra.css.map
@@ -1,6 +1,6 @@
{
"version": 3,
-"mappings": "AAAA;6BAC8B;EAC5B,qBAAqB,CAAC,QAAQ;EAC9B,oBAAoB,CAAC,QAAQ;EAAE,gBAAgB;EAC/C,2BAA2B,CAAC,QAAQ;EAAE,gBAAgB;;AAGxD,cAAe;EACb,MAAM,EAAE,iBAAiB;;AAG3B,yCAA0C;EACxC,KAAK,EAAE,gBAAgB;;AAIvB,oGACQ;EACN,KAAK,EAAE,gCAAgC;EACvC,UAAU,EAAE,OAAO;;AAIvB;0DAC2D;EACzD,KAAK,EAAE,uBAAuB;EAC9B,aAAa,EAAE,oCAAoC;EAEnD;;kEACQ;IACN,KAAK,EAAE,uBAAuB;IAC9B,aAAa,EAAE,0CAA0C;;AAK7D,UAAW;EACT,gBAAgB,EAAE,gCAAgC",
+"mappings": "AAAA;6BAC8B;EAC5B,qBAAqB,CAAC,QAAQ;EAC9B,oBAAoB,CAAC,QAAQ;EAAE,gBAAgB;EAC/C,2BAA2B,CAAC,QAAQ;EAAE,gBAAgB;;AAGxD,cAAe;EACb,MAAM,EAAE,iBAAiB;;AAG3B,yCAA0C;EACxC,KAAK,EAAE,gBAAgB;;AAIvB,oGACQ;EACN,KAAK,EAAE,gCAAgC;EACvC,UAAU,EAAE,OAAO;;AAIvB;0DAC2D;EACzD,KAAK,EAAE,0BAA0B;EACjC,aAAa,EAAE,oCAAoC;EAEnD;;kEACQ;IACN,KAAK,EAAE,gCAAgC;IACvC,aAAa,EAAE,0CAA0C;;AAK7D,UAAW;EACT,gBAAgB,EAAE,gCAAgC",
"sources": ["extra.scss"],
"names": [],
"file": "extra.css"
diff --git a/.docs/stylesheets/extra.scss b/.docs/stylesheets/extra.scss
index a595e585b47eec0ea470ae21af2faa3bc375094c..1262644f003e901721a9b2a9c59d08ed88d4c59a 100644
--- a/.docs/stylesheets/extra.scss
+++ b/.docs/stylesheets/extra.scss
@@ -23,12 +23,12 @@ img.img-border {
.md-main .md-content a:not(.action-button):not([tabindex]),
.md-main .md-content a:not(.action-button):not([tabindex]) {
- color: var(--md-typeset-color);
+ color: var(--md-primary-fg-color);
border-bottom: 2px solid var(--md-primary-fg-color);
&:focus,
&:hover {
- color: var(--md-typeset-color);
+ color: var(--md-primary-fg-color--dark);
border-bottom: 2px solid var(--md-primary-fg-color--dark);
}
}
diff --git a/dbrepo-analyse-service/app.py b/dbrepo-analyse-service/app.py
index 0e3043113a6df2bdea8dc69a8760671cec9d1c37..a02b3d5b859bfbe3eac11c4c7d7617ecc4390af5 100644
--- a/dbrepo-analyse-service/app.py
+++ b/dbrepo-analyse-service/app.py
@@ -132,11 +132,11 @@ app.config["AUTH_SERVICE_CLIENT"] = os.getenv("AUTH_SERVICE_CLIENT", "dbrepo")
app.config["AUTH_SERVICE_CLIENT_SECRET"] = os.getenv("AUTH_SERVICE_CLIENT_SECRET", "MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG")
app.config["ADMIN_USERNAME"] = os.getenv('ADMIN_USERNAME', 'admin')
app.config["ADMIN_PASSWORD"] = os.getenv('ADMIN_PASSWORD', 'admin')
-app.config["S3_ENDPOINT"] = os.getenv('S3_ENDPOINT', 'http://localhost:9000')
app.config["S3_ACCESS_KEY_ID"] = os.getenv('S3_ACCESS_KEY_ID', 'seaweedfsadmin')
-app.config["S3_SECRET_ACCESS_KEY"] = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
+app.config["S3_ENDPOINT"] = os.getenv('S3_ENDPOINT', 'http://localhost:9000')
app.config["S3_EXPORT_BUCKET"] = os.getenv('S3_EXPORT_BUCKET', 'dbrepo-download')
app.config["S3_IMPORT_BUCKET"] = os.getenv('S3_IMPORT_BUCKET', 'dbrepo-upload')
+app.config["S3_SECRET_ACCESS_KEY"] = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
app.json_encoder = LazyJSONEncoder
diff --git a/dbrepo-analyse-service/clients/s3_client.py b/dbrepo-analyse-service/clients/s3_client.py
index 22f21966b73fe4c91e28db666ca002d1d3d4cdfe..5e8f3bb4378f7c448cf9722f63a27e91fb3e9f39 100644
--- a/dbrepo-analyse-service/clients/s3_client.py
+++ b/dbrepo-analyse-service/clients/s3_client.py
@@ -20,7 +20,7 @@ class S3Client:
self.bucket_exists_or_exit(current_app.config['S3_EXPORT_BUCKET'])
self.bucket_exists_or_exit(current_app.config['S3_IMPORT_BUCKET'])
- def upload_file(self, filename: str, path: str = "/tmp/", bucket: str = "dbrepo-upload") -> bool:
+ def upload_file(self, filename: str, path: str = "/tmp", bucket: str = "dbrepo-upload") -> bool:
"""
Uploads a file to the blob storage.
Follows the official API https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html.
@@ -43,15 +43,16 @@ class S3Client:
logging.warning(f"Failed to upload file with key {filename}")
raise ConnectionRefusedError(f"Failed to upload file with key {filename}", e)
- def download_file(self, filename: str, bucket: str):
+ def download_file(self, filename: str, path: str = "/tmp", bucket: str = "dbrepo-download"):
"""
Downloads a file from the blob storage.
Follows the official API https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
:param filename: The filename.
+ :param path: The path the file is located.
:param bucket: The bucket to download the file from.
"""
self.file_exists(bucket, filename)
- filepath = os.path.join("/tmp/", filename)
+ filepath = os.path.join(path, filename)
self.client.download_file(bucket, filename, filepath)
logging.info(f"Downloaded .csv with key {filename} into {filepath}")
diff --git a/dbrepo-data-db/sidecar/app.py b/dbrepo-data-db/sidecar/app.py
index ffca4d3753d09ed300820fc8163920bd0c1dc7fd..1ddd181140353dbe20e2a9b54d404b22f9bcdc1b 100644
--- a/dbrepo-data-db/sidecar/app.py
+++ b/dbrepo-data-db/sidecar/app.py
@@ -121,10 +121,11 @@ app.config["AUTH_SERVICE_CLIENT"] = os.getenv("AUTH_SERVICE_CLIENT", "dbrepo-cli
app.config["AUTH_SERVICE_CLIENT_SECRET"] = os.getenv("AUTH_SERVICE_CLIENT_SECRET", "MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG")
app.config["ADMIN_USERNAME"] = os.getenv('ADMIN_USERNAME', 'admin')
app.config["ADMIN_PASSWORD"] = os.getenv('ADMIN_PASSWORD', 'admin')
-app.config["S3_ENDPOINT"] = os.getenv('S3_ENDPOINT', 'http://localhost:9000')
app.config["S3_ACCESS_KEY_ID"] = os.getenv('S3_ACCESS_KEY_ID', 'seaweedfsadmin')
-app.config["S3_SECRET_ACCESS_KEY"] = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
+app.config["S3_ENDPOINT"] = os.getenv('S3_ENDPOINT', 'http://localhost:9000')
app.config["S3_EXPORT_BUCKET"] = os.getenv('S3_EXPORT_BUCKET', 'dbrepo-download')
+app.config["S3_FILE_PATH"] = os.getenv('S3_FILE_PATH', '/tmp')
+app.config["S3_SECRET_ACCESS_KEY"] = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
app.config["S3_IMPORT_BUCKET"] = os.getenv('S3_IMPORT_BUCKET', 'dbrepo-upload')
app.json_encoder = LazyJSONEncoder
@@ -182,7 +183,7 @@ def import_csv(filename):
auth.current_user()
logging.debug('endpoint import csv, filename=%s, body=%s', filename, request)
s3_client = S3Client()
- response = s3_client.download_file(filename)
+ response = s3_client.download_file(filename, app.config["S3_FILE_PATH"], app.config['S3_IMPORT_BUCKET'])
if response is False:
return Response(), 400
return Response(json.dumps(response)), 202
@@ -194,7 +195,7 @@ def import_csv(filename):
def import_csv(filename):
logging.debug('endpoint export csv, filename=%s, body=%s', filename, request)
s3_client = S3Client()
- response = s3_client.upload_file(filename)
+ response = s3_client.upload_file(filename, app.config["S3_FILE_PATH"], app.config['S3_EXPORT_BUCKET'])
if response is False:
return Response(), 400
return Response(), 202
diff --git a/dbrepo-data-db/sidecar/clients/s3_client.py b/dbrepo-data-db/sidecar/clients/s3_client.py
index 135654da70d47b22d5d38edf6de19a34ba204f51..547a1c3a30d5dce07000a49852aef04673163de1 100644
--- a/dbrepo-data-db/sidecar/clients/s3_client.py
+++ b/dbrepo-data-db/sidecar/clients/s3_client.py
@@ -20,32 +20,35 @@ class S3Client:
self.bucket_exists_or_exit(current_app.config['S3_IMPORT_BUCKET'])
self.bucket_exists_or_exit(current_app.config['S3_EXPORT_BUCKET'])
- def upload_file(self, filename) -> bool:
+ def upload_file(self, filename, path, bucket) -> bool:
"""
Uploads a file to the blob storage.
Follows the official API https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html.
:param filename: The filename.
+ :param path: The path.
+ :param bucket: The bucket.
:return: True if the file was uploaded.
"""
- filepath = os.path.join("/tmp/", filename)
+ filepath = os.path.join(path, filename)
try:
- self.client.upload_file(filepath, current_app.config['S3_EXPORT_BUCKET'], filename)
- logging.info(f"Uploaded .csv {filepath} with key {filename} into bucket dbrepo-download")
+ self.client.upload_file(filepath, bucket, filename)
+ logging.info(f"Uploaded .csv {filepath} with key {filename} into bucket {bucket}")
return True
except ClientError as e:
logging.error(e)
return False
- def download_file(self, filename) -> bool:
+ def download_file(self, filename, path, bucket) -> bool:
"""
Downloads a file from the blob storage.
Follows the official API https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
:param filename: The filename.
+ :param path: The path.
+ :param bucket: The bucket.
:return: True if the file was downloaded and saved.
"""
- self.file_exists(current_app.config['S3_IMPORT_BUCKET'], filename)
- filepath = os.path.join("/tmp/", filename)
- bucket = current_app.config['S3_IMPORT_BUCKET']
+ self.file_exists(bucket, filename)
+ filepath = os.path.join(path, filename)
try:
self.client.download_file(bucket, filename, filepath)
logging.info(f"Downloaded .csv with key {filename} into {filepath} from bucket {bucket}")
diff --git a/dbrepo-data-service/rest-service/src/main/resources/application-local.yml b/dbrepo-data-service/rest-service/src/main/resources/application-local.yml
index 819f9d0f08bc3c588787ce0618f711e977e3dc07..adee5bea5986e7714625bba6d6ea504ead1b3e6c 100644
--- a/dbrepo-data-service/rest-service/src/main/resources/application-local.yml
+++ b/dbrepo-data-service/rest-service/src/main/resources/application-local.yml
@@ -57,6 +57,7 @@ dbrepo:
secretAccessKey: seaweedfsadmin
importBucket: dbrepo-upload
exportBucket: dbrepo-download
+ filePath: /tmp
admin:
username: admin
password: admin
diff --git a/dbrepo-data-service/rest-service/src/main/resources/application.yml b/dbrepo-data-service/rest-service/src/main/resources/application.yml
index 8bb8fe43f19bd4cefb3927c1192ce11f7b2c8cfc..056e2bd75063784619d0b3f0675821ff07324e81 100644
--- a/dbrepo-data-service/rest-service/src/main/resources/application.yml
+++ b/dbrepo-data-service/rest-service/src/main/resources/application.yml
@@ -58,6 +58,7 @@ dbrepo:
secretAccessKey: "${S3_SECRET_ACCESS_KEY:seaweedfsadmin}"
importBucket: "${S3_IMPORT_BUCKET:dbrepo-upload}"
exportBucket: "${S3_EXPORT_BUCKET:dbrepo-download}"
+ filePath: "${S3_FILE_PATH:/tmp}"
admin:
username: "${ADMIN_USERNAME:admin}"
password: "${ADMIN_PASSWORD:admin}"
diff --git a/dbrepo-data-service/services/src/main/java/at/tuwien/config/S3Config.java b/dbrepo-data-service/services/src/main/java/at/tuwien/config/S3Config.java
index 763505b933dd62259b95745e2059dea0c3edc9c6..8adaf38d19ba81ae61a54a57988dd1c08910a09b 100644
--- a/dbrepo-data-service/services/src/main/java/at/tuwien/config/S3Config.java
+++ b/dbrepo-data-service/services/src/main/java/at/tuwien/config/S3Config.java
@@ -33,6 +33,9 @@ public class S3Config {
@Value("${dbrepo.s3.exportBucket}")
private String s3ExportBucket;
+ @Value("${dbrepo.s3.filePath}")
+ private String s3FilePath;
+
@Bean
public S3Client s3client() {
final AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(
diff --git a/dbrepo-data-service/services/src/main/java/at/tuwien/mapper/MariaDbMapper.java b/dbrepo-data-service/services/src/main/java/at/tuwien/mapper/MariaDbMapper.java
index 2bac11bd4f770a02fea20d6c7f037c49364a1df7..9372f5f6ac32fb3289efdfe74de7a5826a74ea9c 100644
--- a/dbrepo-data-service/services/src/main/java/at/tuwien/mapper/MariaDbMapper.java
+++ b/dbrepo-data-service/services/src/main/java/at/tuwien/mapper/MariaDbMapper.java
@@ -369,7 +369,7 @@ public interface MariaDbMapper {
}
default String tableOrViewToRawExportQuery(String databaseName, String tableOrView, List<ColumnDto> columns,
- Instant timestamp, String filename) {
+ Instant timestamp, String filePath) {
final StringBuilder statement = new StringBuilder("SELECT ");
int[] idx = new int[]{0};
columns.forEach(column -> {
@@ -399,21 +399,21 @@ public interface MariaDbMapper {
.append(mariaDbFormatter.format(timestamp))
.append("'");
}
- statement.append(" INTO OUTFILE '/tmp/")
- .append(filename)
+ statement.append(" INTO OUTFILE '")
+ .append(filePath)
.append("' CHARACTER SET utf8 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"';");
statement.append(";");
log.debug("mapped table/view export query: {}", statement);
return statement.toString();
}
- default String subsetToRawExportQuery(String query, Instant timestamp, String filename) {
+ default String subsetToRawExportQuery(String query, Instant timestamp, String filePath) {
final StringBuilder statement = new StringBuilder(query.replaceAll(";", ""))
.append(" FOR SYSTEM_TIME AS OF TIMESTAMP'")
.append(mariaDbFormatter.format(timestamp))
.append("'")
- .append(" INTO OUTFILE '/tmp/")
- .append(filename)
+ .append(" INTO OUTFILE '")
+ .append(filePath)
.append("' CHARACTER SET utf8 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"';");
log.debug("mapped export query: {}", statement);
return statement.toString();
@@ -507,7 +507,7 @@ public interface MariaDbMapper {
}
default String datasetToRawInsertQuery(String databaseName, PrivilegedTableDto table, ImportCsvDto data) {
- final StringBuilder statement = new StringBuilder("LOAD DATA INFILE '/tmp/")
+ final StringBuilder statement = new StringBuilder("LOAD DATA INFILE '")
.append(data.getLocation())
.append("' REPLACE INTO TABLE `")
.append(databaseName)
diff --git a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/SubsetServiceMariaDbImpl.java b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/SubsetServiceMariaDbImpl.java
index 4df35be00b62a662f56bd88b6193f1bef20bef6d..57d7472dde490ae39143ca04fd255454e7c3b724 100644
--- a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/SubsetServiceMariaDbImpl.java
+++ b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/SubsetServiceMariaDbImpl.java
@@ -10,6 +10,7 @@ import at.tuwien.api.database.table.columns.ColumnDto;
import at.tuwien.api.identifier.IdentifierDto;
import at.tuwien.api.identifier.IdentifierTypeDto;
import at.tuwien.api.user.UserDto;
+import at.tuwien.config.S3Config;
import at.tuwien.exception.*;
import at.tuwien.gateway.DataDatabaseSidecarGateway;
import at.tuwien.gateway.MetadataServiceGateway;
@@ -33,6 +34,7 @@ import java.util.UUID;
@Service
public class SubsetServiceMariaDbImpl extends HibernateConnector implements SubsetService {
+ private final S3Config s3Config;
private final MariaDbMapper mariaDbMapper;
private final MetadataMapper metadataMapper;
private final StorageService storageService;
@@ -40,9 +42,10 @@ public class SubsetServiceMariaDbImpl extends HibernateConnector implements Subs
private final DataDatabaseSidecarGateway dataDatabaseSidecarGateway;
@Autowired
- public SubsetServiceMariaDbImpl(MariaDbMapper mariaDbMapper, MetadataMapper metadataMapper,
+ public SubsetServiceMariaDbImpl(S3Config s3Config, MariaDbMapper mariaDbMapper, MetadataMapper metadataMapper,
StorageService storageService, MetadataServiceGateway metadataServiceGateway,
DataDatabaseSidecarGateway dataDatabaseSidecarGateway) {
+ this.s3Config = s3Config;
this.mariaDbMapper = mariaDbMapper;
this.metadataMapper = metadataMapper;
this.storageService = storageService;
@@ -146,11 +149,12 @@ public class SubsetServiceMariaDbImpl extends HibernateConnector implements Subs
public ExportResourceDto export(PrivilegedDatabaseDto database, QueryDto query, Instant timestamp, String filename)
throws SQLException, QueryMalformedException, SidecarExportException, StorageNotFoundException,
StorageUnavailableException {
+ final String filePath = s3Config.getS3FilePath() + "/" + filename;
final ComboPooledDataSource dataSource = getPrivilegedDataSource(database);
final Connection connection = dataSource.getConnection();
try {
/* export to data database sidecar */
- connection.prepareStatement(mariaDbMapper.subsetToRawExportQuery(query.getQuery(), timestamp, filename))
+ connection.prepareStatement(mariaDbMapper.subsetToRawExportQuery(query.getQuery(), timestamp, filePath))
.executeUpdate();
connection.commit();
} catch (SQLException e) {
diff --git a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/TableServiceMariaDbImpl.java b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/TableServiceMariaDbImpl.java
index 32eaaf953332842a93a75c12e90c9f5d55495dbb..2e0952d0f828a7d45c6821565383035167e08ddb 100644
--- a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/TableServiceMariaDbImpl.java
+++ b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/TableServiceMariaDbImpl.java
@@ -9,6 +9,7 @@ import at.tuwien.api.database.table.columns.ColumnDto;
import at.tuwien.api.database.table.columns.ColumnTypeDto;
import at.tuwien.api.database.table.internal.PrivilegedTableDto;
import at.tuwien.api.database.table.internal.TableCreateDto;
+import at.tuwien.config.S3Config;
import at.tuwien.exception.*;
import at.tuwien.gateway.DataDatabaseSidecarGateway;
import at.tuwien.mapper.MariaDbMapper;
@@ -28,13 +29,15 @@ import java.util.*;
@Service
public class TableServiceMariaDbImpl extends HibernateConnector implements TableService {
+ private final S3Config s3Config;
private final MariaDbMapper mariaDbMapper;
private final StorageService storageService;
private final DataDatabaseSidecarGateway dataDatabaseSidecarGateway;
@Autowired
- public TableServiceMariaDbImpl(MariaDbMapper mariaDbMapper, StorageService storageService,
+ public TableServiceMariaDbImpl(S3Config s3Config, MariaDbMapper mariaDbMapper, StorageService storageService,
DataDatabaseSidecarGateway dataDatabaseSidecarGateway) {
+ this.s3Config = s3Config;
this.mariaDbMapper = mariaDbMapper;
this.storageService = storageService;
this.dataDatabaseSidecarGateway = dataDatabaseSidecarGateway;
@@ -213,6 +216,7 @@ public class TableServiceMariaDbImpl extends HibernateConnector implements Table
final Connection connection = dataSource.getConnection();
try {
/* import tuple */
+ data.setLocation(s3Config.getS3FilePath() + "/" + data.getLocation());
connection.prepareStatement(mariaDbMapper.datasetToRawInsertQuery(table.getDatabase().getInternalName(), table, data))
.execute();
connection.commit();
@@ -331,13 +335,14 @@ public class TableServiceMariaDbImpl extends HibernateConnector implements Table
public ExportResourceDto exportDataset(PrivilegedTableDto table, Instant timestamp)
throws SQLException, SidecarExportException, StorageNotFoundException, StorageUnavailableException,
QueryMalformedException {
- final String filename = RandomStringUtils.randomAlphabetic(40) + ".csv";
+ final String fileName = RandomStringUtils.randomAlphabetic(40) + ".csv";
+ final String filePath = s3Config.getS3FilePath() + "/" + fileName;
final ComboPooledDataSource dataSource = getPrivilegedDataSource(table.getDatabase());
final Connection connection = dataSource.getConnection();
try {
/* export to data database sidecar */
connection.prepareStatement(mariaDbMapper.tableOrViewToRawExportQuery(table.getDatabase().getInternalName(),
- table.getInternalName(), table.getColumns(), timestamp, filename))
+ table.getInternalName(), table.getColumns(), timestamp, filePath))
.executeUpdate();
connection.commit();
} catch (SQLException e) {
@@ -347,8 +352,8 @@ public class TableServiceMariaDbImpl extends HibernateConnector implements Table
} finally {
dataSource.close();
}
- dataDatabaseSidecarGateway.exportFile(table.getDatabase().getContainer().getSidecarHost(), table.getDatabase().getContainer().getSidecarPort(), filename);
- return storageService.getResource(filename);
+ dataDatabaseSidecarGateway.exportFile(table.getDatabase().getContainer().getSidecarHost(), table.getDatabase().getContainer().getSidecarPort(), fileName);
+ return storageService.getResource(fileName);
}
}
diff --git a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/ViewServiceMariaDbImpl.java b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/ViewServiceMariaDbImpl.java
index b0a66dfe0fdeea79677430d884b627d8d16b274a..1c245e9e54604472f1543ed8f4c51f96ac71933e 100644
--- a/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/ViewServiceMariaDbImpl.java
+++ b/dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/ViewServiceMariaDbImpl.java
@@ -6,6 +6,7 @@ import at.tuwien.api.database.ViewDto;
import at.tuwien.api.database.internal.PrivilegedDatabaseDto;
import at.tuwien.api.database.internal.PrivilegedViewDto;
import at.tuwien.api.database.query.QueryResultDto;
+import at.tuwien.config.S3Config;
import at.tuwien.exception.*;
import at.tuwien.gateway.DataDatabaseSidecarGateway;
import at.tuwien.mapper.MariaDbMapper;
@@ -27,13 +28,15 @@ import java.time.Instant;
@Service
public class ViewServiceMariaDbImpl extends HibernateConnector implements ViewService {
+ private final S3Config s3Config;
private final MariaDbMapper mariaDbMapper;
private final StorageService storageService;
private final DataDatabaseSidecarGateway dataDatabaseSidecarGateway;
@Autowired
- public ViewServiceMariaDbImpl(MariaDbMapper mariaDbMapper, StorageService storageService,
+ public ViewServiceMariaDbImpl(S3Config s3Config, MariaDbMapper mariaDbMapper, StorageService storageService,
DataDatabaseSidecarGateway dataDatabaseSidecarGateway) {
+ this.s3Config = s3Config;
this.mariaDbMapper = mariaDbMapper;
this.storageService = storageService;
this.dataDatabaseSidecarGateway = dataDatabaseSidecarGateway;
@@ -133,13 +136,14 @@ public class ViewServiceMariaDbImpl extends HibernateConnector implements ViewSe
public ExportResourceDto exportDataset(PrivilegedDatabaseDto database, ViewDto view, Instant timestamp)
throws SQLException, QueryMalformedException, SidecarExportException, StorageNotFoundException,
StorageUnavailableException {
- final String filename = RandomStringUtils.randomAlphabetic(40) + ".csv";
+ final String fileName = RandomStringUtils.randomAlphabetic(40) + ".csv";
+ final String filePath = s3Config.getS3FilePath() + "/" + fileName;
final ComboPooledDataSource dataSource = getPrivilegedDataSource(database);
final Connection connection = dataSource.getConnection();
try {
/* export to data database sidecar */
connection.prepareStatement(mariaDbMapper.tableOrViewToRawExportQuery(database.getInternalName(),
- view.getInternalName(), view.getColumns(), timestamp, filename))
+ view.getInternalName(), view.getColumns(), timestamp, filePath))
.executeUpdate();
connection.commit();
} catch (SQLException e) {
@@ -149,8 +153,9 @@ public class ViewServiceMariaDbImpl extends HibernateConnector implements ViewSe
} finally {
dataSource.close();
}
- dataDatabaseSidecarGateway.exportFile(database.getContainer().getSidecarHost(), database.getContainer().getSidecarPort(), filename);
- return storageService.getResource(filename);
+ dataDatabaseSidecarGateway.exportFile(database.getContainer().getSidecarHost(),
+ database.getContainer().getSidecarPort(), fileName);
+ return storageService.getResource(fileName);
}
}
diff --git a/docker-compose.yml b/docker-compose.yml
index 7b128e1d57292f39502cfb1e120b043b65104a0b..7e4a499a272379d98f56bb3d174fc84f1608bb5e 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -287,6 +287,7 @@ services:
S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID:-seaweedfsadmin}"
S3_ENDPOINT: "${S3_ENDPOINT:-http://storage-service:9000}"
S3_EXPORT_BUCKET: "${S3_EXPORT_BUCKET:-dbrepo-download}"
+ S3_FILE_PATH: "${S3_FILE_PATH:-/tmp}"
S3_IMPORT_BUCKET: "${S3_IMPORT_BUCKET:-dbrepo-upload}"
S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY:-seaweedfsadmin}"
volumes:
@@ -483,7 +484,12 @@ services:
QUEUE_NAME: ${QUEUE_NAME:-dbrepo}
REQUEUE_REJECTED: ${REQUEUE_REJECTED:-false}
ROUTING_KEY: "${ROUTING_KEY:-dbrepo.#}"
- STORAGE_SERVICE_ENDPOINT: ${BROKER_SERVICE_ENDPOINT:-http://storage-service:9000}
+ S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID:-seaweedfsadmin}"
+ S3_ENDPOINT: "${S3_ENDPOINT:-http://storage-service:9000}"
+ S3_EXPORT_BUCKET: "${S3_EXPORT_BUCKET:-dbrepo-download}"
+ S3_FILE_PATH: "${S3_FILE_PATH:-/tmp}"
+ S3_IMPORT_BUCKET: "${S3_IMPORT_BUCKET:-dbrepo-upload}"
+ S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY:-seaweedfsadmin}"
healthcheck:
test: wget -qO- localhost:8080/actuator/health/readiness | grep -q "UP" || exit 1
interval: 10s
diff --git a/helm/dbrepo/templates/data-secret.yaml b/helm/dbrepo/templates/data-secret.yaml
index 57c1ebd1a0db7202de8918f01217909ce54652ff..70cec81ca5314152e9544622b86333ba2c7eec65 100644
--- a/helm/dbrepo/templates/data-secret.yaml
+++ b/helm/dbrepo/templates/data-secret.yaml
@@ -31,8 +31,9 @@ stringData:
MIN_CONCURRENT_CONSUMERS: "{{ .Values.dataservice.consumerConcurrentMin }}"
MAX_CONCURRENT_CONSUMERS: "{{ .Values.dataservice.consumerConcurrentMax }}"
REQUEUE_REJECTED: "{{ .Values.dataservice.requeueRejected }}"
- S3_ENDPOINT: "{{ .Values.dataservice.s3.endpoint }}"
S3_ACCESS_KEY_ID: "{{ .Values.dataservice.s3.auth.username }}"
+ S3_ENDPOINT: "{{ .Values.dataservice.s3.endpoint }}"
+ S3_EXPORT_BUCKET: "{{ .Values.dataservice.s3.bucket.export }}"
+ S3_FILE_PATH: "{{ .Values.dataservice.s3.filePath }}"
S3_SECRET_ACCESS_KEY: "{{ .Values.dataservice.s3.auth.password }}"
S3_IMPORT_BUCKET: "{{ .Values.dataservice.s3.bucket.import }}"
- S3_EXPORT_BUCKET: "{{ .Values.dataservice.s3.bucket.export }}"
diff --git a/helm/dbrepo/templates/metadata-configmap.yaml b/helm/dbrepo/templates/metadata-configmap.yaml
index b0c927e9158b97bf6b147daee506465c3a536ccf..88c761643a70da48cd6032f3225ceb5459543052 100644
--- a/helm/dbrepo/templates/metadata-configmap.yaml
+++ b/helm/dbrepo/templates/metadata-configmap.yaml
@@ -4,8 +4,11 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: metadata-db-setup
- namespace: {{ $.Values.namespace }}
+ namespace: {{ .Values.namespace }}
data:
+ {{- with .Values.metadatadb.extraInitDbScripts }}
+ {{ toYaml . | nindent 2 }}
+ {{- end }}
02-setup-data.sql: |
BEGIN;
INSERT INTO `mdb_containers` (name, internal_name, image_id, host, port, sidecar_host, sidecar_port, privileged_username, privileged_password)
diff --git a/helm/dbrepo/values.yaml b/helm/dbrepo/values.yaml
index 3b2e12c65674aa751381cac3fccaa6f172081cf3..a2fbad6d501d6844c8fa2bd2fdbb7af5d20f1fd7 100644
--- a/helm/dbrepo/values.yaml
+++ b/helm/dbrepo/values.yaml
@@ -42,6 +42,7 @@ admin:
## @skip metadatadb.metrics.enabled The Prometheus settings.
## @skip metadatadb.galera The Galera settings.
## @skip metadatadb.initdbScriptsConfigMap The initial database scripts.
+## @param metadatadb.extraInitDbScripts Additional init.db scripts that are executed on the first start.
## @skip metadatadb.service The initial database scripts.
## @param metadatadb.persistence.enabled Enable persistent storage. Requires PV-provisioner.
## @param metadatadb.replicaCount The number of replicas, should be uneven (2n+1).
@@ -65,6 +66,12 @@ metadatadb:
user: mariabackup
password: mariabackup
initdbScriptsConfigMap: metadata-db-setup
+ extraInitDbScripts: {}
+# 03-additional-data.sql: |
+# BEGIN;
+# INSERT INTO `mdb_containers` (name, internal_name, image_id, host, port, sidecar_host, sidecar_port, privileged_username, privileged_password)
+# VALUES ('MariaDB Galera TEST', 'mariadb_11_1_3', 1, 'data-db', 3306, 'data-db', 80, 'root', 'dbrepo');
+# COMMIT;
service:
type: ClusterIP
annotations: { }
@@ -207,10 +214,10 @@ datadb:
periodSeconds: 30
volumeMounts:
- name: s3
- mountPath: /tmp
+ mountPath: /s3
extraVolumeMounts:
- name: s3
- mountPath: /tmp
+ mountPath: /s3
extraVolumes:
- name: s3
emptyDir: { }
@@ -496,6 +503,7 @@ metadataservice:
## @skip dataservice.s3.bucket
## @param dataservice.s3.auth.username The S3-capable endpoint username (or access key id).
## @param dataservice.s3.auth.password The S3-capable endpoint user password (or access key secret).
+## @param dataservice.s3.filePath The local location to download/upload files from/to S3-capable endpoint.
## @param dataservice.consumerConcurrentMin The minimum broker service consumer number.
## @param dataservice.consumerConcurrentMax The maximum broker service consumer number.
## @param dataservice.requeueRejected Enable re-queueing of rejected messages to the broker service.
@@ -519,6 +527,7 @@ dataservice:
auth:
username: seaweedfsadmin
password: seaweedfsadmin
+ filePath: /s3
consumerConcurrentMin: 1
consumerConcurrentMax: 5
requeueRejected: false
diff --git a/mkdocs.yml b/mkdocs.yml
index d4969e8840cabc4d63aad10005eef409553de6fb..a1b6f5aad0fec446885c1cfe3b12adbb37637011 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -14,25 +14,40 @@ nav:
- Migration Guide: migration.md
- contributing.md
- Concepts:
- - Microservice: concepts/microservice.md
- - Persistent Identifier: concepts/pid.md
- - (Meta-)Data: concepts/data.md
- - Analysis: concepts/analysis.md
+ - Overview: concepts/index.md
- Authentication: concepts/authentication.md
- - Message Broker: concepts/message-broker.md
- - Gateway: concepts/gateway.md
+ - DBMS: concepts/dbms.md
+ - Messaging: concepts/messaging.md
+ - (Meta-)Data: concepts/metadata.md
+ - Persistent Identifier: concepts/pid.md
- Search: concepts/search.md
- Storage: concepts/storage.md
- - Upload: concepts/upload.md
- - Relational Database: concepts/relational-database.md
- User Interface: concepts/ui.md
- - Usage:
- - Overview: usage-overview.md
- - Python Library: usage-python.md
+ - API:
+ - Overview: api/index.md
+ - Databases:
+ - Data Database: api/data-db.md
+ - Metadata Database: api/metadata-db.md
+ - SDK:
+ - Python Library: api/python.md
+ - Swagger / OpenAPI: api/open-api.md
- Services:
- - Authentication Service: usage-authentication.md
- - Storage Service: usage-storage.md
- - Upload Service: usage-upload.md
+ - Analyse Service: api/analyse-service.md
+ - Auth Service: api/auth-service.md
+ - Data Service: api/data-service.md
+ - Gateway Service: api/gateway-service.md
+ - Metadata Service: api/metadata-service.md
+ - Search Service: api/search-service.md
+ - Storage Service: api/storage-service.md
+ - Upload Service: api/upload-service.md
+ - UI:
+ - Customization: api/ui.md
+ - Examples:
+ - COVID-19 Tweets: examples/covid19.md
+ - Influenza Monitoring: examples/influenza.md
+ - Manufacturing Data: examples/manufacturing.md
+ - Power Usage: examples/power.md
+ - Transportation Monitoring: examples/transportation.md
- publications.md
- contact.md
extra_css: