Merge branch 'master' into release-v1.4

56e0714e · Martin Weise · 5f255eb8 · ef11f9c4 · 56e0714e · 56e0714e
Verified Commit 56e0714e authored Jan 19, 2024 by Martin Weise
--- a/.docs/images/screenshots/search-faceted.png
+++ b/.docs/images/screenshots/search-faceted.png
--- a/.docs/images/screenshots/table-amqp.png
+++ b/.docs/images/screenshots/table-amqp.png
--- a/.docs/images/statistics-example-unit-independent-search.png
+++ b/.docs/images/statistics-example-unit-independent-search.png
--- a/.docs/images/statistics-example.png
+++ b/.docs/images/statistics-example.png
--- a/.docs/images/statistics-mirror.png
+++ b/.docs/images/statistics-mirror.png
--- a/.docs/index.md
+++ b/.docs/index.md
@@ -14,8 +14,8 @@ collection. Challenges revolve around organizing, searching and retrieving conte
 constitute a major technical burden as their internal representation greatly differs from static documents most digital
 repositories are designed for.
-[Get Started](./deployment-docker-compose){ .action-button .md-button .md-button--primary }
+[System Description](./system){ .action-button .md-button .md-button--secondary }
-[Learn More](./system){ .action-button .md-button .md-button--secondary }
+[How to use](./usage-overview){ .action-button .md-button .md-button--primary }
 ## Application Areas
@@ -26,5 +26,5 @@ evolving, allows reproducing of query results and supports findable-, accessible
 ## More Information
 - Demonstration instance [https://dbrepo1.ec.tuwien.ac.at](https://dbrepo1.ec.tuwien.ac.at)
- Sandbox instance [https://test.dbrepo.tuwien.ac.at](https://dbrepo2.ec.tuwien.ac.at)
+- Test instance [https://test.dbrepo.tuwien.ac.at](https://test.dbrepo.tuwien.ac.at)
 - System description [https://doi.org/10.2218/ijdc.v17i1.825](https://doi.org/10.2218/ijdc.v17i1.825)
--- a/.docs/overrides/main.html
+++ b/.docs/overrides/main.html
@@ -4,16 +4,17 @@
 <label for="version">Version</label>
 <select style="margin-left:8px;" id="version" name="version" onchange="switchVersion(this.options[this.selectedIndex].value)">
    <option value="1.4.0">1.4.0</option>
-    <option value="1.3.9">1.3.9</option>
+    <option value="1.3.0">1.3.0</option>
    <option value="latest">latest</option>
 </select>
 <script>
    function getVersion() {
        const segments = location.pathname.split('/')
+        console.debug('segments', segments)
        if (segments.length >= 4) {
            return segments[3]
        } else {
-            return '$TAG'
+            return '$VERSION'
        }
    }
    function switchVersion(value) {

--- a/.docs/publications.md
+++ b/.docs/publications.md
@@ -6,6 +6,18 @@ hide:
 # Publications
+## Logos
+DBRepo logo in various formats:
+* [logo.png](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-ui/static/logo.png)
+* [logo.svg](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-ui/static/logo.svg)
+DBRepo logo (short) in various formats:
+* [favicon.svg](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-ui/static/favicon.svg)
+* [favicon.png](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-ui/static/favicon.png)
 ## Refereed
 ##### 2023

--- a/.docs/redirect.html
+++ b/.docs/redirect.html
 <!DOCTYPE html>
 <html lang="en">
+<!-- This file exists only once
+     in the final documentation -->
 <head>
    <meta charset="UTF-8">
    <title>Redirect Notice</title>
-    <meta http-equiv="Refresh" content="0; url='https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/latest/'" />
+    <meta http-equiv="Refresh" content="0; url='https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/$VERSION/'" />
 </head>
 <body>
 <h1>Redirect Notice</h1>
 <p>
-    This page should automatically open the documentation for version <code>latest</code>. In case this page does not load the site is
+    This page should automatically open the documentation for version <code>$VERSION</code>. In case this page does not load the site is
    available at:
 </p>
 <p>
-    <a href="https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/latest/">https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/latest/</a>
+    <a href="https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/$VERSION/">https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/$VERSION/</a>
 </p>
 </body>
 </html>
\ No newline at end of file
--- a/.docs/system-databases-auth.md
+++ b/.docs/system-databases-auth.md
@@ -8,7 +8,7 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`bitnami/mariadb:10.5`](https://hub.docker.com/r/bitnami/mariadb)
+    Image: [`bitnami/mariadb:11.2.2-debian-11-r0`](https://hub.docker.com/r/bitnami/mariadb)
    * Ports: 3306/tcp
    * JDBC: `jdbc://mariadb:<hostname>:3306`

--- a/.docs/system-databases-data.md
+++ b/.docs/system-databases-data.md
@@ -8,11 +8,18 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`bitnami/mariadb:10.5`](https://hub.docker.com/r/bitnami/mariadb)
+    Image: [`bitnami/mariadb-galera:11.2.2-debian-11-r0`](https://hub.docker.com/r/bitnami/mariadb-galera)
    * Ports: 3306/tcp
    * JDBC: `jdbc://mariadb:<hostname>:3306`
+!!! debug "Debug Information"
+    Image: [`dbrepo/data-db-sidecar:$TAG`](https://hub.docker.com/r/dbrepo/data-db-sidecar)
+    * Ports: 3305/tcp
+    * Swagger UI: `http://<hostname>:1080/swagger-ui/` <a href="../swagger/sidecar" target="_blank">:fontawesome-solid-square-up-right: view online</a>
 ## Overview
 By default, only one Data Database is deployed. You can deploy multiple (different) Data Database instances and make
@@ -21,7 +28,7 @@ them available in the repository as follows:
 ```console
 curl \
   -sSL \
-   http://<hostname>:9093/api/container \
+   http://<hostname>/api/container \
   -X POST \
   -d '{"name": "Data Database 2", "imageId": 1, "host": "example.com", "port": 3306, "privilegedUsername": "root", "privilegedPassword": "s3cr3t" }'
 ```

--- a/.docs/system-databases-metadata.md
+++ b/.docs/system-databases-metadata.md
@@ -8,7 +8,7 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`bitnami/mariadb:10.5`](https://hub.docker.com/r/bitnami/mariadb)
+    Image: [`dbrepo/metadata-db:$TAG`](https://hub.docker.com/r/dbrepo/metadata-db)
    * Ports: 3306/tcp
    * JDBC: `jdbc://mariadb:<hostname>:3306`

--- a/.docs/system-databases-search.md
+++ b/.docs/system-databases-search.md
@@ -8,11 +8,9 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/search-db:1.4`](https://hub.docker.com/r/dbrepo/search-db)
+    Image: [`dbrepo/search-db:$TAG`](https://hub.docker.com/r/dbrepo/search-db)
    * Ports: 9200/tcp
-    * Indexes: `http://:9200/_all`
-    * Health: `http://:9200/_cluster/health`
 ## Overview

--- a/.docs/system-other-search-dashboard.md
+++ b/.docs/system-other-search-dashboard.md
@@ -11,7 +11,7 @@ author: Martin Weise
    Image: [`opensearchproject/opensearch-dashboards:2.10.0`](https://hub.docker.com/r/opensearchproject/opensearch-dashboards)
    * Ports: 5601/tcp
-    * UI: `http://:5601/admin/dashboard/`
+    * UI: `http://<hostname>/admin/dashboard/`
 ## Overview

--- a/.docs/system-other-ui.md
+++ b/.docs/system-other-ui.md
@@ -8,11 +8,11 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/ui:1.4`](https://hub.docker.com/r/dbrepo/ui)
+    Image: [`dbrepo/ui:$TAG`](https://hub.docker.com/r/dbrepo/ui)
    * Ports: 3000/tcp, 9100/tcp
-    * Prometheus: `http://:9100/metrics`
+    * Prometheus: `http://<hostname>:9100/metrics`
-    * UI: `http://:3000/`
+    * UI: `http://<hostname>/`
 ## Overview
@@ -24,7 +24,7 @@ It provides a *user interface* (UI) for a researcher to interact with the databa
 </figure>
 <figure markdown>
-![UI microservice architecture detailed](images/architecture-ui.png)
+![UI microservice architecture detailed](images/architecture-ui.svg)
 <figcaption>Architecture of the UI microservice</figcaption>
 </figure>

--- a/.docs/system-services-analyse.md
+++ b/.docs/system-services-analyse.md
@@ -8,12 +8,12 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/analyse-service:1.4`](https://hub.docker.com/r/dbrepo/analyse-service)
+    Image: [`dbrepo/analyse-service:$TAG`](https://hub.docker.com/r/dbrepo/analyse-service)
    * Ports: 5000/tcp
    * Prometheus: `http://<hostname>:5000/metrics`
    * Health: `http://<hostname>:5000/health`
-    * Swagger UI: `http://<hostname>:5000/swagger-ui/index.html` <a href="../swagger/analyse" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+    * Swagger UI: `http://<hostname>:5000/swagger-ui/` <a href="../swagger/analyse" target="_blank">:fontawesome-solid-square-up-right: view online</a>
 ## Overview

--- a/.docs/system-services-authentication.md
+++ b/.docs/system-services-authentication.md
@@ -8,12 +8,12 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/authentication-service:1.4`](https://hub.docker.com/r/dbrepo/authentication-service)
+    Image: [`dbrepo/authentication-service:$TAG`](https://hub.docker.com/r/dbrepo/authentication-service)
    * Ports: 8080/tcp
-    * Admin Console: `http://<hostname>:8080/api/auth/admin/`
    * Health: `http://<hostname>:8080/api/auth/health`
    * Prometheus: `http://<hostname>:8080/api/auth/metrics`
+    * UI: `http://<hostname>/api/auth/admin/`
 ## Overview

--- a/.docs/system-services-broker.md
+++ b/.docs/system-services-broker.md
@@ -12,8 +12,8 @@ author: Martin Weise
    * Ports: 5672/tcp, 15672/tcp, 15692/tcp
    * AMQP: `amqp://<hostname>:5672`
-    * Management: `http://<hostname>:15672`
    * Prometheus: `http://<hostname>:15692/metrics`
+    * Management: `http://<hostname>/admin/broker`
 ## Overview

--- a/.docs/system-services-data.md
+++ b/.docs/system-services-data.md
@@ -8,7 +8,7 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/data-service:1.4`](https://hub.docker.com/r/dbrepo/data-service)
+    Image: [`dbrepo/data-service:$TAG`](https://hub.docker.com/r/dbrepo/data-service)
    * Ports: 9093/tcp
    * Info: `http://<hostname>:9093/actuator/info`

--- a/.docs/system-services-metadata.md
+++ b/.docs/system-services-metadata.md
@@ -8,15 +8,15 @@ author: Martin Weise
 !!! debug "Debug Information"
-    Image: [`dbrepo/metadata-service:1.4`](https://hub.docker.com/r/dbrepo/metadata-service)
+    Image: [`dbrepo/metadata-service:$TAG`](https://hub.docker.com/r/dbrepo/metadata-service)
    * Ports: 9099/tcp
-    * Info: `http://9093:9099/actuator/info`
+    * Info: `http://<hostname>:9099/actuator/info`
-    * Health: `http://9093:9099/actuator/health`
+    * Health: `http://<hostname>:9099/actuator/health`
-        - Readiness: `http://9093:9099/actuator/health/readiness`
+        - Readiness: `http://<hostname>:9099/actuator/health/readiness`
-        - Liveness: `http://9093:9099/actuator/health/liveness`
+        - Liveness: `http://<hostname>:9099/actuator/health/liveness`
-    * Prometheus: `http://9093:9099/actuator/prometheus`
+    * Prometheus: `http://<hostname>:9099/actuator/prometheus`
-    * Swagger UI: `http://9093:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+    * Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
 ## Overview
@@ -37,18 +37,39 @@ data ingest operations.
 ### Identifiers
-The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a query to
+The service is responsible for creating and resolving a *persistent identifier* (PID) attached to a database, subset,
-obtain the metadata attached to it and allow re-execution of a query. We store both the query and hashes of the query
+table or view to obtain the metadata attached to it and allow reproduction of the exact same result.
-and result set to allow equality checks of the originally obtained result set and the currently obtained result set. In
-the reference implementation we currently only use a numerical id column and plan to integrate *digital object
+This service also provides an OAI-PMH endpoint for metadata aggregators 
-identifier* (DOI) through our institutional library soon.
+(e.g. [OpenAIRE Graph](https://graph.openaire.eu/)). Through the User Interface, it also exposes metadata through
+JSON-LD to metadata aggregators (e.g. [Google Datasets](https://datasetsearch.research.google.com/)). PID metadata
-This service provides an OAI-PMH endpoint for metadata aggregators.
+is always exposed, even for private databases.
+The service generates internal PIDs, essentially representing internal URIs in 
+the [DataCite Metadata Schema 4.4](https://doi.org/10.14454/3w3z-sa82). This can be enhanced with activating the 
+external DataCite Fabrica system to generate DOIs, this is disabled by default. 
+To activate DOI minting, pass your DataCite Fabrica credentials in the environment variables:
+```yaml title="docker-compose.yml"
+services:
+  dbrepo-metadata-service:
+    image: docker.io/dbrepo/metadata-service:1.4.0
+    environment:
+      spring_profiles_active: doi
+      DATACITE_URL: https://api.datacite.org
+      DATACITE_PREFIX: 10.12345
+      DATACITE_USERNAME: username
+      DATACITE_PASSWORD: password
+  ...
+```
 ### Queries
 It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query
-execution to the raw data.
+execution to the raw data. Any stale queries (query that have been executed by users in DBRepo but were not saved) are
+periodically being deleted from the query store based on the `DELETE_STALE_QUERIES_RATE` environment variable (defaults
+to 60 seconds).
 ### Semantics
@@ -57,11 +78,12 @@ registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of U
 Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a
 Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc.
 ### Tables
 The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables
-in the [Metadata Database](../system-databases-metadata).
+in the [Metadata Database](../system-databases-metadata). Any tables that are created outside of DBRepo (e.g. directly via the JDBC API) are
+periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable, default interval is 60
+seconds).
 ### Users
@@ -69,10 +91,40 @@ The service manages users in the [Data Database](../system-databases-data)
 and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker)
 and the [Authentication Service](../system-services-authentication).
+The default configuration grants the users only very basic permissions on the databases:
+* `SELECT`
+* `CREATE`
+* `CREATE VIEW`
+* `CREATE ROUTINE`
+* `CREATE TEMPORARY TABLES`
+* `LOCK TABLES`
+* `INDEX`
+* `TRIGGER`
+* `INSERT`
+* `UPDATE`
+* `DELETE`
+This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You
+can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create
+temporary tables:
+```yaml title="docker-compose.yml"
+services:
+  dbrepo-metadata-service:
+    environment:
+      GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES
+      ...
+```
+A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/)
 ### Views
 The service manages views in the [Data Database](../system-databases-data)
-and [Metadata Database](../system-databases-metadata).
+and [Metadata Database](../system-databases-metadata). Any views that are created outside of DBRepo (e.g. directly via 
+the JDBC API) are periodically fetched by this service (based on the `OBTAIN_METADATA_RATE` environment variable,
+default interval is 60 seconds).
 ## Limitations