diff --git a/.docs/deployment-helm.md b/.docs/deployment-helm.md
index c7164af43a70595fa679704746b974a8261ed370..d7a144b8a56aa73839935feacd198b724a1a707e 100644
--- a/.docs/deployment-helm.md
+++ b/.docs/deployment-helm.md
@@ -4,60 +4,30 @@ author: Martin Weise
## TL;DR
-To install DBRepo in your existing cluster, download the sample [`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/raw/master/charts/dbrepo-core/values.yaml?inline=false)
-for your deployment and update the variables, especially `hostname`. The chart depends on
-installed [Keycloak Operator](https://www.keycloak.org/operator/installation) that can be installed following the
-official guide.
+To install DBRepo in your existing cluster, download the sample [`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/raw/dev/charts/dbrepo-core/values.yaml?inline=false)
+for your deployment and update the variables, especially `hostname`.
```shell
helm upgrade --install dbrepo \
-n dbrepo \
"oci://dbrepo.azurecr.io/helm/dbrepo-core" \
--values ./values.yaml \
- --version "0.1.3" \
+ --version "0.1.4-RC2" \
--create-namespace \
--cleanup-on-fail
```
## Dependencies
-The helm chart depends on four components:
+Our chart depends on seven other charts which will be automatically resolved when installing our `dbrepo-core` chart:
-1. [Ingress NGINX Controller](https://kubernetes.github.io/ingress-nginx/) for basic ingress.
-2. [Cert-Manager Controller](https://cert-manager.io/) for TLS certificate management with Let's Encrypt.
-3. [MariaDB Operator](https://github.com/mariadb-operator/mariadb-operator/) for creation of databases.
-4. [Keycloak Operator](https://www.keycloak.org/operator/installation) for creation of the authentication service.
-
-## Configuration before the installation
-
-Define an admin user that the services can use to communicate with
-the [Authentication Service](../system-services-authentication). You will need to manually create this user later after
-the installation.
-
-## Configuration after the installation
-
-After installing, get the initial administrator password created by the [Keycloak operator](https://www.keycloak.org/operator/basic-deployment):
-
-```shell
-kubectl -n dbrepo \
- get \
- secret \
- auth-service-initial-admin \
- -o jsonpath='{.data.password}' | base64 --decode
-```
-
-On success, the output should look like this: `1f5581a01d8e8f47f2dae08cc88f56fd` which is the initial password for the
-user `admin`. This password should be considered as *temporary* and be changed immediately now! Login into
-the [authentication service](../system-services-authentication) as `admin` and:
-
-1. Create a new user in the `master` realm.
-2. Create credentials (non-temporary) for this user in the `master` realm.
-3. Assign this user the role `admin`.
-4. Delete the user `admin`.
-
-Then import the DBRepo realm by clicking the dropdown "master" > Create Realm and import
-the [`dbrepo-realm.json`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-authentication-service/dbrepo-realm.json)
-by uploading the file *or* copying the contents and click "Create".
+* Keycloak (Bitnami, v17.3.3) for [Authentication Service](../system-services-authentication)
+* MariaDB Galera (Bitnami, v10.1.3) for [Data Database](../system-databases-data) & [Metadata Database](../system-databases-metadata)
+* MinIO (Bitnami, v12.9.4) for [Storage Service](../system-services-storage)
+* OpenSearch (OpenSearch Project, v2.16.0) for [Search Database](../system-databases-search)
+* OpenSearch Dashboards (OpenSearch Project, v2.14.0) for [Search Dashboard](../system-other-search-dashboard)
+* PostgreSQL HA (Bitnami, v12.1.7) for [Auth Database](../system-databases-auth)
+* RabbitMQ (Bitnami, v12.5.1) for [Broker Service](../system-services-broker)
### Backup
diff --git a/.docs/deployment-kubernetes-azure.md b/.docs/deployment-kubernetes-azure.md
deleted file mode 100644
index 12f0a2e7de1591d81a7492fb950f8a16ea624184..0000000000000000000000000000000000000000
--- a/.docs/deployment-kubernetes-azure.md
+++ /dev/null
@@ -1,54 +0,0 @@
----
-author: Martin Weise
----
-
-# Special Instructions for Azure Cloud
-
-You can use our pre-built Helm chart for deploying DBRepo in your Kubernetes Cluster
-with Microsoft Azure as infrastructure provider.
-
-## Requirements
-
-### Hardware
-
-For this small cloud, test deployment any public cloud provider would suffice, we recommend a
-small [Kubernetes Service](https://azure.microsoft.com/en-us/products/kubernetes-service)
-with Kubernetes version *1.24.10* and node sizes *Standard_B4ms*
-
-- 4 vCPU cores
-- 16GB RAM memory
-- 200GB SSD storage
-
-This is roughly met by selecting the *Standard_B4ms* flavor and three worker nodes.
-
-## Deployment
-
-### Databases
-
-Since Azure offers a managed [Azure Database for MariaDB](https://azure.microsoft.com/en-us/products/mariadb), we
-recommend to at least deploy the Metadata Database as high-available, managed database.
-
-!!! warning "End of Life software"
-
- Unfortunately, Azure does not (yet) support managed MariaDB 10.5, the latest version supported by Azure is 10.3
- which is End of Life (EOL) from [May 2023 onwards](https://mariadb.com/kb/en/changes-improvements-in-mariadb-10-3/).
- Microsoft decided to still maintain MariaDB 10.3
- until [September 2025](https://learn.microsoft.com/en-us/azure/mariadb/concepts-supported-versions).
-
-### Fileshare
-
-For the shared volume *PersistentVolumeClaim* `dbrepo-shared-volume-claim`, select an appropriate *StorageClass* that
-supports:
-
-1. Access mode `ReadWriteMany`
-2. Hardlinks (TUSd creates lockfiles during upload)
-
-You will need to use a *StorageClass* of either `managed-*` or `azureblob-*` (after enabling the
-proprietary [CSI driver for BLOB storage](https://learn.microsoft.com/en-us/azure/aks/azure-blob-csi?tabs=NFS#azure-blob-storage-csi-driver-features)
-in your Kubernetes Cluster).
-
-We recommend to create
-a [Container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers) for the
-[Upload Service](../system-services-upload) to deposit files and mount the BLOB storage
-via CSI drivers into the *Deployment*. It greatly increases the available interfaces (see below) for file uploads and
-provides a highly-available filesystem for the many deployments that need to use the files.
diff --git a/.docs/deployment-kubernetes-minikube.md b/.docs/deployment-kubernetes-minikube.md
deleted file mode 100644
index 7c9376c575879b92e5407728fb6bc6defd112dbe..0000000000000000000000000000000000000000
--- a/.docs/deployment-kubernetes-minikube.md
+++ /dev/null
@@ -1,139 +0,0 @@
----
-author: Martin Weise
----
-
-# Special Instructions for Minikube
-
-You can use our Helm chart for deploying DBRepo in your Kubernetes Cluster
-using [minikube](https://minikube.sigs.k8s.io/docs/start/) as infrastructure provider which deploys a single-node Kubernetes cluster on your machine,
-suitable for test-deployments.
-
-## Requirements
-
-### Virtual Machine
-
-For this small, local, test deployment any modern hardware would suffice, we recommend a dedicated virtual machine with
-the following settings. Note that most of the vCPU and RAM resources will be needed for starting the infrastructure,
-this is because of Docker. During idle times, the deployment will use significantly less resources.
-
-- 4 vCPU cores
-- 16GB RAM memory
-- 200GB SSD storage
-
-### Minikube
-
-First, install the minikube virtualization tool that provides a single-node Kubernetes environment, e.g. on a virtual
-machine. We do not regularly check these instructions, they are provided on best-effort. Check
-the [official documentation](https://minikube.sigs.k8s.io/docs/start/) for up-to-date information.
-
-For Debian:
-
-```shell
-curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
-sudo dpkg -i minikube_latest_amd64.deb
-```
-
-Start the cluster and enable basic plugins:
-
-```shell
-minikube start --driver='docker'
-minikube kubectl -- get po -A
-minikube addons enable ingress
-```
-
-### NGINX
-
-Deploy a NGINX reverse proxy on the virtual machine to reach your minikube cluster from the public Internet:
-
-```nginx title="/etc/nginx/conf.d/dbrepo.conf"
-resolver 127.0.0.11 valid=30s;
-
-server {
- listen 80;
- server_name _;
-
- location / {
- proxy_set_header Host $host;
- proxy_set_header X-Forwarded-For $remote_addr;
- proxy_pass http://CLUSTER_IP;
- }
-}
-
-server {
- listen 443 ssl;
- server_name DOMAIN_NAME;
- ssl_certificate /etc/nginx/certificate.crt;
- ssl_certificate_key /etc/nginx/certificate.key;
-
- location / {
- proxy_set_header Host $host;
- proxy_set_header X-Forwarded-For $remote_addr;
- proxy_pass https://CLUSTER_IP;
- }
-}
-```
-
-Replace `CLUSTER_IP` with the result of:
-
- $ minikube ip
- 192.168.49.2
-
-Replace `DOMAIN_NAME` with the domain name. You will need also a valid TLS certificate with private key for TLS enabled
-in the cluster. In our test deployment we obtained a certificate from Let's Encrypt.
-
-### Fileshare
-
-Since the Upload Service uses a shared filesystem with the [Analyst Service](../system-services-analyse),
-[Metadata Service](../system-services-metadata) and
-[Data Database](../system-databases-data), the dynamic provision of the *PersistentVolume*
-by the *PersistentVolumeClaim*
-of [`pvc.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/blob/master/charts/dbrepo-core/templates/upload-service/pvc.yaml)
-needs to happen statically. You can make use of the host's filesystem and mount it in each of those deployments.
-
-For example, mount the *hostPath* directly in
-the [`deployment.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/blob/master/charts/dbrepo-core/templates/analyse-service/deployment.yaml).
-
-```yaml title="deployment.yaml"
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: analyse-service
- ...
-spec:
- template:
- spec:
- containers:
- - name: analyse-service
- volumeMounts:
- - name: shared
- hostPath: /path/of/host
- mountPath: /mnt/shared
- ...
-```
-
-## Deployment
-
-To install the DBRepo Helm Chart, download and edit
-the [`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/raw/master/charts/dbrepo-minikube/values.yaml?inline=false)
-file. At minimum you need to change the values for:
-
-* `hostname`, set to your domain, e.g. `subdomain.example.com`
-* `authAdminApiUrl`, similarly but with https and the api to the keycloak server, e.g. `https://subdomain.example.com/api/auth`
-
-It is advised to also change the usernames and passwords for all credentials. Next, install the chart using your edited
-`values.yaml` file:
-
-!!! info "Documentation of values.yaml"
-
- We documented all values in the `values.yaml` file [here](http://127.0.0.1:8000/deployment-helm/#chart-values) with
- default values and description for each value.
-
-```shell
-helm upgrade --install dbrepo \
- -n dbrepo \
- "oci://dbrepo.azurecr.io/helm/dbrepo-core" \
- --values ./values.yaml \
- --version "0.1.3" \
- --create-namespace \
- --cleanup-on-fail
-```
diff --git a/.docs/images/architecture.drawio b/.docs/images/architecture.drawio
index ff24e74e84ab219896bbff012e141df72f7f5140..025aef75dc8ed81db45c57998133f3ab8b316ee9 100644
--- a/.docs/images/architecture.drawio
+++ b/.docs/images/architecture.drawio
@@ -1,4 +1,4 @@
-<mxfile host="Electron" modified="2023-11-17T11:53:24.549Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.1.2 Chrome/106.0.5249.199 Electron/21.4.3 Safari/537.36" etag="BtfeOmnXFlr2YLKq2VZ0" version="21.1.2" type="device" pages="7">
+<mxfile host="Electron" modified="2023-11-22T22:51:30.674Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.1.2 Chrome/106.0.5249.199 Electron/21.4.3 Safari/537.36" etag="rfDZiCzC6Lc0py7Ucjta" version="21.1.2" type="device" pages="7">
<diagram id="mvBsv1rP8O80Qe3yGnn_" name="docker-compose">
<mxGraphModel dx="1434" dy="822" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<root>
@@ -798,38 +798,38 @@
</mxGraphModel>
</diagram>
<diagram id="n3Gsc6DDUkQ8nNTTz0wk" name="data-db">
- <mxGraphModel dx="819" dy="-180" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1019" pageHeight="650" math="0" shadow="0">
+ <mxGraphModel dx="1434" dy="172" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1019" pageHeight="650" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-9" value="shared filesystem<br>/tmp" style="rounded=1;whiteSpace=wrap;html=1;arcSize=3;verticalAlign=bottom;fontStyle=2" vertex="1" parent="1">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-9" value="shared filesystem<br>/tmp" style="rounded=1;whiteSpace=wrap;html=1;arcSize=3;verticalAlign=bottom;fontStyle=2" parent="1" vertex="1">
<mxGeometry x="425" y="840" width="248" height="130" as="geometry" />
</mxCell>
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-11" value="jdbc" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;exitPerimeter=0;endArrow=none;endFill=0;startArrow=classic;startFill=1;" edge="1" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-1">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-11" value="jdbc" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;exitPerimeter=0;endArrow=none;endFill=0;startArrow=classic;startFill=1;" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-1" edge="1">
<mxGeometry x="0.3769" relative="1" as="geometry">
<mxPoint x="472.71428571428555" y="810" as="targetPoint" />
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-1" value="" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=8.600000000000023;fillColor=#dae8fc;strokeColor=#000000;" vertex="1" parent="1">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-1" value="" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=8.600000000000023;fillColor=#dae8fc;strokeColor=#000000;" parent="1" vertex="1">
<mxGeometry x="447.5" y="857" width="50" height="64" as="geometry" />
</mxCell>
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-2" value="data-db" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;spacing=-1;" vertex="1" parent="1">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-2" value="data-db" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;spacing=-1;" parent="1" vertex="1">
<mxGeometry x="431.5" y="919" width="85" height="20" as="geometry" />
</mxCell>
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-12" value="http" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;startArrow=classic;startFill=1;endArrow=none;endFill=0;" edge="1" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-7">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-12" value="http" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;startArrow=classic;startFill=1;endArrow=none;endFill=0;" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-7" edge="1">
<mxGeometry x="0.4743" relative="1" as="geometry">
<mxPoint x="585.0952380952381" y="810" as="targetPoint" />
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
- <mxCell id="m0IQrUpga-DAo2afT193-3" value="S3" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;startArrow=classic;startFill=1;" edge="1" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-7" target="m0IQrUpga-DAo2afT193-1">
+ <mxCell id="m0IQrUpga-DAo2afT193-3" value="S3" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;startArrow=classic;startFill=1;" parent="1" source="S8wz9ZtwZs3Sd4maCRdY-7" target="m0IQrUpga-DAo2afT193-1" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
- <mxCell id="S8wz9ZtwZs3Sd4maCRdY-7" value="Data DB Sidecar" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
+ <mxCell id="S8wz9ZtwZs3Sd4maCRdY-7" value="Data DB Sidecar" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
<mxGeometry x="520" y="869" width="130" height="40" as="geometry" />
</mxCell>
- <mxCell id="m0IQrUpga-DAo2afT193-1" value="Storage Service<br>(minIO)" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
+ <mxCell id="m0IQrUpga-DAo2afT193-1" value="Storage Service<br>(minIO)" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
<mxGeometry x="720" y="869" width="130" height="40" as="geometry" />
</mxCell>
</root>
diff --git a/.docs/images/exchange-binding.png b/.docs/images/exchange-binding.png
new file mode 100644
index 0000000000000000000000000000000000000000..92517861c5b915f30d524dc5f785b30c2ca9b77a
Binary files /dev/null and b/.docs/images/exchange-binding.png differ
diff --git a/.docs/images/queue-quorum.png b/.docs/images/queue-quorum.png
new file mode 100644
index 0000000000000000000000000000000000000000..42897a0691b9c6bd814a76e4cef926c5bdbd8939
Binary files /dev/null and b/.docs/images/queue-quorum.png differ
diff --git a/.docs/system-services-broker.md b/.docs/system-services-broker.md
index ff259c461506ee423f2d4248cc6e622a87b79629..f7919f8bd2855a0c3db7e9fcd215b87fe2d1e6bd 100644
--- a/.docs/system-services-broker.md
+++ b/.docs/system-services-broker.md
@@ -21,7 +21,7 @@ It holds exchanges and topics responsible for holding AMQP messages for later co
use [RabbitMQ](https://www.rabbitmq.com/) in the implementation. By default, the endpoint listens to the insecure port `5672` for incoming
AMQP tuples and insecure port `15672` for the management UI.
-The default configuration creates a user with administrative privileges:
+The default configuration creates a user with administrative privileges on the default virtual host `dbrepo`:
* Username: `fda`
* Password: `fda`
@@ -35,6 +35,22 @@ The Broker Service allows two ways of authentication:
For detailed examples how to authenticate with the Broker Service see
the [usage](/usage-broker) page.
+The architecture of the Broker Service is very simple. There is only one durable, topic exchange `dbrepo` and one quorum
+queue `dbrepo`, connected with a binding of `dbrepo.#` which routes all tuples with routing key prefix `dbrepo.` (mind
+the dot!) to this queue.
+
+<figure markdown>
+ 
+ <figcaption>Replicated quorum queue dbrepo in a cluster with three nodes</figcaption>
+</figure>
+
+The consumer takes care of writing it to the correct table in the [Data Service](../system-services-data).
+
+<figure markdown>
+ 
+ <figcaption>Architecture Broker Service</figcaption>
+</figure>
+
## Limitations
* No support for MQTT in the [Metadata Service](../system-services-metadata)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0f4a06f3286ad5bb400feb131eb8461d3082b48e..f0feb7e23396e18bc323014992004a7b03bfbad1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -349,9 +349,9 @@ scan-data-db:
- master
allow_failure: true
script:
- - trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-data-db-report.json docker.io/bitnami/mariadb:10.5
- - trivy image --insecure --exit-code 0 docker.io/bitnami/mariadb:10.5
- - trivy image --insecure --exit-code 1 --severity CRITICAL docker.io/bitnami/mariadb:10.5
+ - trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-data-db-report.json docker.io/bitnami/mariadb:11.1.3
+ - trivy image --insecure --exit-code 0 docker.io/bitnami/mariadb:11.1.3
+ - trivy image --insecure --exit-code 1 --severity CRITICAL docker.io/bitnami/mariadb:11.1.3
cache:
paths:
- .trivycache/
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 81ad2a31a9a2707ab5c87af223d80c1e13ff9fe1..2ed25b07c2bbba6ed0c21d9c5deb1514775dbddd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -57,7 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
-- Metadata database to use a system-versioned MariaDB 10.5 database
+- Metadata database to use a system-versioned MariaDB 11.1.3 database
- Query Store to use trigger for query result count and query hash as well as result hash calculation
- Query service to allow XML/CSV export for PIDs
- Query service to support subsets of views
diff --git a/Makefile b/Makefile
index a9c6b7d970c94dd4b65d329741751034bab947a6..5bc096951bef0feccda2bcad8a95d7f120769393 100644
--- a/Makefile
+++ b/Makefile
@@ -211,10 +211,10 @@ scan-search-dashboard:
trivy image --insecure --exit-code 1 --severity CRITICAL "opensearchproject/opensearch-dashboards:2.10.0"
scan-data-db:
- docker pull "bitnami/mariadb:10.5"
- trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-data-db-report.json "bitnami/mariadb:10.5"
- trivy image --insecure --exit-code 0 "bitnami/mariadb:10.5"
- trivy image --insecure --exit-code 1 --severity CRITICAL "bitnami/mariadb:10.5"
+ docker pull "bitnami/mariadb:11.1.3"
+ trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-data-db-report.json "bitnami/mariadb:11.1.3"
+ trivy image --insecure --exit-code 0 "bitnami/mariadb:11.1.3"
+ trivy image --insecure --exit-code 1 --severity CRITICAL "bitnami/mariadb:11.1.3"
scan-ui:
trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-ui-report.json dbrepo-ui:latest
diff --git a/dbrepo-data-db/sidecar/Dockerfile b/dbrepo-data-db/sidecar/Dockerfile
index 0d307ae675c01aa543d1da0dfd6164cdb771a7f3..755ee168eba7f9d1005319385f24b11055121f0a 100644
--- a/dbrepo-data-db/sidecar/Dockerfile
+++ b/dbrepo-data-db/sidecar/Dockerfile
@@ -2,7 +2,7 @@ FROM python:3.10-alpine
RUN apk add bash curl jq
-WORKDIR /app
+ENV PYTHONFAULTHANDLER=1
COPY Pipfile Pipfile.lock ./
@@ -10,18 +10,18 @@ RUN pip install pipenv && \
pipenv install gunicorn && \
pipenv install --system --deploy
-USER 1000
+USER 1001
+
+WORKDIR /app
-COPY ./clients ./clients
-COPY ./ds-yml ./ds-yml
-COPY ./app.py ./app.py
+COPY --chown=1001 ./clients ./clients
+COPY --chown=1001 ./ds-yml ./ds-yml
+COPY --chown=1001 ./app.py ./app.py
ENV S3_STORAGE_ENDPOINT="http://storage-service:9000"
ENV S3_ACCESS_KEY_ID="minioadmin"
ENV S3_SECRET_ACCESS_KEY="minioadmin"
-RUN ls -la ./clients
-
EXPOSE 3305
-ENTRYPOINT [ "gunicorn", "-w", "4", "-b", ":3305", "app:app" ]
+ENTRYPOINT [ "gunicorn", "--log-level", "DEBUG", "--workers", "4", "--bind", ":3305", "app:app" ]
diff --git a/dbrepo-data-db/sidecar/clients/minio_client.py b/dbrepo-data-db/sidecar/clients/minio_client.py
index a3f27dbb07e848d0a7823b63f5d717e09df65a9a..9f38a83b497d5ebd4faadcd859a44dd244f4705e 100644
--- a/dbrepo-data-db/sidecar/clients/minio_client.py
+++ b/dbrepo-data-db/sidecar/clients/minio_client.py
@@ -55,6 +55,7 @@ class MinioClient:
def file_exists(self, bucket, filename):
try:
self.client.head_object(Bucket=bucket, Key=filename)
+ logging.debug(f"file with name {filename} exists in bucket with name {bucket}")
except ClientError as e:
if e.response["Error"]["Code"] == "404":
logging.error("Failed to find key %s in bucket %s", filename, bucket)
@@ -66,6 +67,7 @@ class MinioClient:
def bucket_exists_or_exit(self, bucket):
try:
self.client.head_bucket(Bucket=bucket)
+ logging.debug(f"bucket {bucket} exists.")
except ClientError as e:
if e.response["Error"]["Code"] == "404":
logging.error("Failed to find bucket %s", bucket)
diff --git a/dbrepo-metadata-db/setup-schema.sql b/dbrepo-metadata-db/setup-schema.sql
index c42a754e863da64daf563dc29821076e0d8c7891..c418a3f4227e51b40e3785a22e2daadd3895fab4 100644
--- a/dbrepo-metadata-db/setup-schema.sql
+++ b/dbrepo-metadata-db/setup-schema.sql
@@ -1,560 +1,560 @@
-BEGIN;
-
-CREATE TABLE IF NOT EXISTS `mdb_users`
-(
- id character varying(36) NOT NULL,
- username character varying(255) NOT NULL,
- firstname character varying(255),
- lastname character varying(255),
- email character varying(255) NOT NULL,
- orcid character varying(255),
- affiliation character varying(255),
- mariadb_password character varying(255) NOT NULL,
- theme_dark boolean,
- PRIMARY KEY (id),
- UNIQUE (username),
- UNIQUE (email)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_images`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- name character varying(255) NOT NULL,
- version character varying(255) NOT NULL,
- default_port integer NOT NULL,
- dialect character varying(255) NOT NULL,
- driver_class character varying(255) NOT NULL,
- jdbc_method character varying(255) NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- last_modified timestamp,
- PRIMARY KEY (id),
- UNIQUE (name, version)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_images_date`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- iid bigint NOT NULL,
- database_format character varying(255) NOT NULL,
- unix_format character varying(255) NOT NULL,
- example character varying(255) NOT NULL,
- has_time boolean NOT NULL,
- created_at timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (id),
- FOREIGN KEY (iid) REFERENCES mdb_images (id),
- UNIQUE (database_format, unix_format, example)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_containers`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- internal_name character varying(255) NOT NULL,
- name character varying(255) NOT NULL,
- host character varying(255) NOT NULL,
- port integer NOT NULL default 3306,
- ui_host character varying(255) NOT NULL default host,
- ui_port integer NOT NULL default port,
- ui_additional_flags text,
- sidecar_host character varying(255) NOT NULL,
- sidecar_port integer NOT NULL default 3305,
- image_id bigint NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- last_modified timestamp,
- privileged_username character varying(255) NOT NULL,
- privileged_password character varying(255) NOT NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (image_id) REFERENCES mdb_images (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_data`
-(
- ID bigint NOT NULL AUTO_INCREMENT,
- PROVENANCE text,
- FileEncoding text,
- FileType character varying(100),
- Version text,
- Seperator text,
- PRIMARY KEY (ID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_licenses`
-(
- identifier character varying(255) NOT NULL,
- uri text NOT NULL,
- PRIMARY KEY (identifier),
- UNIQUE (uri(200))
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_databases`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- cid bigint NOT NULL,
- name character varying(255) NOT NULL,
- internal_name character varying(255) NOT NULL,
- exchange_name character varying(255) NOT NULL,
- description text,
- engine character varying(20),
- is_public boolean NOT NULL DEFAULT TRUE,
- created_by character varying(36),
- owned_by character varying(36),
- contact_person character varying(36),
- created timestamp NOT NULL DEFAULT NOW(),
- last_modified timestamp,
- PRIMARY KEY (id),
- FOREIGN KEY (cid) REFERENCES mdb_containers (id) /* currently we only support one-to-one */,
- FOREIGN KEY (created_by) REFERENCES mdb_users (id),
- FOREIGN KEY (owned_by) REFERENCES mdb_users (id),
- FOREIGN KEY (contact_person) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_databases_subjects`
-(
- dbid BIGINT NOT NULL,
- subjects character varying(255) NOT NULL,
- PRIMARY KEY (dbid, subjects)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_tables`
-(
- ID bigint NOT NULL AUTO_INCREMENT,
- tDBID bigint NOT NULL,
- internal_name character varying(255) NOT NULL,
- queue_name character varying(255) NOT NULL,
- routing_key character varying(255) NOT NULL,
- tName VARCHAR(50),
- tDescription TEXT,
- NumCols INTEGER,
- NumRows INTEGER,
- `separator` CHAR(1),
- quote CHAR(1),
- element_null VARCHAR(50),
- skip_lines BIGINT,
- element_true VARCHAR(50),
- element_false VARCHAR(50),
- Version TEXT,
- created timestamp NOT NULL DEFAULT NOW(),
- versioned boolean not null default true,
- created_by character varying(36) NOT NULL,
- owned_by character varying(36) NOT NULL,
- last_modified timestamp,
- PRIMARY KEY (ID),
- FOREIGN KEY (tDBID) REFERENCES mdb_databases (id),
- FOREIGN KEY (created_by) REFERENCES mdb_users (id),
- FOREIGN KEY (owned_by) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns`
-(
- ID bigint NOT NULL AUTO_INCREMENT,
- tID bigint NOT NULL,
- dfID bigint,
- cName VARCHAR(100),
- internal_name VARCHAR(100) NOT NULL,
- alias VARCHAR(100),
- Datatype ENUM ('CHAR','VARCHAR','BINARY','VARBINARY','TINYBLOB','TINYTEXT','TEXT','BLOB','MEDIUMTEXT','MEDIUMBLOB','LONGTEXT','LONGBLOB','ENUM','SET','BIT','TINYINT','BOOL','SMALLINT','MEDIUMINT','INT','BIGINT','FLOAT','DOUBLE','DECIMAL','DATE','DATETIME','TIMESTAMP','TIME','YEAR'),
- length INT NULL,
- ordinal_position INTEGER NOT NULL,
- is_primary_key BOOLEAN NOT NULL,
- index_length INT NULL,
- size INT,
- d INT,
- auto_generated BOOLEAN DEFAULT false,
- is_null_allowed BOOLEAN NOT NULL DEFAULT true,
- created timestamp NOT NULL DEFAULT NOW(),
- last_modified timestamp,
- FOREIGN KEY (tID) REFERENCES mdb_tables (ID),
- PRIMARY KEY (ID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_enums`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- column_id bigint NOT NULL,
- value CHARACTER VARYING(255) NOT NULL,
- FOREIGN KEY (column_id) REFERENCES mdb_columns (ID),
- PRIMARY KEY (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_sets`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- column_id bigint NOT NULL,
- value CHARACTER VARYING(255) NOT NULL,
- FOREIGN KEY (column_id) REFERENCES mdb_columns (ID),
- PRIMARY KEY (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_nom`
-(
- tID bigint,
- cID bigint,
- maxlength INTEGER,
- last_modified timestamp,
- created timestamp NOT NULL DEFAULT NOW(),
- FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
- PRIMARY KEY (tID, cID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_num`
-(
- tID bigint,
- cID bigint,
- SIunit TEXT,
- MaxVal NUMERIC,
- MinVal NUMERIC,
- Mean NUMERIC,
- Median NUMERIC,
- Sd Numeric,
--- Histogram INTEGER[],
- last_modified timestamp,
- created timestamp NOT NULL DEFAULT NOW(),
- FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
- PRIMARY KEY (tID, cID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_cat`
-(
- tID bigint,
- cID bigint,
- num_cat INTEGER,
--- cat_array TEXT[],
- last_modified timestamp,
- created timestamp NOT NULL DEFAULT NOW(),
- FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
- PRIMARY KEY (tID, cID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_constraints_foreign_key`
-(
- fkid BIGINT NOT NULL AUTO_INCREMENT,
- tid BIGINT NOT NULL,
- rtid BIGINT NOT NULL,
- on_update VARCHAR(50) NULL,
- on_delete VARCHAR(50) NULL,
- position INT NULL,
- PRIMARY KEY (fkid),
- FOREIGN KEY (tid) REFERENCES mdb_tables (id),
- FOREIGN KEY (rtid) REFERENCES mdb_tables (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_constraints_foreign_key_reference`
-(
- id BIGINT NOT NULL AUTO_INCREMENT,
- fkid BIGINT NOT NULL,
- cid BIGINT NOT NULL,
- rcid BIGINT NOT NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (fkid) REFERENCES mdb_constraints_foreign_key (fkid) ON UPDATE CASCADE,
- FOREIGN KEY (cid) REFERENCES mdb_columns (id),
- FOREIGN KEY (rcid) REFERENCES mdb_columns (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_constraints_unique`
-(
- uid BIGINT NOT NULL AUTO_INCREMENT,
- tid BIGINT NOT NULL,
- position INT NULL,
- PRIMARY KEY (uid),
- FOREIGN KEY (tid) REFERENCES mdb_tables (id)
-);
-
-CREATE TABLE IF NOT EXISTS `mdb_constraints_unique_columns`
-(
- id BIGINT NOT NULL AUTO_INCREMENT,
- uid BIGINT NOT NULL,
- cid BIGINT NOT NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (uid) REFERENCES mdb_constraints_unique (uid),
- FOREIGN KEY (cid) REFERENCES mdb_columns (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_constraints_checks`
-(
- id BIGINT NOT NULL AUTO_INCREMENT,
- tid BIGINT NOT NULL,
- checks VARCHAR(255) NOT NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (tid) REFERENCES mdb_tables (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_concepts`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- uri text not null,
- name VARCHAR(255) null,
- description TEXT null,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (id),
- UNIQUE (uri(200))
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_units`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- uri text not null,
- name VARCHAR(255) null,
- description TEXT null,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (id),
- UNIQUE (uri(200))
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_concepts`
-(
- id bigint NOT NULL,
- cID bigint NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (id),
- FOREIGN KEY (cID) REFERENCES mdb_columns (ID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_columns_units`
-(
- id bigint NOT NULL,
- cID bigint NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (id),
- FOREIGN KEY (cID) REFERENCES mdb_columns (ID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_view`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- vdbid bigint NOT NULL,
- vName VARCHAR(255) NOT NULL,
- internal_name VARCHAR(255) NOT NULL,
- Query TEXT NOT NULL,
- query_hash VARCHAR(255) NOT NULL,
- Public BOOLEAN NOT NULL,
- NumCols INTEGER,
- NumRows INTEGER,
- InitialView BOOLEAN NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- last_modified timestamp,
- created_by character varying(36) NOT NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (vdbid) REFERENCES mdb_databases (id),
- FOREIGN KEY (created_by) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_banner_messages`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- type ENUM ('ERROR', 'WARNING', 'INFO') NOT NULL default 'INFO',
- message TEXT NOT NULL,
- link TEXT NULL,
- link_text VARCHAR(255) NULL,
- display_start timestamp NULL,
- display_end timestamp NULL,
- PRIMARY KEY (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_ontologies`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- prefix VARCHAR(8) NOT NULL,
- uri TEXT NOT NULL,
- uri_pattern TEXT,
- sparql_endpoint TEXT NULL,
- last_modified timestamp,
- created timestamp NOT NULL DEFAULT NOW(),
- UNIQUE (prefix),
- UNIQUE (uri(200)),
- PRIMARY KEY (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_view_columns`
-(
- id BIGINT NOT NULL AUTO_INCREMENT,
- cid BIGINT NOT NULL,
- vid BIGINT NOT NULL,
- position INTEGER NULL,
- PRIMARY KEY (id),
- FOREIGN KEY (vid) REFERENCES mdb_view (id),
- FOREIGN KEY (cid) REFERENCES mdb_columns (ID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifiers`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- dbid bigint,
- qid bigint,
- vid bigint,
- publisher VARCHAR(255) NOT NULL,
- language VARCHAR(2),
- visibility ENUM ('SELF', 'EVERYONE') NOT NULL default 'EVERYONE',
- publication_year INTEGER NOT NULL,
- publication_month INTEGER,
- publication_day INTEGER,
- identifier_type ENUM ('DATABASE', 'SUBSET', 'VIEW') NOT NULL,
- query TEXT,
- query_normalized TEXT,
- query_hash VARCHAR(255),
- execution timestamp,
- result_hash VARCHAR(255),
- result_number bigint,
- doi VARCHAR(255),
- created timestamp NOT NULL DEFAULT NOW(),
- created_by character varying(36) NOT NULL,
- last_modified timestamp,
- PRIMARY KEY (id), /* must be a single id from persistent identifier concept */
- FOREIGN KEY (dbid) REFERENCES mdb_databases (id),
- UNIQUE (dbid, qid),
- FOREIGN KEY (created_by) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifier_licenses`
-(
- pid bigint NOT NULL,
- license_id VARCHAR(255) NOT NULL,
- PRIMARY KEY (pid, license_id),
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id),
- FOREIGN KEY (license_id) REFERENCES mdb_licenses (identifier)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifier_titles`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- pid bigint NOT NULL,
- title text NOT NULL,
- title_type ENUM ('ALTERNATIVE_TITLE', 'SUBTITLE', 'TRANSLATED_TITLE', 'OTHER'),
- language VARCHAR(2),
- PRIMARY KEY (id),
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifier_funders`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- pid bigint NOT NULL,
- funder_name VARCHAR(255) NOT NULL,
- funder_identifier TEXT,
- funder_identifier_type ENUM ('CROSSREF_FUNDER_ID', 'GRID', 'ISNI', 'ROR', 'OTHER'),
- scheme_uri text,
- award_number VARCHAR(255),
- award_title text,
- language VARCHAR(255),
- PRIMARY KEY (id),
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifier_descriptions`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- pid bigint NOT NULL,
- description text NOT NULL,
- description_type ENUM ('ABSTRACT', 'METHODS', 'SERIES_INFORMATION', 'TABLE_OF_CONTENTS', 'TECHNICAL_INFO', 'OTHER'),
- language VARCHAR(2),
- PRIMARY KEY (id),
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_related_identifiers`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- pid bigint NOT NULL,
- value varchar(255) NOT NULL,
- type varchar(255),
- relation varchar(255),
- PRIMARY KEY (id), /* must be a single id from persistent identifier concept */
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id),
- UNIQUE (pid, value)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_identifier_creators`
-(
- id bigint NOT NULL AUTO_INCREMENT,
- pid bigint NOT NULL,
- given_names text,
- family_name text,
- creator_name VARCHAR(255) NOT NULL,
- name_type ENUM ('PERSONAL', 'ORGANIZATIONAL') default 'PERSONAL',
- name_identifier text,
- name_identifier_scheme ENUM ('ROR', 'GRID', 'ISNI', 'ORCID'),
- name_identifier_scheme_uri text,
- affiliation VARCHAR(255),
- affiliation_identifier text,
- affiliation_identifier_scheme ENUM ('ROR', 'GRID', 'ISNI'),
- affiliation_identifier_scheme_uri text,
- PRIMARY KEY (id),
- FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_feed`
-(
- fDBID bigint,
- fID bigint,
- fUserId character varying(36) not null,
- fDataID bigint REFERENCES mdb_data (ID),
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (fDBID, fID, fUserId, fDataID),
- FOREIGN KEY (fDBID, fID) REFERENCES mdb_tables (tDBID, ID),
- FOREIGN KEY (fUserId) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_update`
-(
- uUserID character varying(255) NOT NULL,
- uDBID bigint NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (uUserID, uDBID),
- FOREIGN KEY (uDBID) REFERENCES mdb_databases (id)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_access`
-(
- aUserID character varying(255) NOT NULL,
- aDBID bigint REFERENCES mdb_databases (id),
- attime TIMESTAMP,
- download BOOLEAN,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (aUserID, aDBID)
-) WITH SYSTEM VERSIONING;
-
-CREATE TABLE IF NOT EXISTS `mdb_have_access`
-(
- user_id character varying(36) NOT NULL,
- database_id bigint REFERENCES mdb_databases (id),
- access_type ENUM ('READ', 'WRITE_OWN', 'WRITE_ALL') NOT NULL,
- created timestamp NOT NULL DEFAULT NOW(),
- PRIMARY KEY (user_id, database_id),
- FOREIGN KEY (user_id) REFERENCES mdb_users (id)
-) WITH SYSTEM VERSIONING;
-
-COMMIT;
-BEGIN;
-
-INSERT INTO `mdb_licenses` (identifier, uri)
-VALUES ('MIT', 'https://opensource.org/licenses/MIT'),
- ('GPL-3.0-only', 'https://www.gnu.org/licenses/gpl-3.0-standalone.html'),
- ('BSD-3-Clause', 'https://opensource.org/licenses/BSD-3-Clause'),
- ('BSD-4-Clause', 'http://directory.fsf.org/wiki/License:BSD_4Clause'),
- ('Apache-2.0', 'https://opensource.org/licenses/Apache-2.0'),
- ('CC0-1.0', 'https://creativecommons.org/publicdomain/zero/1.0/legalcode'),
- ('CC-BY-4.0', 'https://creativecommons.org/licenses/by/4.0/legalcode');
-
-INSERT INTO `mdb_images` (name, version, default_port, dialect, driver_class, jdbc_method)
-VALUES ('mariadb', '10.5', 3306, 'org.hibernate.dialect.MariaDBDialect', 'org.mariadb.jdbc.Driver', 'mariadb');
-
-INSERT INTO `mdb_images_date` (iid, database_format, unix_format, example, has_time)
-VALUES (1, '%Y-%c-%d %H:%i:%S.%f', 'yyyy-MM-dd HH:mm:ss.SSSSSS', '2022-01-30 13:44:25.499', true),
- (1, '%Y-%c-%d %H:%i:%S', 'yyyy-MM-dd HH:mm:ss', '2022-01-30 13:44:25', true),
- (1, '%Y-%c-%d', 'yyyy-MM-dd', '2022-01-30', false),
- (1, '%H:%i:%S', 'HH:mm:ss', '13:44:25', true);
-
-INSERT INTO `mdb_ontologies` (prefix, uri, uri_pattern, sparql_endpoint)
-VALUES ('om', 'http://www.ontology-of-units-of-measure.org/resource/om-2/',
- 'http://www.ontology-of-units-of-measure.org/resource/om-2/.*', null),
- ('wd', 'http://www.wikidata.org/', 'http://www.wikidata.org/entity/.*', 'https://query.wikidata.org/sparql'),
- ('mo', 'http://purl.org/ontology/mo/', 'http://purl.org/ontology/mo/.*', null),
- ('dc', 'http://purl.org/dc/elements/1.1/', null, null),
- ('xsd', 'http://www.w3.org/2001/XMLSchema#', null, null),
- ('tl', 'http://purl.org/NET/c4dm/timeline.owl#', null, null),
- ('foaf', 'http://xmlns.com/foaf/0.1/', null, null),
- ('schema', 'http://schema.org/', null, null),
- ('rdf', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', null, null),
- ('rdfs', 'http://www.w3.org/2000/01/rdf-schema#', null, null),
- ('owl', 'http://www.w3.org/2002/07/owl#', null, null),
- ('prov', 'http://www.w3.org/ns/prov#', null, null),
- ('db', 'http://dbpedia.org', 'http://dbpedia.org/ontology/.*', 'http://dbpedia.org/sparql');
-COMMIT;
+ BEGIN;
+
+ CREATE TABLE IF NOT EXISTS `mdb_users`
+ (
+ id character varying(36) NOT NULL,
+ username character varying(255) NOT NULL,
+ firstname character varying(255),
+ lastname character varying(255),
+ email character varying(255) NOT NULL,
+ orcid character varying(255),
+ affiliation character varying(255),
+ mariadb_password character varying(255) NOT NULL,
+ theme_dark boolean,
+ PRIMARY KEY (id),
+ UNIQUE (username),
+ UNIQUE (email)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_images`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ name character varying(255) NOT NULL,
+ version character varying(255) NOT NULL,
+ default_port integer NOT NULL,
+ dialect character varying(255) NOT NULL,
+ driver_class character varying(255) NOT NULL,
+ jdbc_method character varying(255) NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ last_modified timestamp,
+ PRIMARY KEY (id),
+ UNIQUE (name, version)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_images_date`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ iid bigint NOT NULL,
+ database_format character varying(255) NOT NULL,
+ unix_format character varying(255) NOT NULL,
+ example character varying(255) NOT NULL,
+ has_time boolean NOT NULL,
+ created_at timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (id),
+ FOREIGN KEY (iid) REFERENCES mdb_images (id),
+ UNIQUE (database_format, unix_format, example)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_containers`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ internal_name character varying(255) NOT NULL,
+ name character varying(255) NOT NULL,
+ host character varying(255) NOT NULL,
+ port integer NOT NULL default 3306,
+ ui_host character varying(255) NOT NULL default host,
+ ui_port integer NOT NULL default port,
+ ui_additional_flags text,
+ sidecar_host character varying(255) NOT NULL,
+ sidecar_port integer NOT NULL default 3305,
+ image_id bigint NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ last_modified timestamp,
+ privileged_username character varying(255) NOT NULL,
+ privileged_password character varying(255) NOT NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (image_id) REFERENCES mdb_images (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_data`
+ (
+ ID bigint NOT NULL AUTO_INCREMENT,
+ PROVENANCE text,
+ FileEncoding text,
+ FileType character varying(100),
+ Version text,
+ Seperator text,
+ PRIMARY KEY (ID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_licenses`
+ (
+ identifier character varying(255) NOT NULL,
+ uri text NOT NULL,
+ PRIMARY KEY (identifier),
+ UNIQUE (uri(200))
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_databases`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ cid bigint NOT NULL,
+ name character varying(255) NOT NULL,
+ internal_name character varying(255) NOT NULL,
+ exchange_name character varying(255) NOT NULL,
+ description text,
+ engine character varying(20),
+ is_public boolean NOT NULL DEFAULT TRUE,
+ created_by character varying(36),
+ owned_by character varying(36),
+ contact_person character varying(36),
+ created timestamp NOT NULL DEFAULT NOW(),
+ last_modified timestamp,
+ PRIMARY KEY (id),
+ FOREIGN KEY (cid) REFERENCES mdb_containers (id) /* currently we only support one-to-one */,
+ FOREIGN KEY (created_by) REFERENCES mdb_users (id),
+ FOREIGN KEY (owned_by) REFERENCES mdb_users (id),
+ FOREIGN KEY (contact_person) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_databases_subjects`
+ (
+ dbid BIGINT NOT NULL,
+ subjects character varying(255) NOT NULL,
+ PRIMARY KEY (dbid, subjects)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_tables`
+ (
+ ID bigint NOT NULL AUTO_INCREMENT,
+ tDBID bigint NOT NULL,
+ internal_name character varying(255) NOT NULL,
+ queue_name character varying(255) NOT NULL,
+ routing_key character varying(255) NOT NULL,
+ tName VARCHAR(50),
+ tDescription TEXT,
+ NumCols INTEGER,
+ NumRows INTEGER,
+ `separator` CHAR(1),
+ quote CHAR(1),
+ element_null VARCHAR(50),
+ skip_lines BIGINT,
+ element_true VARCHAR(50),
+ element_false VARCHAR(50),
+ Version TEXT,
+ created timestamp NOT NULL DEFAULT NOW(),
+ versioned boolean not null default true,
+ created_by character varying(36) NOT NULL,
+ owned_by character varying(36) NOT NULL,
+ last_modified timestamp,
+ PRIMARY KEY (ID),
+ FOREIGN KEY (tDBID) REFERENCES mdb_databases (id),
+ FOREIGN KEY (created_by) REFERENCES mdb_users (id),
+ FOREIGN KEY (owned_by) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns`
+ (
+ ID bigint NOT NULL AUTO_INCREMENT,
+ tID bigint NOT NULL,
+ dfID bigint,
+ cName VARCHAR(100),
+ internal_name VARCHAR(100) NOT NULL,
+ alias VARCHAR(100),
+ Datatype ENUM ('CHAR','VARCHAR','BINARY','VARBINARY','TINYBLOB','TINYTEXT','TEXT','BLOB','MEDIUMTEXT','MEDIUMBLOB','LONGTEXT','LONGBLOB','ENUM','SET','BIT','TINYINT','BOOL','SMALLINT','MEDIUMINT','INT','BIGINT','FLOAT','DOUBLE','DECIMAL','DATE','DATETIME','TIMESTAMP','TIME','YEAR'),
+ length INT NULL,
+ ordinal_position INTEGER NOT NULL,
+ is_primary_key BOOLEAN NOT NULL,
+ index_length INT NULL,
+ size INT,
+ d INT,
+ auto_generated BOOLEAN DEFAULT false,
+ is_null_allowed BOOLEAN NOT NULL DEFAULT true,
+ created timestamp NOT NULL DEFAULT NOW(),
+ last_modified timestamp,
+ FOREIGN KEY (tID) REFERENCES mdb_tables (ID),
+ PRIMARY KEY (ID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_enums`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ column_id bigint NOT NULL,
+ value CHARACTER VARYING(255) NOT NULL,
+ FOREIGN KEY (column_id) REFERENCES mdb_columns (ID),
+ PRIMARY KEY (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_sets`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ column_id bigint NOT NULL,
+ value CHARACTER VARYING(255) NOT NULL,
+ FOREIGN KEY (column_id) REFERENCES mdb_columns (ID),
+ PRIMARY KEY (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_nom`
+ (
+ tID bigint,
+ cID bigint,
+ maxlength INTEGER,
+ last_modified timestamp,
+ created timestamp NOT NULL DEFAULT NOW(),
+ FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
+ PRIMARY KEY (tID, cID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_num`
+ (
+ tID bigint,
+ cID bigint,
+ SIunit TEXT,
+ MaxVal NUMERIC,
+ MinVal NUMERIC,
+ Mean NUMERIC,
+ Median NUMERIC,
+ Sd Numeric,
+ -- Histogram INTEGER[],
+ last_modified timestamp,
+ created timestamp NOT NULL DEFAULT NOW(),
+ FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
+ PRIMARY KEY (tID, cID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_cat`
+ (
+ tID bigint,
+ cID bigint,
+ num_cat INTEGER,
+ -- cat_array TEXT[],
+ last_modified timestamp,
+ created timestamp NOT NULL DEFAULT NOW(),
+ FOREIGN KEY (tID, cID) REFERENCES mdb_columns (tID, ID),
+ PRIMARY KEY (tID, cID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_constraints_foreign_key`
+ (
+ fkid BIGINT NOT NULL AUTO_INCREMENT,
+ tid BIGINT NOT NULL,
+ rtid BIGINT NOT NULL,
+ on_update VARCHAR(50) NULL,
+ on_delete VARCHAR(50) NULL,
+ position INT NULL,
+ PRIMARY KEY (fkid),
+ FOREIGN KEY (tid) REFERENCES mdb_tables (id),
+ FOREIGN KEY (rtid) REFERENCES mdb_tables (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_constraints_foreign_key_reference`
+ (
+ id BIGINT NOT NULL AUTO_INCREMENT,
+ fkid BIGINT NOT NULL,
+ cid BIGINT NOT NULL,
+ rcid BIGINT NOT NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (fkid) REFERENCES mdb_constraints_foreign_key (fkid) ON UPDATE CASCADE,
+ FOREIGN KEY (cid) REFERENCES mdb_columns (id),
+ FOREIGN KEY (rcid) REFERENCES mdb_columns (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_constraints_unique`
+ (
+ uid BIGINT NOT NULL AUTO_INCREMENT,
+ tid BIGINT NOT NULL,
+ position INT NULL,
+ PRIMARY KEY (uid),
+ FOREIGN KEY (tid) REFERENCES mdb_tables (id)
+ );
+
+ CREATE TABLE IF NOT EXISTS `mdb_constraints_unique_columns`
+ (
+ id BIGINT NOT NULL AUTO_INCREMENT,
+ uid BIGINT NOT NULL,
+ cid BIGINT NOT NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (uid) REFERENCES mdb_constraints_unique (uid),
+ FOREIGN KEY (cid) REFERENCES mdb_columns (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_constraints_checks`
+ (
+ id BIGINT NOT NULL AUTO_INCREMENT,
+ tid BIGINT NOT NULL,
+ checks VARCHAR(255) NOT NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (tid) REFERENCES mdb_tables (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_concepts`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ uri text not null,
+ name VARCHAR(255) null,
+ description TEXT null,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (id),
+ UNIQUE (uri(200))
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_units`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ uri text not null,
+ name VARCHAR(255) null,
+ description TEXT null,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (id),
+ UNIQUE (uri(200))
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_concepts`
+ (
+ id bigint NOT NULL,
+ cID bigint NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (id),
+ FOREIGN KEY (cID) REFERENCES mdb_columns (ID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_columns_units`
+ (
+ id bigint NOT NULL,
+ cID bigint NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (id),
+ FOREIGN KEY (cID) REFERENCES mdb_columns (ID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_view`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ vdbid bigint NOT NULL,
+ vName VARCHAR(255) NOT NULL,
+ internal_name VARCHAR(255) NOT NULL,
+ Query TEXT NOT NULL,
+ query_hash VARCHAR(255) NOT NULL,
+ Public BOOLEAN NOT NULL,
+ NumCols INTEGER,
+ NumRows INTEGER,
+ InitialView BOOLEAN NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ last_modified timestamp,
+ created_by character varying(36) NOT NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (vdbid) REFERENCES mdb_databases (id),
+ FOREIGN KEY (created_by) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_banner_messages`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ type ENUM ('ERROR', 'WARNING', 'INFO') NOT NULL default 'INFO',
+ message TEXT NOT NULL,
+ link TEXT NULL,
+ link_text VARCHAR(255) NULL,
+ display_start timestamp NULL,
+ display_end timestamp NULL,
+ PRIMARY KEY (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_ontologies`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ prefix VARCHAR(8) NOT NULL,
+ uri TEXT NOT NULL,
+ uri_pattern TEXT,
+ sparql_endpoint TEXT NULL,
+ last_modified timestamp,
+ created timestamp NOT NULL DEFAULT NOW(),
+ UNIQUE (prefix),
+ UNIQUE (uri(200)),
+ PRIMARY KEY (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_view_columns`
+ (
+ id BIGINT NOT NULL AUTO_INCREMENT,
+ cid BIGINT NOT NULL,
+ vid BIGINT NOT NULL,
+ position INTEGER NULL,
+ PRIMARY KEY (id),
+ FOREIGN KEY (vid) REFERENCES mdb_view (id),
+ FOREIGN KEY (cid) REFERENCES mdb_columns (ID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifiers`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ dbid bigint,
+ qid bigint,
+ vid bigint,
+ publisher VARCHAR(255) NOT NULL,
+ language VARCHAR(2),
+ visibility ENUM ('SELF', 'EVERYONE') NOT NULL default 'EVERYONE',
+ publication_year INTEGER NOT NULL,
+ publication_month INTEGER,
+ publication_day INTEGER,
+ identifier_type ENUM ('DATABASE', 'SUBSET', 'VIEW') NOT NULL,
+ query TEXT,
+ query_normalized TEXT,
+ query_hash VARCHAR(255),
+ execution timestamp,
+ result_hash VARCHAR(255),
+ result_number bigint,
+ doi VARCHAR(255),
+ created timestamp NOT NULL DEFAULT NOW(),
+ created_by character varying(36) NOT NULL,
+ last_modified timestamp,
+ PRIMARY KEY (id), /* must be a single id from persistent identifier concept */
+ FOREIGN KEY (dbid) REFERENCES mdb_databases (id),
+ UNIQUE (dbid, qid),
+ FOREIGN KEY (created_by) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifier_licenses`
+ (
+ pid bigint NOT NULL,
+ license_id VARCHAR(255) NOT NULL,
+ PRIMARY KEY (pid, license_id),
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id),
+ FOREIGN KEY (license_id) REFERENCES mdb_licenses (identifier)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifier_titles`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ pid bigint NOT NULL,
+ title text NOT NULL,
+ title_type ENUM ('ALTERNATIVE_TITLE', 'SUBTITLE', 'TRANSLATED_TITLE', 'OTHER'),
+ language VARCHAR(2),
+ PRIMARY KEY (id),
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifier_funders`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ pid bigint NOT NULL,
+ funder_name VARCHAR(255) NOT NULL,
+ funder_identifier TEXT,
+ funder_identifier_type ENUM ('CROSSREF_FUNDER_ID', 'GRID', 'ISNI', 'ROR', 'OTHER'),
+ scheme_uri text,
+ award_number VARCHAR(255),
+ award_title text,
+ language VARCHAR(255),
+ PRIMARY KEY (id),
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifier_descriptions`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ pid bigint NOT NULL,
+ description text NOT NULL,
+ description_type ENUM ('ABSTRACT', 'METHODS', 'SERIES_INFORMATION', 'TABLE_OF_CONTENTS', 'TECHNICAL_INFO', 'OTHER'),
+ language VARCHAR(2),
+ PRIMARY KEY (id),
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_related_identifiers`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ pid bigint NOT NULL,
+ value varchar(255) NOT NULL,
+ type varchar(255),
+ relation varchar(255),
+ PRIMARY KEY (id), /* must be a single id from persistent identifier concept */
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id),
+ UNIQUE (pid, value)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_identifier_creators`
+ (
+ id bigint NOT NULL AUTO_INCREMENT,
+ pid bigint NOT NULL,
+ given_names text,
+ family_name text,
+ creator_name VARCHAR(255) NOT NULL,
+ name_type ENUM ('PERSONAL', 'ORGANIZATIONAL') default 'PERSONAL',
+ name_identifier text,
+ name_identifier_scheme ENUM ('ROR', 'GRID', 'ISNI', 'ORCID'),
+ name_identifier_scheme_uri text,
+ affiliation VARCHAR(255),
+ affiliation_identifier text,
+ affiliation_identifier_scheme ENUM ('ROR', 'GRID', 'ISNI'),
+ affiliation_identifier_scheme_uri text,
+ PRIMARY KEY (id),
+ FOREIGN KEY (pid) REFERENCES mdb_identifiers (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_feed`
+ (
+ fDBID bigint,
+ fID bigint,
+ fUserId character varying(36) not null,
+ fDataID bigint REFERENCES mdb_data (ID),
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (fDBID, fID, fUserId, fDataID),
+ FOREIGN KEY (fDBID, fID) REFERENCES mdb_tables (tDBID, ID),
+ FOREIGN KEY (fUserId) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_update`
+ (
+ uUserID character varying(255) NOT NULL,
+ uDBID bigint NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (uUserID, uDBID),
+ FOREIGN KEY (uDBID) REFERENCES mdb_databases (id)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_access`
+ (
+ aUserID character varying(255) NOT NULL,
+ aDBID bigint REFERENCES mdb_databases (id),
+ attime TIMESTAMP,
+ download BOOLEAN,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (aUserID, aDBID)
+ ) WITH SYSTEM VERSIONING;
+
+ CREATE TABLE IF NOT EXISTS `mdb_have_access`
+ (
+ user_id character varying(36) NOT NULL,
+ database_id bigint REFERENCES mdb_databases (id),
+ access_type ENUM ('READ', 'WRITE_OWN', 'WRITE_ALL') NOT NULL,
+ created timestamp NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (user_id, database_id),
+ FOREIGN KEY (user_id) REFERENCES mdb_users (id)
+ ) WITH SYSTEM VERSIONING;
+
+ COMMIT;
+ BEGIN;
+
+ INSERT INTO `mdb_licenses` (identifier, uri)
+ VALUES ('MIT', 'https://opensource.org/licenses/MIT'),
+ ('GPL-3.0-only', 'https://www.gnu.org/licenses/gpl-3.0-standalone.html'),
+ ('BSD-3-Clause', 'https://opensource.org/licenses/BSD-3-Clause'),
+ ('BSD-4-Clause', 'http://directory.fsf.org/wiki/License:BSD_4Clause'),
+ ('Apache-2.0', 'https://opensource.org/licenses/Apache-2.0'),
+ ('CC0-1.0', 'https://creativecommons.org/publicdomain/zero/1.0/legalcode'),
+ ('CC-BY-4.0', 'https://creativecommons.org/licenses/by/4.0/legalcode');
+
+ INSERT INTO `mdb_images` (name, version, default_port, dialect, driver_class, jdbc_method)
+ VALUES ('mariadb', '11.1.3', 3306, 'org.hibernate.dialect.MariaDBDialect', 'org.mariadb.jdbc.Driver', 'mariadb');
+
+ INSERT INTO `mdb_images_date` (iid, database_format, unix_format, example, has_time)
+ VALUES (1, '%Y-%c-%d %H:%i:%S.%f', 'yyyy-MM-dd HH:mm:ss.SSSSSS', '2022-01-30 13:44:25.499', true),
+ (1, '%Y-%c-%d %H:%i:%S', 'yyyy-MM-dd HH:mm:ss', '2022-01-30 13:44:25', true),
+ (1, '%Y-%c-%d', 'yyyy-MM-dd', '2022-01-30', false),
+ (1, '%H:%i:%S', 'HH:mm:ss', '13:44:25', true);
+
+ INSERT INTO `mdb_ontologies` (prefix, uri, uri_pattern, sparql_endpoint)
+ VALUES ('om', 'http://www.ontology-of-units-of-measure.org/resource/om-2/',
+ 'http://www.ontology-of-units-of-measure.org/resource/om-2/.*', null),
+ ('wd', 'http://www.wikidata.org/', 'http://www.wikidata.org/entity/.*', 'https://query.wikidata.org/sparql'),
+ ('mo', 'http://purl.org/ontology/mo/', 'http://purl.org/ontology/mo/.*', null),
+ ('dc', 'http://purl.org/dc/elements/1.1/', null, null),
+ ('xsd', 'http://www.w3.org/2001/XMLSchema#', null, null),
+ ('tl', 'http://purl.org/NET/c4dm/timeline.owl#', null, null),
+ ('foaf', 'http://xmlns.com/foaf/0.1/', null, null),
+ ('schema', 'http://schema.org/', null, null),
+ ('rdf', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', null, null),
+ ('rdfs', 'http://www.w3.org/2000/01/rdf-schema#', null, null),
+ ('owl', 'http://www.w3.org/2002/07/owl#', null, null),
+ ('prov', 'http://www.w3.org/ns/prov#', null, null),
+ ('db', 'http://dbpedia.org', 'http://dbpedia.org/ontology/.*', 'http://dbpedia.org/sparql');
+ COMMIT;
diff --git a/dbrepo-metadata-db/setup-schema_local.sql b/dbrepo-metadata-db/setup-schema_local.sql
index c7d132bc0c0da980f3776e3b7e82041b72620f24..1c144e31e35cb22bd09702ae683050454b66337d 100644
--- a/dbrepo-metadata-db/setup-schema_local.sql
+++ b/dbrepo-metadata-db/setup-schema_local.sql
@@ -2,6 +2,6 @@ BEGIN;
INSERT INTO `mdb_containers` (name, internal_name, image_id, host, port, sidecar_host, sidecar_port,
privileged_username, privileged_password)
-VALUES ('MariaDB 10.5', 'mariadb_10_5', 1, 'data-db', 3306, 'data-db-sidecar', 3305, 'root', 'dbrepo');
+VALUES ('MariaDB 11.1.3', 'mariadb_11_1_3', 1, 'data-db', 3306, 'data-db-sidecar', 3305, 'root', 'dbrepo');
COMMIT;
diff --git a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java
index f02e38456f5f284b3afb9a96034b2570339deed5..a96053ee95bd3e07c91b3d1a6217d702497d524d 100644
--- a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java
+++ b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java
@@ -157,7 +157,7 @@ public interface QueryMapper {
}
default PreparedStatement pathToRawInsertQuery(Connection connection, Table table, ImportDto data) throws QueryMalformedException {
- final StringBuilder statement = new StringBuilder("LOAD DATA LOCAL INFILE '/tmp/")
+ final StringBuilder statement = new StringBuilder("LOAD DATA INFILE '/tmp/")
.append(data.getLocation())
.append("' INTO TABLE `")
.append(table.getDatabase().getInternalName())
diff --git a/dbrepo-ui/api/index.js b/dbrepo-ui/api/index.js
index 9fb169e7b56434d32caef300446a106436677acf..3050d758e88968c1cb5c25f8c056c426a9935538 100644
--- a/dbrepo-ui/api/index.js
+++ b/dbrepo-ui/api/index.js
@@ -1,8 +1,8 @@
import axios from 'axios'
-import config from '../dbrepo.config.json'
-const protocol = config.api.useSsl ? 'https' : 'http'
-const baseUrl = `${protocol}://${config.api.endpoint}:${config.api.port}`
+const baseUrl = `${location.protocol}//${location.host}`
+
+console.debug('base url', baseUrl)
const instance = axios.create({
timeout: 10000,
diff --git a/dbrepo-ui/api/upload.service.js b/dbrepo-ui/api/upload.service.js
index c5499bbc07efcd628a0c5b6292617d8b9e53c1c4..f37cf24d1f3bb7e91c9183d5ca570ff46bafa7bb 100644
--- a/dbrepo-ui/api/upload.service.js
+++ b/dbrepo-ui/api/upload.service.js
@@ -1,19 +1,18 @@
import Vue from 'vue'
-import config from '../dbrepo.config'
const tus = require('tus-js-client')
class UploadService {
upload (file) {
return new Promise((resolve, reject) => {
- const protocol = config.api.useSsl ? 'https' : 'http'
- const baseUrl = `${protocol}://${config.api.endpoint}:${config.api.port}`
+ const endpoint = `${location.protocol}//${location.host}/api/upload/files`
+ console.debug('upload endpoint', endpoint)
if (!tus.isSupported) {
console.error('Your browser does not support uploads!')
Vue.$toast.error('Your browser does not support uploads!')
return
}
const upload = new tus.Upload(file, {
- endpoint: `${baseUrl}/api/upload/files`,
+ endpoint,
retryDelays: [0, 3000, 5000, 10000, 20000],
metadata: {
filename: file.name,
diff --git a/dbrepo-ui/dbrepo.config.json b/dbrepo-ui/dbrepo.config.json
index 007198b35d83ec004997d9771529560fa7488592..e908ff37abe93a321824fd10cef69cb0af9ae7d5 100644
--- a/dbrepo-ui/dbrepo.config.json
+++ b/dbrepo-ui/dbrepo.config.json
@@ -11,13 +11,10 @@
"path": "/favicon.ico"
},
"api": {
- "endpoint": "localhost",
- "port": 80,
"useSsl": false
},
"broker": {
"connection": {
- "host": "localhost",
"ports": [
5672
],
@@ -25,6 +22,9 @@
}
},
"storage": {
+ "endpoint": "storage-service",
+ "port": 9000,
+ "useSsl": false,
"accessKey": {
"id": "minioadmin",
"secret": "minioadmin"
diff --git a/dbrepo-ui/nuxt.config.js b/dbrepo-ui/nuxt.config.js
index 5b2c854f8716a510bf185b966a87b89bba5eb3fc..0476e44fc7d8ccdd91de8a63117ff0aabcecc755 100644
--- a/dbrepo-ui/nuxt.config.js
+++ b/dbrepo-ui/nuxt.config.js
@@ -4,9 +4,8 @@ import config from './dbrepo.config.json'
const proxy = {}
-const api = 'http://localhost'
-
if (process.env.NODE_ENV === 'development') {
+ const api = 'http://localhost'
proxy['/api'] = api
proxy['/pid'] = {
target: api + '/api',
diff --git a/dbrepo-ui/pages/database/_database_id/table/import.vue b/dbrepo-ui/pages/database/_database_id/table/import.vue
index fdcf96391eba222149d97577ebc72be0c74086f5..5501a4929b3b5718ac21b457e95466bd832f1711 100644
--- a/dbrepo-ui/pages/database/_database_id/table/import.vue
+++ b/dbrepo-ui/pages/database/_database_id/table/import.vue
@@ -272,14 +272,6 @@ export default {
token () {
return this.$store.state.token
},
- config () {
- if (this.token === null) {
- return {}
- }
- return {
- headers: { Authorization: `Bearer ${this.token}` }
- }
- },
user () {
return this.$store.state.user
},
diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml
index 58e2f0cea52b241f3283596a73317ce193eb3091..f27ba6337fd003beca1e20f50d440db588efc734 100644
--- a/docker-compose.prod.yml
+++ b/docker-compose.prod.yml
@@ -39,7 +39,7 @@ services:
restart: "no"
container_name: dbrepo-data-db
hostname: data-db
- image: docker.io/bitnami/mariadb:10.5
+ image: docker.io/bitnami/mariadb:11.1.3
volumes:
- data-db-data:/bitnami/mariadb
- "${SHARED_FILESYSTEM:-/tmp}:/tmp"
@@ -59,7 +59,7 @@ services:
restart: "no"
container_name: dbrepo-auth-db
hostname: auth-db
- image: docker.io/bitnami/mariadb:10.5
+ image: docker.io/bitnami/mariadb:11.1.3
volumes:
- auth-db-data:/bitnami/mariadb
ports:
diff --git a/docker-compose.yml b/docker-compose.yml
index c0074047d7bf7695d3d1546b857b36d70d1a170a..4074c172ab10e56b7d89d75535f807aad4260ea4 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -40,7 +40,7 @@ services:
restart: "no"
container_name: dbrepo-data-db
hostname: data-db
- image: docker.io/bitnami/mariadb:10.5
+ image: docker.io/bitnami/mariadb:11.1.3
volumes:
- data-db-data:/bitnami/mariadb
- "${SHARED_FILESYSTEM:-/tmp}:/tmp"
@@ -60,7 +60,7 @@ services:
restart: "no"
container_name: dbrepo-auth-db
hostname: auth-db
- image: docker.io/bitnami/mariadb:10.5
+ image: docker.io/bitnami/mariadb:11.1.3
volumes:
- auth-db-data:/bitnami/mariadb
ports:
@@ -281,6 +281,8 @@ services:
hostname: ui
image: dbrepo-ui:latest
build: ./dbrepo-ui
+ volumes:
+ - ./dbrepo-ui/dbrepo.config.json:/app/dbrepo.config.json
depends_on:
dbrepo-search-service:
condition: service_started
diff --git a/mkdocs.yml b/mkdocs.yml
index a632fe06c95aba0b47f1c71965abd91e655e222f..2e236a5dc435c8b8ff2455b1f32cbcb5a1831473 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -7,11 +7,8 @@ docs_dir: .docs
nav:
- Home: index.md
- Deployment:
+ - Kubernetes: deployment-helm.md
- Docker Compose: deployment-docker-compose.md
- - Kubernetes:
- - Helm Chart: deployment-helm.md
- - "Special: Minikube": deployment-kubernetes-minikube.md
- - "Special: Azure Cloud": deployment-kubernetes-azure.md
- System:
- Overview: system.md
- Services: