Feature/s3 data db

.docs/.swagger/swagger-generate.sh

@@ -4,12 +4,11 @@ services[4000]=search
 services[5000]=analyse
 services[9093]=data
 services[9099]=metadata
-services[3305]=sidecar
 
 # requires https://github.com/mikefarah/yq/ -> v4.44.3
 
 function retrieve () {
-  if [[ "$2" == analyse ]] || [[ "$2" == search ]] || [[ "$2" == sidecar ]]; then
+  if [[ "$2" == analyse ]] || [[ "$2" == search ]]; then
     echo "... retrieve json api from localhost:$1"
     curl -sSL "http://localhost:$1/api-$2.json" | yq -p=json > "./.docs/.swagger/api-$2.yaml"
   else

.docs/api/analyse-service.md

@@ -6,12 +6,18 @@ author: Martin Weise
 
 !!! debug "Debug Information"
 
-    Image: [`registry.datalab.tuwien.ac.at/dbrepo/analyse-service:1.4.5`](https://hub.docker.com/r/dbrepo/analyse-service)
+    Image: [`registry.datalab.tuwien.ac.at/dbrepo/analyse-service:1.4.7`](https://hub.docker.com/r/dbrepo/analyse-service)
 
     * Ports: 5000/tcp
     * Prometheus: `http://<hostname>:5000/metrics`
     * Health: `http://<hostname>:5000/health`
-    * Swagger UI: `http://<hostname>:5000/swagger-ui/` <a href="./swagger/analyse" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+    * Swagger UI: `http://<hostname>:5000/swagger-ui/` <a href="../../rest/" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/analyse-service 5000:80
+    ```
 
 ## Overview
 

.docs/api/auth-service.md

@@ -11,6 +11,12 @@ author: Martin Weise
     * Ports: 8080/tcp
     * UI: `http://<hostname>:8080/`
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/auth-service 8080:80
+    ```
+
 ## Overview
 
 By default, users are created using the [User Interface](../ui) and the sign-up page in the User Interface.

.docs/api/broker-service.md

@@ -13,6 +13,12 @@ author: Martin Weise
     * Prometheus: `http://<hostname>:15692/metrics`
     * Management: `http://<hostname>:15672`
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/broker-service 15672:15672
+    ```
+
 ## Overview
 
 It holds exchanges and topics responsible for holding AMQP messages for later consumption. We

.docs/api/dashboard-service.md

@@ -8,8 +8,14 @@ author: Martin Weise
 
     Image: [`docker.io/bitnami/grafana:10.4.9-debian-12-r0`](https://hub.docker.com/r/bitnami/grafana)
 
-    * Ports: `http://<hostname>/dashboard`
-    * Prometheus: `http://<hostname>/dashboard/prometheus`
+    * Ports: `http://<hostname>:3000`
+    * Prometheus: `http://<hostname>:3000/prometheus`
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/broker-service 3000:3000
+    ```
 
 ## Overview
 

.docs/api/data-db.md

@@ -9,11 +9,11 @@ author: Martin Weise
     * Ports: 3306/tcp
     * JDBC: `jdbc://mariadb:<hostname>:3306`
 
-!!! debug "Debug Information"
-
-    Image: [`dbrepo/data-db-sidecar:1.4.5`](https://hub.docker.com/r/dbrepo/data-db-sidecar)
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
 
-    * Ports: 8080/tcp
+    ```shell
+    kubectl [-n namespace] port-forward svc/data-db 3306:3306
+    ```
 
 ## Overview
 
@@ -22,20 +22,6 @@ Any number of MariaDB ata databases can be integrated into DBRepo, even non-empt
 registered in the Metadata Database to be visible in the [User Interface](../ui) and usable from e.g. the Python 
 Library.
 
-## Architecture
-
-### Sidecar
-
-We deploy a sidecar that handles the CSV-file upload/download operations between
-the [Storage Service](../system-services-storage) and the Data Database using a Python Flask application and
-the [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) client until MariaDB supports S3
-natively.
-
-<figure markdown>
-![Sidecar architecture detailed](../images/architecture-data-db.svg)
-<figcaption>Sidecar that handles the CSV-file upload/download.</figcaption>
-</figure>
-
 ## Data
 
 The procedures require the user-generated databases to have the same collation (because of comparison operations).

.docs/api/data-service.md

@@ -6,7 +6,7 @@ author: Martin Weise
 
 !!! debug "Debug Information"
 
-    Image: [`registry.datalab.tuwien.ac.at/dbrepo/data-service:1.4.5`](https://hub.docker.com/r/dbrepo/data-service)
+    Image: [`registry.datalab.tuwien.ac.at/dbrepo/data-service:1.4.7`](https://hub.docker.com/r/dbrepo/data-service)
 
     * Ports: 9093/tcp
     * Info: `http://<hostname>:9093/actuator/info`
@@ -14,7 +14,13 @@ author: Martin Weise
         - Readiness: `http://<hostname>:9093/actuator/health/readiness`
         - Liveness: `http://<hostname>:9093/actuator/health/liveness`
     * Prometheus: `http://<hostname>:9093/actuator/prometheus`
-    * Swagger UI: `http://<hostname>:9093/swagger-ui/index.html` <a href="./swagger/data" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+    * Swagger UI: `http://<hostname>:9093/swagger-ui/index.html` <a href="../../rest/" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/data-service 9093:80
+    ```
 
 ## Overview
 

.docs/api/identity-service.md

@@ -10,6 +10,12 @@ author: Martin Weise
 
     * Ports: 1389/tcp, 1636/tcp
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/identity-service 1389:389
+    ```
+
 ## Overview
 
 This optional service holds the user identities which we simply call identities in the following. It is integrated into

.docs/api/metadata-db.md

@@ -9,6 +9,12 @@ author: Martin Weise
     * Ports: 3306/tcp
     * JDBC: `jdbc://mariadb:<hostname>:3306`
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/metadata-db 3306:3306
+    ```
+
 ## Overview
 
 The metadata database is the single, central source of truth within DBRepo and holds all metadata information for

.docs/api/metadata-service.md

@@ -6,7 +6,7 @@ author: Martin Weise
 
 !!! debug "Debug Information"
 
-    Image: [`registry.datalab.tuwien.ac.at/dbrepo/metadata-service:1.4.5`](https://hub.docker.com/r/dbrepo/metadata-service)
+    Image: [`registry.datalab.tuwien.ac.at/dbrepo/metadata-service:1.4.7`](https://hub.docker.com/r/dbrepo/metadata-service)
 
     * Ports: 9099/tcp
     * Info: `http://<hostname>:9099/actuator/info`
@@ -14,7 +14,13 @@ author: Martin Weise
         - Readiness: `http://<hostname>:9099/actuator/health/readiness`
         - Liveness: `http://<hostname>:9099/actuator/health/liveness`
     * Prometheus: `http://<hostname>:9099/actuator/prometheus`
-    * Swagger UI: `http://<hostname>:9099/swagger-ui/index.html`
+    * Swagger UI: `http://<hostname>:9099/swagger-ui/index.html` <a href="../../rest/" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/metadata-service 9099:80
+    ```
 
 ## Overview
 

.docs/api/metric-db.md

@@ -8,6 +8,14 @@ author: Martin Weise
 
     Image: [`bitnami/prometheus:2.54.1-debian-12-r4`](https://hub.docker.com/r/bitnami/prometheus)
 
+    * Ports: 8080/tcp
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine on port `8080`:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/data-db 8080:80
+    ```
+
 ## Overview
 
 The Metric Database is responsible for saving time-series data for the [Dashboard Service](../../api/dashboard-service).
@@ -140,8 +148,6 @@ The [Data Database Sidecar](#) metrics are:
 | `dbrepo_search_update_database` | Time needed to update a database in the search database |
 | `dbrepo_search_delete_database` | Time needed to delete a database in the search database |
 
-
-
 ## Limitations
 
 !!! question "Do you miss functionality? Do these limitations affect you?"

.docs/api/search-service.md

@@ -6,12 +6,18 @@ author: Martin Weise
 
 !!! debug "Debug Information"
 
-    Image: [`registry.datalab.tuwien.ac.at/dbrepo/search-service:1.4.5`](https://hub.docker.com/r/dbrepo/search-service)
+    Image: [`registry.datalab.tuwien.ac.at/dbrepo/search-service:1.4.7`](https://hub.docker.com/r/dbrepo/search-service)
 
     * Ports: 4000/tcp
     * Health: `http://<hostname>:4000/api/search/health`
     * Prometheus: `http://<hostname>:4000/metrics`
-    * Swagger UI: `http://<hostname>:4000/swagger-ui/`
+    * Swagger UI: `http://<hostname>:4000/swagger-ui/` <a href="../../rest/" target="_blank">:fontawesome-solid-square-up-right: view online</a>
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/search-service 4000:80
+    ```
 
 ## Overview
 

.docs/api/storage-service.md

@@ -8,8 +8,15 @@ author: Martin Weise
 
     Image: [`docker.io/chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs)
 
-    * Ports: 9000/tcp
+    * Ports: 8888/tcp, 9000/tcp
     * Prometheus: `http://<hostname>:9091/metrics`
+    * Filer UI: `http://<hostname>:8888`
+
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/storage-service-s3 9000:8333
+    ```
 
 ## Overview
 

.docs/api/ui.md

@@ -10,6 +10,12 @@ author: Martin Weise
 
     * Ports: 3000/tcp
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/ui 3000:80
+    ```
+
 The User Interface is configured in the `runtimeConfig` section of the `nuxt.config.ts` file during build time. For the
 runtime, you need to override those values through environment variables or by mounting a `.env` file. As a small
 example, you can configure the logo :material-numeric-1-circle-outline: in Figure 2. Make sure you mount the logo as

.docs/api/upload-service.md

@@ -12,6 +12,12 @@ author: Martin Weise
     * Prometheus: `http://<hostname>:1080/api/upload/metrics`
     * API: `http://<hostname>:1080/api/upload`
 
+    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
+
+    ```shell
+    kubectl [-n namespace] port-forward svc/upload-service 1080:80
+    ```
+
 ## Overview
 
 We use the [TUS](https://tus.io/) open protocol for resume-able file uploads which based entirely on HTTP. Even though

.docs/changelog.md

@@ -19,6 +19,7 @@ author: Martin Weise
 
 #### Changes
 
+* Remove the Data Database Sidecar and replace it with Apache Spark 4.
 * Allow anonymous users to create subsets for public databases
   in [#449](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/issues/449).
 * Show file upload progress

.docs/index.md

@@ -31,11 +31,12 @@ Installing DBRepo is very easy or
 ## Who is using DBRepo?
 
 - [TU Wien](https://dbrepo1.ec.tuwien.ac.at)
+- TU Graz
 - TU Darmstadt
 - [Universität Hamburg](https://dbrepo.fdm.uni-hamburg.de/)
 - [Universiti Teknikal Malaysia Melaka](https://dbrepo.utem.edu.my/)
 - University of the Philippines
-- [Universiti Sains Malaysia](https://dbrepo.wrfexpress.com/)
+- [Universiti Sains Malaysia](https://atmosfera.usm.my/dbrepo/)
 
 ## How can I try DBRepo?
 

.docs/installation.md

@@ -29,7 +29,6 @@ settings.
 - min. 8 vCPU cores
 - min. 8GB free RAM memory
 - min. 200GB free SSD storage
-- min. 100Mbit/s connection
 
 Since DBRepo is intended to be a publicly available repository, an optional fixed/static IP-address with optional
 SSL/TLS certificate is recommended. Follow the [secure install](#secure-install) guide.

.docs/kubernetes.md

@@ -26,6 +26,15 @@ helm upgrade --install dbrepo \
 * Kubernetes 3.8.0+
 * PV provisioner support in the underlying infrastructure
 
+### Resource Quota
+
+* `requests.cpu=8`
+* `requests.memory=8Gi`
+* `requests.storage` >= 25
+* `pods` >= 50
+* `services` >= 50
+* `secrets` >= 50
+
 ## Limitations
 
 1. The entire Helm deployment is rootless (=`runAsNonRoot=true`) except for

.gitlab-ci.yml

@@ -59,15 +59,6 @@ build-analyse-service:
     - "pip install pipenv"
     - "pipenv install gunicorn && pipenv install --dev --system --deploy"
 
-build-data-db-sidecar:
-  image: docker.io/python:3.11-alpine
-  stage: build
-  variables:
-    PIPENV_PIPFILE: "./dbrepo-data-db/sidecar/Pipfile"
-  script:
-    - "pip install pipenv"
-    - "pipenv install gunicorn && pipenv install --dev --system --deploy"
-
 build-lib:
   image: docker.io/python:3.11-alpine
   stage: build
@@ -147,7 +138,6 @@ lint-docker-compose:
     - "IGNORE_IMAGE=1 IGNORE_PORTS=1 bash .scripts/check-service.sh 'dbrepo-broker-service'"
     - "IGNORE_PORTS=1 bash .scripts/check-service.sh 'dbrepo-dashboard-service'"
     - "bash .scripts/check-service.sh 'dbrepo-data-db'"
-    - "IGNORE_IMAGE=1 IGNORE_PORTS=1 bash .scripts/check-service.sh 'dbrepo-data-db-sidecar'"
     - "IGNORE_IMAGE=1 IGNORE_PORTS=1 bash .scripts/check-service.sh 'dbrepo-data-service'"
     - "IGNORE_VOLUMES=1 bash .scripts/check-service.sh 'dbrepo-gateway-service'"
     - "IGNORE_VOLUMES=1 bash .scripts/check-service.sh 'dbrepo-identity-service'"