Merge branch 'dev' into 'master'

Dev

See merge request !249

.docs/deployment-helm.md

@@ -4,7 +4,8 @@ author: Martin Weise
 
 ## TL;DR
 
-To install DBRepo in your existing cluster, download the sample [`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/raw/master/charts/dbrepo-core/values.yaml?inline=false)
+To install DBRepo in your existing cluster, download the
+sample [`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/raw/master/charts/dbrepo-core/values.yaml?inline=false)
 for your deployment and update the variables, especially `hostname`.
 
 ```shell
@@ -30,6 +31,8 @@ about values, etc.
 
 1. MariaDB Galera does not (yet) support XA-transactions required by the authentication service (=Keycloak). Therefore
    only a single MariaDB pod can be deployed at once for the [auth database](../system-databases-authentication).
+2. The entire Helm deployment is rootless (=`runAsNonRoot=true`) except for
+   the [Storage Service](../system-services-storage/) which still requires a root user.
 
 !!! question "Do you miss functionality? Do these limitations affect you?"
 

.docs/system-other-ui.md

@@ -73,43 +73,17 @@ services:
 
 ### Architecture
 
-<figure markdown>
+The server-client architecture of the User Interface is shown in [Figure 3](#fig3), it is supposed to help debug the
+User Interface on development.
+
+<figure id="fig3" markdown>
 ![Architecture of the UI microservice](images/architecture-ui.svg)
-<figcaption>Figure 3: Architecture of the UI microservice</figcaption>
+<figcaption>Figure 3: Architecture of the User Interface</figcaption>
 </figure>
 
 ### Example
 
-Upload a file to the `dbrepo-upload` bucket in the [Storage Service](../system-services-storage/) using the Node.js
-middleware. The request must be sent with the `Content-Type: multipart/form-data` header and the file must be placed
-in the `file` field of the form. For example:
-
-```shell
-curl -X POST \
-  -F "file=@path/to/file/gps.csv" \
-  http://<hostname>/server-middleware/upload
-```
-
-The response looks like this:
-
-```json
-{
-    "fieldname": "file",
-    "originalname": "gps.csv",
-    "encoding": "7bit",
-    "mimetype": "text/csv",
-    "buffer": {
-        "type": "Buffer",
-        "data": [
-            34,
-            73,
-            ...
-        ]
-    },
-    "size": 130279,
-    "etag": "9d23e73f4ed9f7e5afc80e696db69ebb"
-}
-```
+See the [Usage Overview](../usage-overview/) page for detailed examples.
 
 ## Limitations
 

.docs/system-services-analyse.md

@@ -38,7 +38,7 @@ the [Storage Service](../system-services-storage), analysis for data types and p
 
 ### Examples
 
-See the [usage page](../usage-analyse).
+See the [usage page](../usage-analyse/) for examples.
 
 ## Limitations
 

.docs/system-services-authentication.md

@@ -16,7 +16,8 @@ author: Martin Weise
 ## Overview
 
 By default, users are created using the [User Interface](../system-other-ui) and the sign-up page in the User Interface.
-This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then managed by the
+This creates a new user in the [Authentication Database](../system-databases-authentication), the user identity is then
+managed by the
 Authentication Service.
 
 ## Groups
@@ -59,7 +60,7 @@ public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
 ### Default Container Handling
 
 | Name              | Description                   |
-|--------------------------|--------------------------------------|
+|-------------------|-------------------------------|
 | `find-container`  | Can find a specific container |
 | `list-containers` | Can list all containers       |
 
@@ -73,8 +74,9 @@ public ResponseEntity<DatabaseBriefDto> create(@NotNull Long containerId,
 | `delete-database-access`     | Can delete the access to a database of a user        |
 | `find-database`              | Can find a specific database in a container          |
 | `list-databases`             | Can list all databases in a container                |
-| `modify-database-visibility` | Can modify the database visibility (public, private) |
+| `modify-database-image`      | Can update the database image                        |
 | `modify-database-owner`      | Can modify the database owner                        |
+| `modify-database-visibility` | Can modify the database visibility (public, private) |
 | `update-database-access`     | Can update the access to a database of a user        |
 
 ### Default Table Handling

.docs/system-services-upload.md

@@ -17,11 +17,36 @@ author: Martin Weise
 
 ## Overview
 
-We use the [TUS](https://tus.io/) open protocol for resumable file uploads which based entirely on HTTP.
+We use the [TUS](https://tus.io/) open protocol for resumable file uploads which based entirely on HTTP. Even though
+the Upload Service is part of the standard installation, it is an entirely optional component and can be replaced with
+any S3-compatible Blob Storage.
+
+### Settings
+
+The Upload Service is responsible for uploading files (mainly CSV-files) into a Blob Storage that can be accesses trough
+the S3 protocol (e.g. our [Storage Service](../system-services-storage)). Make sure that the Upload Service can be
+accessed from the Gateway Service and set the url in the User Interface configuration file.
+
+```json title="dbrepo.config.json"
+{
+    "upload": {
+       "url": "example.com",
+       "useSsl": true
+    },
+    ...
+}
+```
+
+If your deployment is secured with SSL/TLS (recommended) set the `useSsl` variable to `true`.
+
+### Architecture
 
-### Examples
+The Upload Service communicates internally with the [Storage Service](../system-services-storage) (c.f. [Figure 1](#fig1)).
 
-See the [usage page](../usage-upload).
+<figure id="fig1" markdown>
+![Architecture of the Upload Service](images/architecture-upload-service.svg)
+<figcaption>Figure 1: Architecture of the Upload Service</figcaption>
+</figure>
 
 ## Limitations
 

.docs/usage-analyse.md

@@ -7,6 +7,8 @@ author: Martin Weise
 Given a [CSV-file](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-datasets/-/raw/master/gps.csv)
 containing GPS-data `gps.csv` already uploaded in the `dbrepo-upload` bucket of the Storage Service with key `gps.csv`:
 
+=== "Terminal"
+
     ```shell
     curl -X POST \
       -d '{"filename":"gps.csv","separator":","}'

.docs/usage-upload.md

@@ -4,9 +4,6 @@ author: Martin Weise
 
 # Upload Service
 
-Upload a CSV-file into the `dbrepo-upload` bucket with the console
-via `http://<hostname>/admin/storage/browser/dbrepo-upload`.
-
 We recommend using a TUS-compatible client:
 
 * [tus-py-client](https://github.com/tus/tus-py-client) (Python)
@@ -14,14 +11,33 @@ We recommend using a TUS-compatible client:
 * [tus-js-client](https://github.com/tus/tus-js-client) (JavaScript/Node.js)
 * [tusd](https://github.com/tus/tusd) (Go)
 
-You can also upload a file `file.csv` in 200 byte chunks with Python:
+Upload a file to the `dbrepo-upload` bucket in the [Storage Service](../system-services-storage/) using the Node.js
+middleware.
+
+=== "Terminal"
+    
+    ```shell
+    curl -X POST \
+      -H "Content-Type: multipart/form-data" \
+      -F "file=@path/to/file/gps.csv" \
+      http://<hostname>/server-middleware/upload
+    ```
+
+    !!! info "TUS protocol from terminal"
+
+        Alternatively, use the [`tusc.sh`](https://github.com/adhocore/tusc.sh) terminal client to directly upload to the
+        Upload Service.
+
+        ```shell
+        tusc -H <hostname>/api/upload/files -f path/to/file/gps.csv
+        ```
 
 === "Python"
 
     ```python
     from tusclient import client
 
-    my_client = client.TusClient('http://localhost/api/upload/files')
+    my_client = client.TusClient('http://<hostname>/api/upload/files')
     uploader = my_client.uploader('/path/to/file.csv', chunk_size=200)
     uploader.upload()
     ```

dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/QueryServiceIntegrationTest.java

@@ -368,14 +368,14 @@ public class QueryServiceIntegrationTest extends BaseUnitTest {
         assertEquals("boar", result.get(0).get("animal_name"));
         assertEquals("Moritz", result.get(0).get("firstname"));
         assertEquals("Staudinger", result.get(0).get("lastname"));
-        assertEquals("1990", result.get(0).get("birth"));
+        assertEquals(Short.parseShort("1990"), result.get(0).get("birth"));
         assertEquals("11:22:33", result.get(0).get("reminder"));
         assertEquals(BigInteger.valueOf(1L), result.get(1).get("id"));
         assertEquals(4, result.get(1).get("legs"));
         assertEquals("cavy", result.get(1).get("animal_name"));
         assertEquals("Moritz", result.get(1).get("firstname"));
         assertEquals("Staudinger", result.get(1).get("lastname"));
-        assertEquals("1990", result.get(1).get("birth"));
+        assertEquals(Short.parseShort("1990"), result.get(1).get("birth"));
         assertEquals("11:22:33", result.get(1).get("reminder"));
         assertEquals(BigInteger.valueOf(3L), result.get(2).get("id"));
         assertEquals(4, result.get(2).get("legs"));

dbrepo-ui/api/upload.service.js

 import Vue from 'vue'
-import config from '../dbrepo.config.json'
 const tus = require('tus-js-client')
 
 class UploadService {
-  upload (file) {
+  upload (url, file) {
     return new Promise((resolve, reject) => {
-      const endpoint = `http${config.upload.useSsl ? 's' : ''}://${config.upload.endpoint}:${config.upload.port}/api/upload/files`
+      const endpoint = `${url}/api/upload/files`
       console.debug('upload endpoint', endpoint)
       if (!tus.isSupported) {
         console.error('Your browser does not support uploads!')

dbrepo-ui/components/dialogs/EditTuple.vue

@@ -320,7 +320,7 @@ export default {
       if (!file) {
         return
       }
-      UploadService.upload(file)
+      UploadService.upload(this.$config.uploadEndpointUrl, file)
         .then((metadata) => {
           console.debug('uploaded file', metadata)
           const { s3key } = metadata

dbrepo-ui/dbrepo.config.json

@@ -35,8 +35,7 @@
     }
   },
   "upload": {
-    "endpoint": "localhost",
-    "port": 1080,
+    "url": "localhost:80",
     "useSsl": false
   },
   "database": {

dbrepo-ui/nuxt.config.js

@@ -112,7 +112,8 @@ export default {
     brokerHost: config.broker.connection.host,
     brokerPorts: config.broker.connection.ports,
     brokerExtraInfo: config.broker.connection.extraInfo,
-    databaseExtraInfo: config.database.connection.extraInfo
+    databaseExtraInfo: config.database.connection.extraInfo,
+    uploadEndpointUrl: `http${config.upload.useSsl ? 's' : ''}://${config.upload.url}`
   },
 
   serverMiddleware: [

dbrepo-ui/pages/database/_database_id/settings.vue

@@ -312,7 +312,7 @@ export default {
     },
     uploadFile () {
       this.loadingUpload = true
-      UploadService.upload(this.fileModel)
+      UploadService.upload(this.$config.uploadEndpointUrl, this.fileModel)
         .then((metadata) => {
           console.debug('uploaded image', metadata)
           this.modifyImage.key = metadata.s3key

dbrepo-ui/pages/database/_database_id/table/_table_id/import.vue

@@ -205,7 +205,7 @@ export default {
     isNonNegativeInteger,
     uploadAndImport () {
       this.loading = true
-      UploadService.upload(this.fileModel)
+      UploadService.upload(this.$config.uploadEndpointUrl, this.fileModel)
         .then((metadata) => {
           console.debug('uploaded file', metadata)
           const { s3key } = metadata

dbrepo-ui/pages/database/_database_id/table/import.vue

@@ -345,7 +345,7 @@ export default {
     upload () {
       this.loading = true
       return new Promise((resolve, reject) => {
-        UploadService.upload(this.fileModel)
+        UploadService.upload(this.$config.uploadEndpointUrl, this.fileModel)
           .then((metadata) => {
             console.debug('uploaded file', metadata)
             this.loading = false

helm-charts/dbrepo/templates/analyse-service/deployment.yaml

@@ -23,9 +23,10 @@ spec:
         service: analyse-service
     spec:
       securityContext:
-        fsGroup: 0
-        runAsUser: 0
-        runAsGroup: 0
+        runAsNonRoot: true
+        fsGroup: 1000
+        runAsUser: 1000
+        runAsGroup: 1000
       containers:
         - name: analyse-service
           image: {{ printf "%s/%s:%s" .Values.analyseService.image.registry .Values.analyseService.image.repository .Values.analyseService.image.tag }}