Skip to content
Snippets Groups Projects
Verified Commit 010fc76d authored by Martin Weise's avatar Martin Weise
Browse files

Updated docs

parent dc057674
Branches
Tags
No related merge requests found
...@@ -12,7 +12,7 @@ with Microsoft Azure as infrastructure provider. ...@@ -12,7 +12,7 @@ with Microsoft Azure as infrastructure provider.
### Hardware ### Hardware
For this small cloud, test deployment any public cloud provider would suffice, we recommend a For this small cloud, test deployment any public cloud provider would suffice, we recommend a
small [:simple-microsoftazure: Azure Kubernetes Service](https://azure.microsoft.com/en-us/products/kubernetes-service) small [:simple-microsoftazure: Kubernetes Service](https://azure.microsoft.com/en-us/products/kubernetes-service)
with Kubernetes version *1.24.10* and node sizes *Standard_B4ms* with Kubernetes version *1.24.10* and node sizes *Standard_B4ms*
- 4 vCPU cores - 4 vCPU cores
...@@ -35,15 +35,20 @@ recommend to at least deploy the Metadata Database as high-available, managed da ...@@ -35,15 +35,20 @@ recommend to at least deploy the Metadata Database as high-available, managed da
Microsoft decided to still maintain MariaDB 10.3 Microsoft decided to still maintain MariaDB 10.3
until [September 2025](https://learn.microsoft.com/en-us/azure/mariadb/concepts-supported-versions). until [September 2025](https://learn.microsoft.com/en-us/azure/mariadb/concepts-supported-versions).
### Shared Volume ### Fileshare
For the shared volume PersistentVolumeClaim `dbrepo-shared-volume-claim`, select an appropriate StorageClass that For the shared volume *PersistentVolumeClaim* `dbrepo-shared-volume-claim`, select an appropriate *StorageClass* that
supports `ReadWriteMany` access modes and modify the `premiumStorageClassName` variable accordingly. supports:
It is sufficient, to select the cost-efficient `azurefile` StorageClass for Azure: 1. Access mode `ReadWriteMany`
2. Hardlinks (TUSd creates lockfiles during upload)
```yaml title="values.yaml" You will need to use a *StorageClass* of either `managed-*` or `azureblob-*` (after enabling the
... proprietary [:simple-microsoftazure: CSI driver for BLOB storage](https://learn.microsoft.com/en-us/azure/aks/azure-blob-csi?tabs=NFS#azure-blob-storage-csi-driver-features)
premiumStorageClassName: azurefile in your Kubernetes Cluster).
...
``` We recommend to create
a [Container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers) for the
[Upload Service](/infrastructures/dbrepo/latest/system-services-upload/) to deposit files and mount the BLOB storage
via CSI drivers into the *Deployment*. It greatly increases the available interfaces (see below) for file uploads and
provides a highly-available filesystem for the many deployments that need to use the files.
...@@ -10,7 +10,7 @@ suitable for test-deployments. ...@@ -10,7 +10,7 @@ suitable for test-deployments.
## Requirements ## Requirements
### Hardware ### Virtual Machine
For this small, local, test deployment any modern hardware would suffice, we recommend a dedicated virtual machine with For this small, local, test deployment any modern hardware would suffice, we recommend a dedicated virtual machine with
the following settings. Note that most of the vCPU and RAM resources will be needed for starting the infrastructure, the following settings. Note that most of the vCPU and RAM resources will be needed for starting the infrastructure,
...@@ -20,7 +20,7 @@ this is because of Docker. During idle times, the deployment will use significan ...@@ -20,7 +20,7 @@ this is because of Docker. During idle times, the deployment will use significan
- 16GB RAM memory - 16GB RAM memory
- 100GB SSD storage - 100GB SSD storage
### Software ### Minikube
First, install the minikube virtualization tool that provides a single-node Kubernetes environment, e.g. on a virtual First, install the minikube virtualization tool that provides a single-node Kubernetes environment, e.g. on a virtual
machine. We do not regularly check these instructions, they are provided on best-effort. Check machine. We do not regularly check these instructions, they are provided on best-effort. Check
...@@ -41,6 +41,8 @@ minikube kubectl -- get po -A ...@@ -41,6 +41,8 @@ minikube kubectl -- get po -A
minikube addons enable ingress minikube addons enable ingress
``` ```
### NGINX
Deploy a NGINX reverse proxy on the virtual machine to reach your minikube cluster from the public Internet: Deploy a NGINX reverse proxy on the virtual machine to reach your minikube cluster from the public Internet:
```nginx title="/etc/nginx/conf.d/dbrepo.conf" ```nginx title="/etc/nginx/conf.d/dbrepo.conf"
...@@ -79,6 +81,36 @@ Replace `CLUSTER_IP` with the result of: ...@@ -79,6 +81,36 @@ Replace `CLUSTER_IP` with the result of:
Replace `DOMAIN_NAME` with the domain name. You will need also a valid TLS certificate with private key for TLS enabled Replace `DOMAIN_NAME` with the domain name. You will need also a valid TLS certificate with private key for TLS enabled
in the cluster. In our test deployment we obtained a certificate from Let's Encrypt. in the cluster. In our test deployment we obtained a certificate from Let's Encrypt.
### Fileshare
Since the Upload Service uses a shared filesystem with the [Analyst Service](/infrastructures/dbrepo/latest/system-services-analyse/),
[Metadata Service](/infrastructures/dbrepo/latest/system-services-metadata/) and
[Data Database](/infrastructures/dbrepo/latest/system-databases-data/), the dynamic provision of the *PersistentVolume*
by the *PersistentVolumeClaim*
of [`pvc.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/blob/master/charts/dbrepo-core/templates/upload-service/pvc.yaml)
needs to happen statically. You can make use of the host's filesystem and mount it in each of those deployments.
For example, mount the *hostPath* directly in
the [`deployment.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/blob/master/charts/dbrepo-core/templates/analyse-service/deployment.yaml).
```yaml title="deployment.yaml"
apiVersion: apps/v1
kind: Deployment
metadata:
name: analyse-service
...
spec:
template:
spec:
containers:
- name: analyse-service
volumeMounts:
- name: shared
hostPath: /path/of/host
mountPath: /mnt/shared
...
```
## Deployment ## Deployment
To install the DBRepo Helm Chart, download and edit To install the DBRepo Helm Chart, download and edit
......
...@@ -9,6 +9,8 @@ author: Martin Weise ...@@ -9,6 +9,8 @@ author: Martin Weise
!!! debug "Debug Information" !!! debug "Debug Information"
* Ports: 1080/tcp * Ports: 1080/tcp
* TUSd: `http://:1080/api/upload/files`
* Prometheus: `http://:1080/metrics`
## Overview ## Overview
...@@ -18,10 +20,16 @@ Upload files using one of the official the TUSd clients: ...@@ -18,10 +20,16 @@ Upload files using one of the official the TUSd clients:
* [Java](https://github.com/tus/tus-java-client) * [Java](https://github.com/tus/tus-java-client)
* [Python](https://github.com/tus/tus-py-client) * [Python](https://github.com/tus/tus-py-client)
The [TUS](https://tus.io/) protocol allows for flexible file uploads that, when interrupted, can be resumed at a later
point. It is based on the open HTTP protocol and uploading a new file is a sequence of `HEAD`, `POST` and `PATCH`
requests for large files.
For more information, see the [official Docker image](https://hub.docker.com/r/tusproject/tusd).
## Limitations ## Limitations
(none) * No support for authentication
## Security ## Security
(none) 1. Since authentication is not supported, use IP-based ingress rules to limit access to the upload endpoint.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment