diff --git a/.docs/system-services-storage.md b/.docs/system-services-storage.md index 37c469898af6043f676d4e683ae65f9ffd4bc8ef..c82a78e27ff78c4e38b7f43d05abe1458492df5f 100644 --- a/.docs/system-services-storage.md +++ b/.docs/system-services-storage.md @@ -8,19 +8,21 @@ author: Martin Weise !!! debug "Debug Information" - Image: [`bitnami/minio:2023-debian-11`](https://hub.docker.com/r/bitnami/minio) + Image: [`chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs) - * Ports: 9000/tcp, 9001/tcp - * Console: `http://<hostname>/admin/storage` + * Ports: 9000/tcp + * Prometheus: `http://<hostname>:9091/metrics` ## Overview -We use [minIO](https://min.io) as a high-performance, S3 compatible object store packaged by Bitnami (VMware) for easy -cloud-ready deployments that by default support replication and monitoring. +We use [SeaweedFS](https://seaweedfs.github.io/) as a high-performance, S3 compatible object store for easy, cloud-ready +deployments that by default support replication and monitoring. No graphical user interface is provided out-of-the-box, +administrators can access the S3 storage via S3-compatible clients +e.g. [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/) (see below). ### Users -The default configuration creates one user `minioadmin` with password `minioadmin`. +The default configuration creates one user `seaweedfsadmin` with password `seaweedfsadmin`. ### Buckets @@ -29,42 +31,40 @@ The default configuration creates two buckets `dbrepo-upload`, `dbrepo-download` * `dbrepo-upload` for CSV-file upload (for import of data, analysis, etc.) from the User Interface * `dbrepo-download` for CSV-file download (exporting data, metadata, etc.) -### Metrics Collection +### Examples -By default, Prometheus metrics are not enabled as they require a running Prometheus server in the background. You can -enable the metrics endpoint by setting the following environment variables in the `docker-compose.yml` (deployment with -[Docker Compose](../deployment-docker-compose)) or `values.yml` (deployment with [Helm](../deployment-helm/)) according -to the [minIO documentation](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html). +Upload a CSV-file into the `dbrepo-upload` bucket with the AWS CLI: -### Examples +```console +$ aws --endpoint-url http://<hostname>:9000 s3 cp /path/to/file.csv s3://dbrepo-upload/ +upload: /path/to/file.csv to s3://dbrepo-upload/file.csv +``` -Upload a CSV-file into the `dbrepo-upload` bucket with the console -via `http://<hostname>/admin/storage/browser/dbrepo-upload`. +You can list the buckets: -<figure markdown> - { .img-border } - <figcaption>Uploading a file with the minIO console storage browser.</figcaption> -</figure> +```console +$ aws --endpoint-url http://<hostname>:9000 s3 ls +2023-12-03 16:23:15 dbrepo-download +2023-12-03 16:28:05 dbrepo-upload +``` -Alternatively, you can use the middleware of the [User Interface](../system-other-ui/) to upload files. +And list the files in the bucket `dbrepo-upload` with: -Download a CSV-file from the `dbrepo-download` bucket with the console -via `http://<hostname>/admin/storage/browser/dbrepo-download`. +```console +$ aws --endpoint-url http://<hostname>:9000 s3 ls dbrepo-upload +2023-12-03 16:28:05 535219 file.csv +``` -<figure markdown> - { .img-border } - <figcaption>Downloading a file with the minIO console storage browser.</figcaption> -</figure> +Alternatively, you can use the middleware of the [User Interface](../system-other-ui/) to upload files. Alternatively, you can use a S3-compatible client: -* [minIO Client](https://min.io/docs/minio/linux/reference/minio-mc.html) (most generic implementation of S3) * [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) (generic Python implementation of S3) * AWS SDK (tailored towards Amazon S3) ## Limitations -* Prometheus metrics are not enabled by default (they require a running Prometheus server). +* No support for multiple regions. !!! question "Do you miss functionality? Do these limitations affect you?" diff --git a/dbrepo-storage-service/Dockerfile b/dbrepo-storage-service/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..39eaae4603993fe24f9ff0b8d113c0d7149301bc --- /dev/null +++ b/dbrepo-storage-service/Dockerfile @@ -0,0 +1,10 @@ +FROM chrislusf/seaweedfs:3.59 as runtime + +RUN apk add curl + +WORKDIR /app + +COPY ./create-buckets.sh ./create-buckets.sh +COPY ./docker-entrypoint.sh ./docker-entrypoint.sh + +ENTRYPOINT [ "/bin/sh", "./docker-entrypoint.sh" ] \ No newline at end of file diff --git a/dbrepo-storage-service/create-buckets.sh b/dbrepo-storage-service/create-buckets.sh new file mode 100644 index 0000000000000000000000000000000000000000..bc57fdf5cc90edef167bc9b849bb48d8e3a29ebd --- /dev/null +++ b/dbrepo-storage-service/create-buckets.sh @@ -0,0 +1,18 @@ +#!/bin/bash +function log { + echo "$(date '+%Y-%m-%d %H:%M:%S') $1" +} + +log "Sleep 15s to start S3 API" +sleep 15 +log "Start polling" +until curl -sSL 127.0.0.1:9000 +do + log "S3 API not ready on port 9000, wait 5s ..." + sleep 5 +done +log "Ready" +echo "s3.bucket.create -name dbrepo-upload" | weed shell +log "Created bucket dbrepo-upload" +echo "s3.bucket.create -name dbrepo-download" | weed shell +log "Created bucket dbrepo-download" \ No newline at end of file diff --git a/dbrepo-storage-service/docker-entrypoint.sh b/dbrepo-storage-service/docker-entrypoint.sh new file mode 100644 index 0000000000000000000000000000000000000000..a1121f5443b2ba26b3e56b4f50da2b6b0e4f7b8d --- /dev/null +++ b/dbrepo-storage-service/docker-entrypoint.sh @@ -0,0 +1,3 @@ +#!/bin/sh +/bin/sh ./create-buckets.sh & +/entrypoint.sh server -dir=/data -s3 -s3.port=9000 -s3.config=/app/s3_config.json -metricsPort=9091 \ No newline at end of file diff --git a/dbrepo-storage-service/s3_config.json b/dbrepo-storage-service/s3_config.json new file mode 100644 index 0000000000000000000000000000000000000000..f270753cdc96278a039e483966ea864a16781cfe --- /dev/null +++ b/dbrepo-storage-service/s3_config.json @@ -0,0 +1,20 @@ +{ + "identities": [ + { + "name": "admin", + "credentials": [ + { + "accessKey": "seaweedfsadmin", + "secretKey": "seaweedfsadmin" + } + ], + "actions": [ + "Read", + "Write", + "List", + "Tagging", + "Admin" + ] + } + ] +} \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index 6431cb7a414b62fffdac3abe41222b1ce13cf800..711bea696e40dc3b073eea11a31c52b47bc54d66 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -183,8 +183,8 @@ services: - "5000:5000" environment: S3_STORAGE_ENDPOINT: "${STORAGE_ENDPOINT:-http://storage-service:9000}" - S3_ACCESS_KEY_ID: "${STORAGE_USERNAME:-minioadmin}" - S3_SECRET_ACCESS_KEY: "${STORAGE_PASSWORD:-minioadmin}" + S3_ACCESS_KEY_ID: "${STORAGE_USERNAME:-seaweedfsadmin}" + S3_SECRET_ACCESS_KEY: "${STORAGE_PASSWORD:-seaweedfsadmin}" volumes: - "${SHARED_FILESYSTEM:-/tmp}:/tmp" healthcheck: @@ -266,8 +266,8 @@ services: environment: FLASK_DEBUG: ${SEARCH_DEBUG_MODE:-true} S3_STORAGE_ENDPOINT: "${STORAGE_ENDPOINT:-http://storage-service:9000}" - S3_ACCESS_KEY_ID: "${STORAGE_USERNAME:-minioadmin}" - S3_SECRET_ACCESS_KEY: ${STORAGE_PASSWORD:-minioadmin} + S3_ACCESS_KEY_ID: "${STORAGE_USERNAME:-seaweedfsadmin}" + S3_SECRET_ACCESS_KEY: ${STORAGE_PASSWORD:-seaweedfsadmin} volumes: - "${SHARED_FILESYSTEM:-/tmp}:/tmp" healthcheck: @@ -352,22 +352,18 @@ services: restart: "no" container_name: dbrepo-storage-service hostname: storage-service - image: docker.io/bitnami/minio:2023-debian-11 - environment: - MINIO_ROOT_USER: "${STORAGE_USERNAME:-minioadmin}" - MINIO_ROOT_PASSWORD: "${STORAGE_PASSWORD:-minioadmin}" - MINIO_DEFAULT_BUCKETS: "${STORAGE_DBREPO_BUCKET:-dbrepo-upload:upload,dbrepo-download:download}" - MINIO_REGION_NAME: "${STORAGE_REGION_NAME:-eu-west-1}" - MINIO_BROWSER_REDIRECT_URL: "${STORAGE_BASE_URL:-http://localhost/admin/storage/}" + image: dbrepo-storage-service:latest + build: ./dbrepo-storage-service ports: - 9000:9000 + volumes: + - ./dbrepo-storage-service/s3_config.json:/app/s3_config.json + - storage-service-data:/data healthcheck: - test: [ "CMD", "mc", "ready", "local" ] - interval: 5s + test: curl -sSL 127.0.0.1:9000 || exit 1 + interval: 10s timeout: 5s - retries: 5 - volumes: - - storage-service-data:/bitnami/minio/data + retries: 12 logging: driver: json-file @@ -383,9 +379,9 @@ services: - "-s3-endpoint=${STORAGE_ENDPOINT:-http://storage-service:9000}" - "-s3-bucket=dbrepo-upload" environment: - AWS_ACCESS_KEY_ID: "${STORAGE_USERNAME:-minioadmin}" - AWS_SECRET_ACCESS_KEY: "${STORAGE_PASSWORD:-minioadmin}" - AWS_REGION: "${STORAGE_REGION_NAME:-eu-west-1}" + AWS_ACCESS_KEY_ID: "${STORAGE_USERNAME:-seaweedfsadmin}" + AWS_SECRET_ACCESS_KEY: "${STORAGE_PASSWORD:-seaweedfsadmin}" + AWS_REGION: "${STORAGE_REGION_NAME:-default}" depends_on: dbrepo-storage-service: condition: service_healthy