Merge branch '398-replace-minio-with-seaweedfs' into 'dev'

Replacing minIO with SeaweedFS

See merge request !218

.docs/system-services-storage.md

@@ -8,19 +8,21 @@ author: Martin Weise
 
 !!! debug "Debug Information"
 
-    Image: [`bitnami/minio:2023-debian-11`](https://hub.docker.com/r/bitnami/minio)
+    Image: [`chrislusf/seaweedfs:3.59`](https://hub.docker.com/r/chrislusf/seaweedfs)
 
-    * Ports: 9000/tcp, 9001/tcp
-    * Console: `http://<hostname>/admin/storage`
+    * Ports: 9000/tcp
+    * Prometheus: `http://<hostname>:9091/metrics`
 
 ## Overview
 
-We use [minIO](https://min.io) as a high-performance, S3 compatible object store packaged by Bitnami (VMware) for easy
-cloud-ready deployments that by default support replication and monitoring.
+We use [SeaweedFS](https://seaweedfs.github.io/) as a high-performance, S3 compatible object store for easy, cloud-ready
+deployments that by default support replication and monitoring. No graphical user interface is provided out-of-the-box,
+administrators can access the S3 storage via S3-compatible clients 
+e.g. [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/) (see below).
 
 ### Users
 
-The default configuration creates one user `minioadmin` with password `minioadmin`.
+The default configuration creates one user `seaweedfsadmin` with password `seaweedfsadmin`.
 
 ### Buckets
 
@@ -29,42 +31,48 @@ The default configuration creates two buckets `dbrepo-upload`, `dbrepo-download`
 * `dbrepo-upload` for CSV-file upload (for import of data, analysis, etc.) from the User Interface
 * `dbrepo-download` for CSV-file download (exporting data, metadata, etc.)
 
-### Metrics Collection
+### Examples
 
-By default, Prometheus metrics are not enabled as they require a running Prometheus server in the background. You can
-enable the metrics endpoint by setting the following environment variables in the `docker-compose.yml` (deployment with 
-[Docker Compose](../deployment-docker-compose)) or `values.yml` (deployment with [Helm](../deployment-helm/)) according 
-to the [minIO documentation](https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html).
+Upload a CSV-file into the `dbrepo-upload` bucket with the AWS CLI:
 
-### Examples
+```console
+$ aws --endpoint-url http://<hostname>:9000 \
+    s3 \
+    cp /path/to/file.csv \
+    s3://dbrepo-upload/
+upload: /path/to/file.csv to s3://dbrepo-upload/file.csv
+```
 
-Upload a CSV-file into the `dbrepo-upload` bucket with the console 
-via `http://<hostname>/admin/storage/browser/dbrepo-upload`.
+You can list the buckets:
 
-<figure markdown>
-   ![Data ingest](images/minio-upload.png){ .img-border }
-   <figcaption>Uploading a file with the minIO console storage browser.</figcaption>
-</figure>
+```console
+$ aws --endpoint-url http://<hostname>:9000 \
+    s3 \
+    ls
+2023-12-03 16:23:15 dbrepo-download
+2023-12-03 16:28:05 dbrepo-upload
+```
 
-Alternatively, you can use the middleware of the [User Interface](../system-other-ui/) to upload files.
+And list the files in the bucket `dbrepo-upload` with:
 
-Download a CSV-file from the `dbrepo-download` bucket with the console
-via `http://<hostname>/admin/storage/browser/dbrepo-download`.
+```console
+$ aws --endpoint-url http://<hostname>:9000 \
+    s3 \
+    ls \
+    dbrepo-upload
+2023-12-03 16:28:05     535219 file.csv
+```
 
-<figure markdown>
-   ![Data ingest](images/minio-download.png){ .img-border }
-   <figcaption>Downloading a file with the minIO console storage browser.</figcaption>
-</figure>
+Alternatively, you can use the middleware of the [User Interface](../system-other-ui/) to upload files.
 
 Alternatively, you can use a S3-compatible client:
 
-* [minIO Client](https://min.io/docs/minio/linux/reference/minio-mc.html) (most generic implementation of S3)
 * [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) (generic Python implementation of S3)
 * AWS SDK (tailored towards Amazon S3)
 
 ## Limitations
 
-* Prometheus metrics are not enabled by default (they require a running Prometheus server).
+* No support for multiple regions.
 
 !!! question "Do you miss functionality? Do these limitations affect you?"
 

dbrepo-analyse-service/Dockerfile

@@ -15,8 +15,8 @@ ENV FLASK_ENV=production
 ENV HOSTNAME=analyse-service
 ENV LOG_LEVEL=INFO
 ENV S3_STORAGE_ENDPOINT="http://storage-service:9000"
-ENV S3_ACCESS_KEY_ID="minioadmin"
-ENV S3_SECRET_ACCESS_KEY="minioadmin"
+ENV S3_ACCESS_KEY_ID="seaweedfsadmin"
+ENV S3_SECRET_ACCESS_KEY="seaweedfsadmin"
 
 COPY ./as-yml ./as-yml
 COPY ./clients ./clients

dbrepo-analyse-service/clients/minio_client.py → dbrepo-analyse-service/clients/s3_client.py

@@ -5,12 +5,12 @@ import logging
 from botocore.exceptions import ClientError
 
 
-class MinioClient:
+class S3Client:
 
     def __init__(self):
         endpoint_url = os.getenv('S3_STORAGE_ENDPOINT', 'http://localhost:9000')
-        aws_access_key_id = os.getenv('S3_ACCESS_KEY_ID', 'minioadmin')
-        aws_secret_access_key = os.getenv('S3_SECRET_ACCESS_KEY', 'minioadmin')
+        aws_access_key_id = os.getenv('S3_ACCESS_KEY_ID', 'seaweedfsadmin')
+        aws_secret_access_key = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
         logging.info("retrieve file from S3, endpoint_url=%s, aws_access_key_id=%s, aws_secret_access_key=(hidden)",
                      endpoint_url, aws_access_key_id)
         self.client = boto3.client(service_name='s3', endpoint_url=endpoint_url, aws_access_key_id=aws_access_key_id,

dbrepo-analyse-service/determine_dt.py

@@ -6,7 +6,7 @@ import json
 import csv
 import logging
 import io
-from clients.minio_client import MinioClient
+from clients.s3_client import S3Client
 
 import messytables, pandas as pd
 from messytables import CSVTableSet, type_guess, \
@@ -17,9 +17,9 @@ def determine_datatypes(filename, enum=False, enum_tol=0.0001, separator=None) -
     # Use option enum=True for searching Postgres ENUM Types in CSV file. Remark
     # Enum is not SQL standard, hence, it might not be supported by all db-engines.
     # However, it can be used in Postgres and MySQL.
-    minio_client = MinioClient()
-    minio_client.file_exists('dbrepo-upload', filename)
-    response = minio_client.get_file('dbrepo-upload', filename)
+    s3_client = S3Client()
+    s3_client.file_exists('dbrepo-upload', filename)
+    response = s3_client.get_file('dbrepo-upload', filename)
     stream = response['Body']
     if response['ContentLength'] == 0:
         logging.warning(f'Failed to determine data types: file {filename} has empty body')

dbrepo-analyse-service/determine_pk.py

@@ -5,7 +5,7 @@ import random
 import numpy as np
 import math
 from determine_dt import determine_datatypes
-from clients.minio_client import MinioClient
+from clients.s3_client import S3Client
 
 
 def determine_pk(filename, separator=','):
@@ -15,9 +15,9 @@ def determine_pk(filename, separator=','):
     colnames = dt.keys()
     colindex = list(range(0, len(colnames)))
 
-    minio_client = MinioClient()
-    minio_client.file_exists('dbrepo-upload', filename)
-    response = minio_client.get_file('dbrepo-upload', filename)
+    s3_client = S3Client()
+    s3_client.file_exists('dbrepo-upload', filename)
+    response = s3_client.get_file('dbrepo-upload', filename)
     stream = response['Body']
     if response['ContentLength'] == 0:
         logging.warning(f'Failed to determine primary key: file {filename} has empty body')

dbrepo-analyse-service/test.sh

 #!/bin/bash
 source ./dbrepo-analyse-service/venv/bin/activate
-cd ./dbrepo-analyse-service/ && coverage run -m pytest test/test_determine_dt.py test/test_determine_pk.py test/test_minio_client.py --junitxml=report.xml && coverage html --omit="test/*" && coverage report --omit="test/*" > ./coverage.txt
\ No newline at end of file
+cd ./dbrepo-analyse-service/ && coverage run -m pytest test/test_determine_dt.py test/test_determine_pk.py test/test_s3_client.py --junitxml=report.xml && coverage html --omit="test/*" && coverage report --omit="test/*" > ./coverage.txt
\ No newline at end of file

dbrepo-analyse-service/test/conftest.py

@@ -6,8 +6,6 @@ import logging
 from minio.deleteobjects import DeleteObject
 from testcontainers.minio import MinioContainer
 
-from clients.minio_client import MinioClient
-
 
 @pytest.fixture(scope="session")
 def session(request):
@@ -16,9 +14,9 @@ def session(request):
     :param request: /
     :return: The minIO container
     """
-    logging.debug("[fixture] creating minIO container")
-    container = MinioContainer(access_key="minioadmin", secret_key="minioadmin")
-    logging.debug("[fixture] starting minIO container")
+    logging.debug("[fixture] creating container")
+    container = MinioContainer(access_key="seaweedfsadmin", secret_key="seaweedfsadmin")
+    logging.debug("[fixture] starting container")
     container.start()
     # set the environment for the client
     endpoint = 'http://' + container.get_container_host_ip() + ':' + container.get_exposed_port(9000)

dbrepo-analyse-service/test/s3_config.json

+{
+  "identities": [
+    {
+      "name": "admin",
+      "credentials": [
+        {
+          "accessKey": "seaweedfsadmin",
+          "secretKey": "seaweedfsadmin"
+        }
+      ],
+      "actions": [
+        "Read",
+        "Write",
+        "List",
+        "Tagging",
+        "Admin"
+      ]
+    }
+  ]
+}
\ No newline at end of file

dbrepo-analyse-service/test/test_determine_dt.py

@@ -9,7 +9,7 @@ import unittest
 import json
 
 
-from clients.minio_client import MinioClient
+from clients.s3_client import S3Client
 from botocore.exceptions import ClientError
 from determine_dt import determine_datatypes
 
@@ -32,7 +32,7 @@ class DetermineDatatypesTest(unittest.TestCase):
         }
 
         # mock
-        MinioClient().upload_file("datetime.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("datetime.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="datetime.csv", separator=",")
@@ -54,7 +54,7 @@ class DetermineDatatypesTest(unittest.TestCase):
         }
 
         # mock
-        MinioClient().upload_file("datetime_tz.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("datetime_tz.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="datetime_tz.csv", separator=",")
@@ -76,7 +76,7 @@ class DetermineDatatypesTest(unittest.TestCase):
         }
 
         # mock
-        MinioClient().upload_file("datetime_t.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("datetime_t.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="datetime_t.csv", separator=",")
@@ -98,7 +98,7 @@ class DetermineDatatypesTest(unittest.TestCase):
         }
 
         # mock
-        MinioClient().upload_file("datatypes.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("datatypes.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="datatypes.csv", separator=",")
@@ -121,7 +121,7 @@ class DetermineDatatypesTest(unittest.TestCase):
     def test_determine_datatypes_fileEmpty_succeeds(self):
 
         # mock
-        MinioClient().upload_file("empty.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("empty.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes("empty.csv")
@@ -133,7 +133,7 @@ class DetermineDatatypesTest(unittest.TestCase):
     def test_determine_datatypes_separatorSemicolon_succeeds(self):
 
         # mock
-        MinioClient().upload_file("separator.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("separator.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="separator.csv", separator=";")
@@ -144,7 +144,7 @@ class DetermineDatatypesTest(unittest.TestCase):
     def test_determine_datatypes_separatorGuess_succeeds(self):
 
         # mock
-        MinioClient().upload_file("separator.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("separator.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="separator.csv")
@@ -155,7 +155,7 @@ class DetermineDatatypesTest(unittest.TestCase):
     def test_determine_datatypes_separatorGuessLargeDataset_succeeds(self):
 
         # mock
-        MinioClient().upload_file("large.csv", './data/test_dt/', 'dbrepo-upload')
+        S3Client().upload_file("large.csv", './data/test_dt/', 'dbrepo-upload')
 
         # test
         response = determine_datatypes(filename="large.csv")

dbrepo-analyse-service/test/test_determine_pk.py

@@ -9,127 +9,99 @@ import unittest
 import os
 import json
 
-from clients.minio_client import MinioClient
-from testcontainers.minio import MinioContainer
+from clients.s3_client import S3Client
 
 from determine_pk import determine_pk
 
-
-def before():
-    container = MinioContainer(access_key="minioadmin", secret_key="minioadmin").start()
-    endpoint = 'http://' + container.get_container_host_ip() + ':' + container.get_exposed_port(9000)
-    os.environ['S3_STORAGE_ENDPOINT'] = endpoint
-    client = container.get_client()
-    # create buckets
-    client.make_bucket('dbrepo-upload')
-    client.make_bucket('dbrepo-download')
-    return container
-
 class DeterminePrimaryKeyTest(unittest.TestCase):
 
     # @Test
     def test_determine_pk_largeFileIdFirst_succeeds(self):
 
-        with before() as minio:
-
-            # mock
-            MinioClient().upload_file("largefile_idfirst.csv", './data/test_pk/', 'dbrepo-upload')
+        # mock
+        S3Client().upload_file("largefile_idfirst.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # test
-            response = determine_pk('largefile_idfirst.csv')
-            data = json.loads(response)
-            self.assertEqual(1, int(data['id']))
+        # test
+        response = determine_pk('largefile_idfirst.csv')
+        data = json.loads(response)
+        self.assertEqual(1, int(data['id']))
 
     # @Test
     def test_determine_pk_largeFileIdInBetween_succeeds(self):
 
-        with before() as minio:
+        # mock
+        S3Client().upload_file("largefile_idinbtw.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # mock
-            MinioClient().upload_file("largefile_idinbtw.csv", './data/test_pk/', 'dbrepo-upload')
-
-            # test
-            response = determine_pk('largefile_idinbtw.csv')
-            data = json.loads(response)
-            self.assertEqual(1, int(data['id']))
+        # test
+        response = determine_pk('largefile_idinbtw.csv')
+        data = json.loads(response)
+        self.assertEqual(1, int(data['id']))
 
     # @Test
     def test_determine_pk_largeFileNoPrimaryKey_fails(self):
 
-        with before() as minio:
-
-            # mock
-            MinioClient().upload_file("largefile_no_pk.csv", './data/test_pk/', 'dbrepo-upload')
+        # mock
+        S3Client().upload_file("largefile_no_pk.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # test
-            response = determine_pk('largefile_no_pk.csv')
-            data = json.loads(response)
-            self.assertEqual({}, data)
+        # test
+        response = determine_pk('largefile_no_pk.csv')
+        data = json.loads(response)
+        self.assertEqual({}, data)
 
     # @Test
     def test_determine_pk_largeFileNullInUnique_fails(self):
 
-        with before() as minio:
+        # mock
+        S3Client().upload_file("largefile_nullinunique.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # mock
-            MinioClient().upload_file("largefile_nullinunique.csv", './data/test_pk/', 'dbrepo-upload')
-
-            # test
-            response = determine_pk('largefile_nullinunique.csv')
-            data = json.loads(response)
-            self.assertFalse('uniquestr' in data)
+        # test
+        response = determine_pk('largefile_nullinunique.csv')
+        data = json.loads(response)
+        self.assertFalse('uniquestr' in data)
 
     # @Test
     def test_determine_pk_smallFileIdFirst_fails(self):
 
-        with before() as minio:
-
-            # mock
-            MinioClient().upload_file("smallfile_idfirst.csv", './data/test_pk/', 'dbrepo-upload')
+        # mock
+        S3Client().upload_file("smallfile_idfirst.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # test
-            response = determine_pk('smallfile_idfirst.csv')
-            data = json.loads(response)
-            self.assertEqual(1, int(data['id']))
+        # test
+        response = determine_pk('smallfile_idfirst.csv')
+        data = json.loads(response)
+        self.assertEqual(1, int(data['id']))
 
     # @Test
     def test_determine_pk_smallFileIdIntBetween_fails(self):
 
-        with before() as minio:
-
-            # mock
-            MinioClient().upload_file("smallfile_idinbtw.csv", './data/test_pk/', 'dbrepo-upload')
+        # mock
+        S3Client().upload_file("smallfile_idinbtw.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # test
-            response = determine_pk('smallfile_idinbtw.csv')
-            data = json.loads(response)
-            self.assertEqual(1, int(data['id']))
+        # test
+        response = determine_pk('smallfile_idinbtw.csv')
+        data = json.loads(response)
+        self.assertEqual(1, int(data['id']))
 
     # @Test
     def test_determine_pk_smallFileNoPrimaryKey_fails(self):
 
-        with before() as minio:
+        # mock
+        S3Client().upload_file("smallfile_no_pk.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # mock
-            MinioClient().upload_file("smallfile_no_pk.csv", './data/test_pk/', 'dbrepo-upload')
-
-            # test
-            response = determine_pk('smallfile_no_pk.csv')
-            data = json.loads(response)
-            self.assertEqual({}, data)
+        # test
+        response = determine_pk('smallfile_no_pk.csv')
+        data = json.loads(response)
+        self.assertEqual({}, data)
 
     # @Test
     def test_determine_pk_smallFileNullInUnique_fails(self):
 
-        with before() as minio:
-
-            # mock
-            MinioClient().upload_file("smallfile_nullinunique.csv", './data/test_pk/', 'dbrepo-upload')
+        # mock
+        S3Client().upload_file("smallfile_nullinunique.csv", './data/test_pk/', 'dbrepo-upload')
 
-            # test
-            response = determine_pk('smallfile_nullinunique.csv')
-            data = json.loads(response)
-            self.assertFalse('uniquestr' in data)
+        # test
+        response = determine_pk('smallfile_nullinunique.csv')
+        data = json.loads(response)
+        self.assertFalse('uniquestr' in data)
 
 
 if __name__ == '__main__':

dbrepo-analyse-service/test/test_minio_client.py → dbrepo-analyse-service/test/test_s3_client.py

@@ -9,16 +9,16 @@ import unittest
 
 from botocore.exceptions import ClientError
 
-from clients.minio_client import MinioClient
+from clients.s3_client import S3Client
 
 
-class MinioClientTest(unittest.TestCase):
+class S3ClientTest(unittest.TestCase):
 
     # @Test
     def test_upload_file_succeeds(self):
 
         # test
-        response = MinioClient().upload_file(filename="testdt01.csv", path="./data/")
+        response = S3Client().upload_file(filename="testdt01.csv", path="./data/")
         self.assertTrue(response)
 
     # @Test
@@ -26,7 +26,7 @@ class MinioClientTest(unittest.TestCase):
 
         # test
         try:
-            MinioClient().upload_file(filename="testdt06.csv", path="./data/")
+            S3Client().upload_file(filename="testdt06.csv", path="./data/")
         except FileNotFoundError:
             pass
         except Exception:
@@ -38,10 +38,10 @@ class MinioClientTest(unittest.TestCase):
     def test_download_file_succeeds(self):
 
         # mock
-        MinioClient().upload_file(filename="testdt01.csv", path="./data/", bucket="dbrepo-upload")
+        S3Client().upload_file(filename="testdt01.csv", path="./data/", bucket="dbrepo-upload")
 
         # test
-        response = MinioClient().download_file(filename="testdt01.csv")
+        response = S3Client().download_file(filename="testdt01.csv")
         self.assertTrue(response)
 
     # @Test
@@ -49,7 +49,7 @@ class MinioClientTest(unittest.TestCase):
 
         # test
         try:
-            MinioClient().download_file(filename="testdt01.csv")
+            S3Client().download_file(filename="testdt01.csv")
         except ClientError:
             pass
         except Exception:
@@ -61,10 +61,10 @@ class MinioClientTest(unittest.TestCase):
     def test_get_file_succeeds(self):
 
         # mock
-        MinioClient().upload_file(filename="testdt01.csv", path="./data/", bucket="dbrepo-upload")
+        S3Client().upload_file(filename="testdt01.csv", path="./data/", bucket="dbrepo-upload")
 
         # test
-        response = MinioClient().get_file(bucket="dbrepo-upload", filename="testdt01.csv")
+        response = S3Client().get_file(bucket="dbrepo-upload", filename="testdt01.csv")
         self.assertIsNotNone(response)
 
     # @Test
@@ -72,7 +72,7 @@ class MinioClientTest(unittest.TestCase):
 
         # test
         try:
-            MinioClient().get_file(bucket="dbrepo-upload", filename="idonotexist.csv")
+            S3Client().get_file(bucket="dbrepo-upload", filename="idonotexist.csv")
         except ClientError:
             pass
         except Exception:
@@ -84,7 +84,7 @@ class MinioClientTest(unittest.TestCase):
     def test_bucket_exists_succeeds(self):
 
         # test
-        response = MinioClient().bucket_exists_or_exit("dbrepo-upload")
+        response = S3Client().bucket_exists_or_exit("dbrepo-upload")
         self.assertIsNotNone(response)
 
     # @Test
@@ -92,7 +92,7 @@ class MinioClientTest(unittest.TestCase):
 
         # test
         try:
-            MinioClient().bucket_exists_or_exit("idnonotexist")
+            S3Client().bucket_exists_or_exit("idnonotexist")
         except FileNotFoundError:
             pass
         except Exception:
@@ -105,7 +105,7 @@ class MinioClientTest(unittest.TestCase):
 
         # test
         try:
-            MinioClient().bucket_exists_or_exit("idnonotexist")
+            S3Client().bucket_exists_or_exit("idnonotexist")
         except FileNotFoundError:
             pass
         except Exception:

dbrepo-data-db/sidecar/Dockerfile

@@ -19,8 +19,8 @@ COPY --chown=1001 ./ds-yml ./ds-yml
 COPY --chown=1001 ./app.py ./app.py
 
 ENV S3_STORAGE_ENDPOINT="http://storage-service:9000"
-ENV S3_ACCESS_KEY_ID="minioadmin"
-ENV S3_SECRET_ACCESS_KEY="minioadmin"
+ENV S3_ACCESS_KEY_ID="seaweedfsadmin"
+ENV S3_SECRET_ACCESS_KEY="seaweedfsadmin"
 
 EXPOSE 3305
 

dbrepo-data-db/sidecar/app.py

@@ -5,7 +5,7 @@ import logging
 from flasgger import LazyJSONEncoder, Swagger
 from flask import Flask, request, Response
 from flasgger.utils import swag_from
-from clients.minio_client import MinioClient
+from clients.s3_client import S3Client
 from prometheus_flask_exporter import PrometheusMetrics
 
 logging.basicConfig(level=logging.DEBUG)
@@ -106,8 +106,8 @@ def health():
 @swag_from("ds-yml/import.yml")
 def import_csv(filename):
     logging.debug('endpoint import csv, filename=%s, body=%s', filename, request)
-    minio_client = MinioClient()
-    response = minio_client.download_file(filename)
+    s3_client = S3Client()
+    response = s3_client.download_file(filename)
     if response is False:
         return Response(), 400
     return Response(json.dumps(response)), 202
@@ -117,8 +117,8 @@ def import_csv(filename):
 @swag_from("ds-yml/export.yml")
 def import_csv(filename):
     logging.debug('endpoint export csv, filename=%s, body=%s', filename, request)
-    minio_client = MinioClient()
-    response = minio_client.upload_file(filename)
+    s3_client = S3Client()
+    response = s3_client.upload_file(filename)
     if response is False:
         return Response(), 400
     return Response(), 202

dbrepo-data-db/sidecar/clients/minio_client.py → dbrepo-data-db/sidecar/clients/s3_client.py

@@ -6,14 +6,14 @@ import sys
 from botocore.exceptions import ClientError
 
 
-class MinioClient:
+class S3Client:
 
     def __init__(self):
         endpoint_url = os.getenv('S3_STORAGE_ENDPOINT', 'http://localhost:9000')
-        aws_access_key_id = os.getenv('S3_ACCESS_KEY_ID', 'minioadmin')
-        aws_secret_access_key = os.getenv('S3_SECRET_ACCESS_KEY', 'minioadmin')
-        logging.info("retrieve file from S3, endpoint_url=%s, aws_access_key_id=%s, aws_secret_access_key=(hidden)",
-                     endpoint_url, aws_access_key_id)
+        aws_access_key_id = os.getenv('S3_ACCESS_KEY_ID', 'seaweedfsadmin')
+        aws_secret_access_key = os.getenv('S3_SECRET_ACCESS_KEY', 'seaweedfsadmin')
+        logging.info(
+            f"retrieve file from S3, endpoint_url={endpoint_url}, aws_access_key_id={aws_access_key_id}, aws_secret_access_key=(hidden)")
         self.client = boto3.client(service_name='s3', endpoint_url=endpoint_url, aws_access_key_id=aws_access_key_id,
                                    aws_secret_access_key=aws_secret_access_key)
         self.bucket_exists_or_exit("dbrepo-upload")
@@ -29,7 +29,7 @@ class MinioClient:
         filepath = os.path.join("/tmp/", filename)
         try:
             self.client.upload_file(filepath, "dbrepo-download", filename)
-            logging.info("Uploaded .csv %s with key %s", filepath, filename)
+            logging.info(f"Uploaded .csv {filepath} with key {filename} into bucket dbrepo-download")
             return True
         except ClientError as e:
             logging.error(e)
@@ -46,7 +46,7 @@ class MinioClient:
         filepath = os.path.join("/tmp/", filename)
         try:
             self.client.download_file("dbrepo-upload", filename, filepath)
-            logging.info("Downloaded .csv with key %s into %s", filename, filepath)
+            logging.info(f"Downloaded .csv with key {filename} into {filepath} from bucket dbrepo-upload")
             return True
         except ClientError as e:
             logging.error(e)
@@ -58,10 +58,10 @@ class MinioClient:
             logging.debug(f"file with name {filename} exists in bucket with name {bucket}")
         except ClientError as e:
             if e.response["Error"]["Code"] == "404":
-                logging.error("Failed to find key %s in bucket %s", filename, bucket)
+                logging.error(f"Failed to find key {filename} in bucket {bucket}")
             else:
-                logging.error("Unexpected error when finding key %s in bucket %s: %s", filename, bucket,
-                              e.response["Error"]["Code"])
+                logging.error(
+                    f"Unexpected error when finding key {filename} in bucket {bucket}: {e.response['Error']['Code']}")
             raise e
 
     def bucket_exists_or_exit(self, bucket):
@@ -70,8 +70,7 @@ class MinioClient:
             logging.debug(f"bucket {bucket} exists.")
         except ClientError as e:
             if e.response["Error"]["Code"] == "404":
-                logging.error("Failed to find bucket %s", bucket)
+                logging.error(f"Failed to find bucket {bucket}")
             else:
-                logging.error("Unexpected error when finding bucket %s: %s", bucket,
-                              e.response["Error"]["Code"])
+                logging.error(f"Unexpected error when finding bucket {bucket}: {e.response['Error']['Code']}")
             sys.exit(1)

dbrepo-gateway-service/dbrepo.conf

@@ -30,10 +30,6 @@ upstream search-db-dashboard {
     server search-db-dashboard:5601;
 }
 
-upstream storage-service {
-    server storage-service:9001;
-}
-
 upstream upload {
     server upload-service:1080;
 }
@@ -60,22 +56,6 @@ server {
         proxy_read_timeout      90;
     }
 
-    location /admin/storage/ {
-        rewrite /admin/storage/(.*) /$1 break;
-        # http
-        proxy_http_version 1.1;
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-        chunked_transfer_encoding off;
-        # proxy
-        proxy_set_header        Host $host;
-        proxy_set_header        X-Real-IP $remote_addr;
-        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header        X-Forwarded-Proto $scheme;
-        proxy_pass              http://storage-service;
-        proxy_read_timeout      90;
-    }
-
     location /api/broker {
         rewrite /api/broker/(.*) /admin/broker/api/$1 break;
         proxy_set_header        Host $host;

dbrepo-metadata-service/Dockerfile

@@ -75,8 +75,8 @@ ENV DATACITE_PREFIX=""
 ENV DATACITE_USERNAME=""
 ENV DATACITE_PASSWORD=""
 ENV S3_STORAGE_ENDPOINT="http://storage-service:9000"
-ENV S3_ACCESS_KEY_ID="minioadmin"
-ENV S3_SECRET_ACCESS_KEY="minioadmin"
+ENV S3_ACCESS_KEY_ID="seaweedfsadmin"
+ENV S3_SECRET_ACCESS_KEY="seaweedfsadmin"
 
 WORKDIR /app
 

dbrepo-storage-service/Dockerfile

+FROM chrislusf/seaweedfs:3.59 as runtime
+
+RUN apk add curl
+
+WORKDIR /app
+
+COPY ./create-buckets.sh ./create-buckets.sh
+COPY ./docker-entrypoint.sh ./docker-entrypoint.sh
+
+ENTRYPOINT [ "/bin/sh", "./docker-entrypoint.sh" ]
\ No newline at end of file

dbrepo-storage-service/create-buckets.sh

+#!/bin/bash
+function log {
+  echo "$(date '+%Y-%m-%d %H:%M:%S') $1"
+}
+
+log "Sleep 15s to start S3 API"
+sleep 15
+log "Start polling"
+until curl -sSL 127.0.0.1:9000
+do
+    log "S3 API not ready on port 9000, wait 5s ..."
+    sleep 5
+done
+log "Ready"
+echo "s3.bucket.create -name dbrepo-upload" | weed shell
+log "Created bucket dbrepo-upload"
+echo "s3.bucket.create -name dbrepo-download" | weed shell
+log "Created bucket dbrepo-download"
\ No newline at end of file

dbrepo-storage-service/docker-entrypoint.sh

+#!/bin/sh
+/bin/sh ./create-buckets.sh &
+/entrypoint.sh server -dir=/data -s3 -s3.port=9000 -s3.config=/app/s3_config.json -metricsPort=9091
\ No newline at end of file

dbrepo-storage-service/s3_config.json

+{
+  "identities": [
+    {
+      "name": "admin",
+      "credentials": [
+        {
+          "accessKey": "seaweedfsadmin",
+          "secretKey": "seaweedfsadmin"
+        }
+      ],
+      "actions": [
+        "Read",
+        "Write",
+        "List",
+        "Tagging",
+        "Admin"
+      ]
+    }
+  ]
+}
\ No newline at end of file