Skip to content
Snippets Groups Projects
Select Git revision
  • 7fbe466dc6b8f96f3517aa41205e2ffbd4d6cbb5
  • master default protected
  • dev protected
  • release-1.10 protected
  • replication_test
  • 556-usage-statistics
  • 553-semantic-recommendation-2
  • 553-semantic-recommendation
  • release-1.9 protected
  • 551-init-broker-service-permissions
  • 549-test-oai-pmh
  • 545-saving-multiple-times-breaks-pid-metadata
  • 499-standalone-compute-service-2
  • 539-load-tests
  • hotfix/helm-chart
  • luca_ba_new_interface
  • 534-bug-when-adding-access-to-user-that-is-not-registered-at-dashboard-service
  • release-1.8 protected
  • 533-integrate-semantic-recommendation
  • feature/openshift
  • 518-spark-doesn-t-map-the-headers-correct
  • v1.10.4 protected
  • v1.10.3 protected
  • v1.10.2 protected
  • v1.10.1 protected
  • v1.10.0-rc13 protected
  • v1.10.0-rc12 protected
  • v1.10.0-rc11 protected
  • v1.10.0-rc10 protected
  • v1.10.0-rc9 protected
  • v1.10.0-rc8 protected
  • v1.10.0-rc7 protected
  • v1.10.0-rc6 protected
  • v1.10.0-rc5 protected
  • v1.10.0-rc4 protected
  • v1.10.0-rc3 protected
  • v1.10.0-rc2 protected
  • v1.10.0rc1 protected
  • v1.10.0rc0 protected
  • v1.10.0 protected
  • v1.9.3 protected
41 results

deployment-docker-compose.md

Blame
  • Martin Weise's avatar
    7fbe466d
    History
    author: Martin Weise

    Docker Compose

    TL;DR

    If you have Docker already installed on your system, you can install DBRepo with:

    curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/install.sh | bash

    Requirements

    Hardware

    For this small, local, test deployment any modern hardware would suffice, we recommend a dedicated virtual machine with the following settings.

    !!! tip "Resource Consumption"

    Note that most of the vCPU and RAM resources will be needed for starting the infrastructure, this is because of
    Docker. During operation and especially idle times, the deployment will use significantly less resources.
    • 8 vCPU cores
    • 16GB RAM memory
    • 100GB SSD storage

    Software

    We only test the Docker Compose deployment with the official Docker engine installed on a Debian-based operating system. Other software deployments (e.g. Docker Desktop on Windows) are not recommended and not tested.

    Architecture

    Overview

    The repository is designed as a service-based architecture to ensure scalability and the utilization of various technologies. The conceptualized microservices operate the basic database operations, data versioning as well as findability, accessability, interoperability and reuseability (FAIR).

    DBRepo architecture
    Architecture of the services deployed via Docker Compose

    Notes

    Please note that we only save the state of the databases as well as the Broker Service since RabbitMQ maintains state inside the container.

    Deployment

    We maintain a rapid prototype deployment option through Docker Compose (v2.17.0 and newer). This deployment creates the core infrastructure and a single Docker container for all user-generated databases.

    curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/install.sh | sudo bash

    View the logs:

    docker compose logs -f

    You should now be able to view the front end at http://localhost:80.

    Please be warned that the default configuration is not intended for public deployments. It is only intended to have a running system within minutes to play around within the system and explore features. It is strongly advised to change the default .env environment variables.

    !!! warning "Known security issues with the default configuration"

    The system is auto-configured for a small, local, test deployment and is *not* secure! You need to make modifications
    in various places to make it secure:
    
    * **Authentication Service**:
    
        a. You need to use your own instance or configure a secure instance using a (self-signed) certificate.
           Additionally, when serving from a non-default Authentication Service, you need to put it into the 
           `JWT_ISSUER` environment variable (`.env`).
    
        b. You need to change the default admin user `fda` password in Realm
           master > Users > fda > Credentials > Reset password.
    
        c. You need to change the client secrets for the clients `dbrepo-client` and `broker-client`. Do this in Realm
           dbrepo > Clients > dbrepo-client > Credentials > Client secret > Regenerate. Do the same for the
           broker-client.
    
        d. You need to regenerate the public key of the `RS256` algorithm which is shared with all services to verify 
           the signature of JWT tokens. Add your securely generated private key in Realm 
           dbrepo > Realm settings > Keys > Providers > Add provider > rsa.
    
    * **Broker Service**: by default, this service is configured with an administrative user that has major privileges.
      You need to change the password of the user *fda* in Admin > Update this user > Password. We found this
      [simple guide](https://onlinehelp.coveo.com/en/ces/7.0/administrator/changing_the_rabbitmq_administrator_password.htm)
      to be very useful.
    
    * **Search Database**: by default, this service is configured to require authentication with an administrative user
      that is allowed to write into the indizes. Following
      this [simple guide](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/reset-password.html), this can be
      achieved using the command line.
    
    * **Gateway Service**: by default, no HTTPS is used that protects the services behind. You need to provide a trusted
      SSL/TLS certificate in the configuration file or use your own proxy in front of the Gateway Service. See this
      [simple guide](http://nginx.org/en/docs/http/configuring_https_servers.html) on how to install a SSL/TLS
      certificate on NGINX.

    Upgrade Guide

    1.2 to 1.3

    In case you have a previous deployment from version 1.2, shut down the containers and back them up manually. You can do this by using the busybox image. Replace deadbeef with your container name or hash:

    export NAME=dbrepo-userdb-xyz
    docker run --rm --volumes-from $NAME -v /home/$USER/backup:/backup busybox tar pcvfz /backup/$NAME.tar.gz /var/lib/mysql

    !!! danger "Wipe all traces of DBRepo from your system"

    To erase all traces of DBRepo from your computer or virtual machine, the following commands delete all containers,
    volumes and networks that are present, execute the following **dangerous** command. It will **wipe** all information
    about DBRepo from your system (excluding the images).
    
    ```console
    docker container stop $(docker container ls -aq -f name=^/dbrepo-.*) || true
    docker container rm $(docker container ls -aq -f name=^/dbrepo-.*) || true
    docker volume rm $(docker volume ls -q -f name=^dbrepo-.*) || true
    docker network rm $(docker network ls -q -f name=^dbrepo-.*) || true
    ```

    You can restore the volume after downloading the new 1.3 images and creating the infrastructure:

    export NAME=dbrepo-userdb-xyz
    export PORT=12345
    docker container create -h $NAME --name $NAME -p $PORT:3306 -e MARIADB_ROOT_PASSWORD=mariadb --network userdb -v /backup mariadb:10.5
    docker run --rm --volumes-from $NAME -v /home/$USER/backup/.tar.gz:/backup/$NAME.tar.gz busybox sh -c 'cd /backup && tar xvfz /backup/$NAME.tar.gz && cp -r /backup/var/lib/mysql/* /var/lib/mysql'

    Future releases will be backwards compatible and will come with migration scripts.

    Limitations

    !!! info "Alternative Deployments"

    Alternatively, you can also deploy DBRepo with [Helm](../deployment-helm/) in your virtual machine instead.