diff --git a/docs/contact.md b/docs/contact.md index cfad049c1d674ad7cbdee8393ad282d8e9a3fbd0..651ff1903a70bea65564d49e486929fc149471c8 100644 --- a/docs/contact.md +++ b/docs/contact.md @@ -32,5 +32,6 @@ A-1040 Vienna, Austria - Cornelia Michlits - Eva Gergely - Kirill Stytsenko +- Josef Taha - Raman Ganguly - Andreas Rauber diff --git a/docs/getting-started.md b/docs/getting-started.md index 81c65c5c5d4c6b82027771e913506e5155b247ad..f1af07db710059f6ad4e3a534951f0dd7bb0e885 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -48,19 +48,23 @@ We currently maintain 14 images on [Dockerhub](https://hub.docker.com/u/dbrepo). [//]: # (=== ":fontawesome-brands-windows: Windows") [//]: # () + [//]: # ( Download the latest [`docker-compose.yml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml) ) [//]: # ( and [`.env`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example) ) [//]: # ( ) + [//]: # ( $ curl.exe -o docker-compose.yml https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml) [//]: # ( $ curl.exe -o .env https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example) [//]: # () + [//]: # ( Start the services) [//]: # () + [//]: # ( $ docker-compose up -d) === ":fontawesome-brands-apple: macOS" @@ -92,8 +96,7 @@ Activate them both in your virtualization infrastructure and the local firewall ## Development We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by -sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise. Clone the repository and create a feature branch -from `dev` and implement your changes. +sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise. ### Software @@ -108,22 +111,58 @@ We develop all packages with the following software requirements: ### Building -For local development you need to install the entities from the metadata database and the general DTOs that are -exchanged between the services by installing the package: +#### Services and GUI + +For local development you need to clone the repository and build the services from scratch: + +```console +$ git clone git@gitlab.phaidra.org:fair-data-austria-db-repository/fda-services.git +$ make build +``` + +#### Swagger Codegen + +It is possible to use the Swagger documentation of the services to generate HTTP clients to interact with DBRepo. +Clients can only begenerated when DBRepo is running. Just download +the [`swagger-codegen-cli.jar`](https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/) and start the services: ```console -mvn -f ./fda-metadata-db/pom.xml clean install +$ docker-compose up -d ``` -All other services depend on this package. You have to build them later: +Run the codegen tool: ```console -mvn -f ./fda-container-service/pom.xml clean package -DskipTests -... +$ java -jar ./swagger-codegen-cli-2.x.y.jar generate -i ./api-docs.yaml -l python -o ./api-container ``` -Conviniently, you can build them all through [`make`](https://www.gnu.org/software/make/): +### Debugging + +#### Backend + +Start all containers with docker-compose: ```console -make build-backend -``` \ No newline at end of file +$ docker-compose up -d +$ docker container stop fda-xxxxx-service # <<< replace with the service you want to debug +$ docker container rm fda-xxxxx-service # <<< replace with the service you want to debug +$ docker-compose logs -f +``` + +Then start the container in your IDE and debug it. + +#### Frontend + +Start all containers with docker-compose: + +```console +$ docker-compose up -d +$ docker container stop fda-ui && docker container rm fda-ui +$ docker-compose logs -f +``` + +Then start the frontend in another console window: + +```console +$ yarn --cwd=./fda-ui dev +``` diff --git a/docs/system.md b/docs/system.md index b3fffe1d2bc697f010cf0f510a6d71a546710566..788fef061a7f32b55a063bc2c2c5234086c5124e 100644 --- a/docs/system.md +++ b/docs/system.md @@ -13,45 +13,6 @@ The repository is designed as a microservice architecture to ensure scalability technologies. The conceptualized microservices operate the basic database operations, data versioning as well as *findability*, *accessability*, *interoperability* and *reuseability* (FAIR). -## Database - -This container runs a relational database engine that allows data versioning and contains the Query Store, a special -table that stores all queries issued to the Researcher Database along with metadata. We store the queries here and not -in the metadata database level to ensure that they are preserved along with the original database for a regular backup -and archival together with the original database once the container is retired. - -### Container - -Currently, we only support databases with -the [MariaDB engine](https://hub.docker.com/_/mariadb?tab=tags&page=1&name=10.5&ordering=-name). -DBRepo creates a *root* user for managing the tables, inserting data, etc. and provides a *mariadb* user that is only -granted `select` access to all tables. The default passwords need to be changed at -[`AbstractSeeder.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-container-service/services/src/main/java/at/tuwien/seeder/impl/AbstractSeeder.java#L39-L51) - -### Query Store - -The Query Store is a special table (`qs_queries`) that stores all queries issued to the database via the HTTP API. It -stores meta-information about the queries directly in the database container: - -<figure markdown> -| Name | Type | Constraint | Default | Comment | -|------------------|--------------|-------------|-------------------------|-------------------------------| -| id | bigint | primary key | nextval(qs_queries_seq) | | -| cid | bigint | | | Column ID | -| dbid | bigint | | | Database ID | -| created | datetime | | now() | | -| created_by | bigint | | | Creator User-ID | -| execution | datetime | | | | -| last_modified | datetime | | | | -| query | text | | | | -| query_normalized | text | | | removing *, randomness | -| query_hash | varchar(255) | | | sha256 hash of `query` field | -| result_hash | varchar(255) | | | sha256 hash of the result set | -| result_number | bigint | | | | - -<figcaption>Query Store table <code>qs_queries</code> schema</figcaption> -</figure> - ## Services ### Discovery Service @@ -109,9 +70,9 @@ that has permissions for writing and configuring the RabbitMQ queues that feed i !!! debug "Debug Information" * Port(s): 9097 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9097/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/authentication-service](http://localhost:9097/v3/api-docs/authentication-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9097/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9097/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/authentication-service](http://dbrepo.ossdip.at:9097/v3/api-docs/authentication-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9097/v3/api-docs.yaml) ### Metadata Database @@ -164,8 +125,8 @@ $ curl -X POST -d '{"cdbid":1,"tid":1,"cid":1,"uri":...}' https://dbrepo.ossdip. !!! debug "Debug Information" * Port(s): 5010 - * Swagger UI: [/swagger-ui/](http://localhost:9097/swagger-ui/) - * Swagger API .json: [/api-units.json](http://localhost:9097/api-units.json) + * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:9097/swagger-ui/) + * Swagger API .json: [/api-units.json](http://dbrepo.ossdip.at:9097/api-units.json) ### Identifier Service @@ -178,9 +139,9 @@ identifier* (DOI) through our institutional library soon. !!! debug "Debug Information" * Port(s): 9096 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9096/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/identifier-service](http://localhost:9096/v3/api-docs/identifier-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9096/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9096/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/identifier-service](http://dbrepo.ossdip.at:9096/v3/api-docs/identifier-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9096/v3/api-docs.yaml) ### Search Service @@ -192,13 +153,13 @@ with each save operation on databases in the metadata database. The database nam to e.g. match the term "Airquality" ```console -$ curl http://localhost:9200/databaseindex/_search?q=name:Airquality +$ curl http://dbrepo.ossdip.at:9200/databaseindex/_search?q=name:Airquality ``` !!! debug "Debug Information" * Port(s): 9200, 9600 - * ElasticSearch: [/databaseindex](http://localhost:9000/databaseindex) + * ElasticSearch: [/databaseindex](http://dbrepo.ossdip.at:9000/databaseindex) ### Container Service @@ -207,9 +168,9 @@ It is responsible for Docker container lifecycle operations and updating the loc !!! debug "Debug Information" * Port(s): 9091 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9091/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/container-service](http://localhost:9091/v3/api-docs/container-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9091/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9091/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/container-service](http://dbrepo.ossdip.at:9091/v3/api-docs/container-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9091/v3/api-docs.yaml) ### Database Service @@ -219,9 +180,9 @@ support [MariaDB](https://mariadb.org/) images that allow table versioning with !!! debug "Debug Information" * Port(s): 9092 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9092/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/database-service](http://localhost:9092/v3/api-docs/database-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9092/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9092/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/database-service](http://dbrepo.ossdip.at:9092/v3/api-docs/database-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9092/v3/api-docs.yaml) ### Table Service @@ -231,9 +192,9 @@ use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operation !!! debug "Debug Information" * Port(s): 9094 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9094/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/table-service](http://localhost:9094/v3/api-docs/table-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9094/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9094/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/table-service](http://dbrepo.ossdip.at:9094/v3/api-docs/table-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9094/v3/api-docs.yaml) ### Broker Service @@ -250,10 +211,10 @@ regular declares and offers a management interface at port 15672. !!! debug "Debug Information" * Port(s): 9098, 5672, 15672 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9098/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/broker-service](http://localhost:9098/v3/api-docs/broker-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9098/v3/api-docs.yaml) - * RabbitMQ Management: [/#](http://localhost:15672/#) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9098/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/broker-service](http://dbrepo.ossdip.at:9098/v3/api-docs/broker-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9098/v3/api-docs.yaml) + * RabbitMQ Management: [/#](http://dbrepo.ossdip.at:15672/#) ### Query Service @@ -264,9 +225,9 @@ Service. !!! debug "Debug Information" * Port(s): 9093 - * Swagger UI: [/swagger-ui/index.html](http://localhost:9093/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/query-service](http://localhost:9093/v3/api-docs/query-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9093/v3/api-docs.yaml) + * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9093/swagger-ui/index.html) + * Swagger API .json: [/v3/api-docs/query-service](http://dbrepo.ossdip.at:9093/v3/api-docs/query-service) + * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9093/v3/api-docs.yaml) ### FAIR Portal @@ -275,7 +236,7 @@ It provides a *graphical user interface* (GUI) for a researcher to interact with !!! debug "Debug Information" * Port(s): 3000 - * GUI: [/#](http://localhost:3000) + * GUI: [/#](http://dbrepo.ossdip.at:3000) ### Analyse Service @@ -287,5 +248,44 @@ numerical columns. !!! debug "Debug Information" * Port(s): 5000 - * Swagger UI: [/swagger-ui/](http://localhost:5000/swagger-ui/) - * Swagger API .json: [/api-analyze.json](http://localhost:5000/api-analyze.json) + * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:5000/swagger-ui/) + * Swagger API .json: [/api-analyze.json](http://dbrepo.ossdip.at:5000/api-analyze.json) + +## Database + +This container runs a relational database engine that allows data versioning and contains the Query Store, a special +table that stores all queries issued to the Researcher Database along with metadata. We store the queries here and not +in the metadata database level to ensure that they are preserved along with the original database for a regular backup +and archival together with the original database once the container is retired. + +### Container + +Currently, we only support databases with +the [MariaDB engine](https://hub.docker.com/_/mariadb?tab=tags&page=1&name=10.5&ordering=-name). +DBRepo creates a *root* user for managing the tables, inserting data, etc. and provides a *mariadb* user that is only +granted `select` access to all tables. The default passwords need to be changed at +[`AbstractSeeder.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-container-service/services/src/main/java/at/tuwien/seeder/impl/AbstractSeeder.java#L39-L51) + +### Query Store + +The Query Store is a special table (`qs_queries`) that stores all queries issued to the database via the HTTP API. It +stores meta-information about the queries directly in the database container: + +<figure markdown> +| Name | Type | Constraint | Default | Comment | +|------------------|--------------|-------------|-------------------------|-------------------------------| +| id | bigint | primary key | nextval(qs_queries_seq) | | +| cid | bigint | | | Column ID | +| dbid | bigint | | | Database ID | +| created | datetime | | now() | | +| created_by | bigint | | | Creator User-ID | +| execution | datetime | | | | +| last_modified | datetime | | | | +| query | text | | | | +| query_normalized | text | | | removing *, randomness | +| query_hash | varchar(255) | | | sha256 hash of `query` field | +| result_hash | varchar(255) | | | sha256 hash of the result set | +| result_number | bigint | | | | + +<figcaption>Query Store table <code>qs_queries</code> schema</figcaption> +</figure> diff --git a/swagger/api-query.yaml b/swagger/api-query.yaml index 800d2579980ac76575b13bab03067ac99eab2f91..718b3faa2ff50e4c49711a97ccbadfd8222289aa 100644 --- a/swagger/api-query.yaml +++ b/swagger/api-query.yaml @@ -21,7 +21,7 @@ paths: tags: - table-history-endpoint summary: Find all history - operationId: getAll + operationId: getAll_1 parameters: - name: id in: path @@ -86,7 +86,7 @@ paths: tags: - table-history-endpoint summary: Find all history - operationId: getAll_1 + operationId: getAll parameters: - name: id in: path @@ -152,7 +152,7 @@ paths: tags: - table-data-endpoint summary: Find data - operationId: getAll_2 + operationId: getAll_3 parameters: - name: id in: path @@ -428,7 +428,7 @@ paths: tags: - table-data-endpoint summary: Find data - operationId: getAll_3 + operationId: getAll_2 parameters: - name: id in: path diff --git a/swagger/generate.sh b/swagger/generate.sh index ce679dfa6bc465b01bf6b5c9d9962a7ca131656a..bfbd59172292efca8c3398e5d03517b8b6d25068 100755 --- a/swagger/generate.sh +++ b/swagger/generate.sh @@ -25,20 +25,20 @@ function generate () { function remove () { echo "... removing old python api" - rm -rf "${HOME}/Projects/fda-services/.jupyter/api_$1" + rm -rf "${HOME}/Projects/fda-services/.gitlab/tests/api_$1" rm -rf "${HOME}/Projects/fda-services/.demo/api_$1" } function copy () { echo "... copying python api" - cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.jupyter/api_$1" + cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.gitlab/tests/api_$1" cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.demo/api_$1" } function replace () { echo "... replacing swagger client package name and gateway" - find "${HOME}/Projects/fda-services/.jupyter/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \; - find "${HOME}/Projects/fda-services/.jupyter/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \; + find "${HOME}/Projects/fda-services/.gitlab/tests/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \; + find "${HOME}/Projects/fda-services/.gitlab/tests/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \; find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \; find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \; }