diff --git a/docs/contact.md b/docs/contact.md index 651ff1903a70bea65564d49e486929fc149471c8..c0d5d499d76bb88d663109b12251ec9ae33b8501 100644 --- a/docs/contact.md +++ b/docs/contact.md @@ -33,5 +33,6 @@ A-1040 Vienna, Austria - Eva Gergely - Kirill Stytsenko - Josef Taha +- Tobias Grantner - Raman Ganguly - Andreas Rauber diff --git a/docs/index.md b/docs/index.md index 651294be122abd96ae872bcc8aa4fb116c5cf0dd..ee1c343bfb28e16680b102bfb9fc3a9a60defda2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -24,9 +24,9 @@ evolving, allows reproducing of query results and supports findable-, accessible ## More Information -- Public demonstration instance [https://dbrepo.ossdip.at](https://dbrepo.ossdip.at) -- Join our [mailing list](https://lists.univie.ac.at/mailman/listinfo/fairdata_dbrepo) to receive bi-monthly updates - about the progress, new features and events! +- Demonstration instance [https://dbrepo1.ec.tuwien.ac.at](https://dbrepo1.ec.tuwien.ac.at) +- Sandbox instance [https://dbrepo2.ec.tuwien.ac.at](https://dbrepo2.ec.tuwien.ac.at) +- System description [https://doi.org/10.2218/ijdc.v17i1.825](https://doi.org/10.2218/ijdc.v17i1.825) <style> video { diff --git a/docs/operation.md b/docs/operation.md index 4b14568ba18a9538d50437827d50eca57555627a..6371ff03cd5c1edfca7de94735c1bf44505715c7 100644 --- a/docs/operation.md +++ b/docs/operation.md @@ -10,18 +10,3 @@ hide: !!! abstract "Abstract" We compiled a short [getting started guide](/getting-started) that quickly allows a stand-alone test-deployment of the infrastructure on your server without the need of infrastructure configuration. - -## External Systems - -We thought the easiest way to make documentation of the endpoints available was through the interactive in-code -documentation of Swagger: - -* [Analyse Service](/swagger/analyse) -* [Authentication Service](/swagger/authentication) -* [Container Service](/swagger/container) -* [Database Service](/swagger/database) -* [Identifier Service](/swagger/identifier) -* [Metadata Service](/swagger/metadata) -* [Query Service](/swagger/query) -* [Table Service](/swagger/table) -* [Units Service](/swagger/units) \ No newline at end of file diff --git a/docs/publications.md b/docs/publications.md index d2cceadae9a00723fd53499661494ecccf2d0049..242d49a21957f0330f378c2dfcf6e03d67505cd3 100644 --- a/docs/publications.md +++ b/docs/publications.md @@ -11,7 +11,11 @@ hide: ##### 2022 -1. Weise, M., Staudinger, M., Michlits, C., Gergely, E., Stytsenko, K., Ganguly, R., & Rauber, A. (2022). +1. Ekaputra, F. E., Weise, M., Flicker, K., Salleh, M. R., Rahman, N. A., Miksa, T., & Rauber, A. (2022). Towards A + Data Repository for Educational Factories. *Proceedings of the 8th International Conference on Data and Software + Engineering*, pp. 149-154. DOI: [10.1109/ICoDSE56892.2022.9971958](https://doi.org/10.1109/ICoDSE56892.2022.9971958). + +2. Weise, M., Staudinger, M., Michlits, C., Gergely, E., Stytsenko, K., Ganguly, R., & Rauber, A. (2022). DBRepo: a Semantic Digital Repository for Relational Databases. *International Journal of Digital Curation*, 17(1), 11. DOI: [10.2218/ijdc.v17i1.825](https://doi.org/10.2218/ijdc.v17i1.825) diff --git a/docs/system.md b/docs/system.md index 11f9b596567ed97fc086050c1f385a3c261b192c..5ede2ebcf01cf705903a730b19472c78e441e7dd 100644 --- a/docs/system.md +++ b/docs/system.md @@ -13,8 +13,16 @@ The repository is designed as a microservice architecture to ensure scalability technologies. The conceptualized microservices operate the basic database operations, data versioning as well as *findability*, *accessability*, *interoperability* and *reuseability* (FAIR). +<figure markdown> + +<figcaption>Microservice cloud architecture</figcaption> +</figure> + + ## Services +View the docker images for the documentation of the service. + ### Discovery Service This microservice allows service discovery and registration of containers that provide services. It configures @@ -24,6 +32,7 @@ services. !!! debug "Debug Information" * Port(s): 9090 + * Docker Image: [dbrepo/discovery-service](https://hub.docker.com/repository/docker/dbrepo/discovery-service) * Swagger: not configured ### Gateway Service @@ -34,99 +43,47 @@ the [Spring Cloud Gateway](https://spring.io/projects/spring-cloud-gateway) to r !!! debug "Debug Information" * Port(s): 9095 + * Docker Image: [dbrepo/gateway-service](https://hub.docker.com/repository/docker/dbrepo/gateway-service) * Swagger: not configured -<figure markdown> - -<figcaption>Microservice cloud architecture</figcaption> -</figure> - ### Authentication Service Very specific to the deployment of the organization. In our reference implementation we implement a *security assertion markup language* (SAML) service provider and use our institutional SAML identity provider for obtaining account data through an encrypted channel. -The Authentication Service configures [Spring Boot Starter Security](https://spring.io/guides/gs/securing-web/) -with [Java JWT](https://github.com/auth0/java-jwt) for internal authentication once the user details are known in the -metadata database. By default, a token is valid for 24 hours and is used on all HTTP API endpoints. - -For the **HTTP API**, obtaining a new token can be done via, e.g. cURL. - -```console -curl -X POST -d '{"username":"username","password":"password"}' -H "Content-Type: application/json" https://dbrepo.ossdip.at/api/auth -``` - -Call a secured method by setting the JWT Token as [Bearer Token](https://www.rfc-editor.org/rfc/rfc6750.html) via, -e.g. cURL. - -```console -curl -X PUT -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" https://dbrepo.ossdip.at/api/auth -``` - -For the **AMQP API**, the Authentication Service also creates a dedicated user at the [Broker Service](#broker-service) -that has permissions for writing and configuring the RabbitMQ queues that feed into the database owned by this user. - !!! debug "Debug Information" * Port(s): 9097 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9097/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/authentication-service](http://dbrepo.ossdip.at:9097/v3/api-docs/authentication-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9097/v3/api-docs.yaml) + * Docker Image: [dbrepo/authentication-service](https://hub.docker.com/repository/docker/dbrepo/authentication-service) + * Swagger UI: [/swagger/authentication](/swagger/authentication) ### Metadata Database It is the core component of the project. It is a relational database that contains metadata about all researcher databases created in the database repository like column names, check expressions, value enumerations or key/value constraints and relevant data for citing data sets. Additionally, the concept, e.g. URI of units of measurements of numerical columns is -stored in the Metadata Database in order to provide semantic knowledge context. We -use [PostgreSQL](https://www.postgresql.org/) for its rich capabilities in the reference implementation. - -The default credentials are `postgres:postgres` for the database `fda`. Connect to the database via, e.g. *psql*. +stored in the Metadata Database in order to provide semantic knowledge context. We use MariaDB for its rich capabilities +in the reference implementation. -```console -psql -d fda -h localhost -p 5432 -U postgres -W -``` +The default credentials are `root:dbrepo` for the database `fda`. Connect to the database via the JDBC connector. !!! debug "Debug Information" - * Port(s): 5432 - * Swagger: not configured + * Port(s): 3306 -### Unit Service +### Semantics Service It is designed to map terms in the domain of units of measurement to controlled vocabulary, modelled in the [ontology of units of measure](https://github.com/HajoRijgersberg/OM). This service validates researcher provided in units and provides a *uniform resource identifier* (URI) to the related concept, which will be stored in the system. Furthermore, there is a method for auto-completing text and listing a description as well as commonly used unit symbols. -The Unit Service reads units of measurement from [`om-2.ttl`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-units-service/onto/om-2.ttl) -and registers a unit. It is used to assign a unit of measurement to a table column. - -For the **HTTP API**, the Unit Service assigns a unit of measurement via, e.g. cURL. First the list of concepts can be -queried. - -```console -curl -X POST -d '{"offset":0,"ustring":"met"}' https://dbrepo.ossdip.at/api/units/suggest -``` - -Then the concept needs to be saved into the metadata database. - -```console -curl -X POST -d '{"name":"metre","uri":...}' https://dbrepo.ossdip.at/api/units/saveconcept -``` - -Then the column can be assigned to a table column. - -```console -curl -X POST -d '{"cdbid":1,"tid":1,"cid":1,"uri":...}' https://dbrepo.ossdip.at/api/units/savecolumnsconcept -``` - !!! debug "Debug Information" * Port(s): 5010 - * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:9097/swagger-ui/) - * Swagger API .json: [/api-units.json](http://dbrepo.ossdip.at:9097/api-units.json) + * Docker Image: [dbrepo/semantics-service](https://hub.docker.com/repository/docker/dbrepo/semantics-service) + * Swagger UI: [/swagger/semantics](/swagger/semantics) ### Identifier Service @@ -139,9 +96,8 @@ identifier* (DOI) through our institutional library soon. !!! debug "Debug Information" * Port(s): 9096 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9096/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/identifier-service](http://dbrepo.ossdip.at:9096/v3/api-docs/identifier-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9096/v3/api-docs.yaml) + * Docker Image: [dbrepo/identifier-service](https://hub.docker.com/repository/docker/dbrepo/identifier-service) + * Swagger UI: [/swagger/identifier](/swagger/identifier) ### Search Service @@ -153,13 +109,14 @@ with each save operation on databases in the metadata database. The database nam to e.g. match the term "Airquality" ```console -curl http://dbrepo.ossdip.at:9200/databaseindex/_search?q=name:Airquality +curl http://localhost:9200/databaseindex/_search?q=name:Airquality ``` !!! debug "Debug Information" * Port(s): 9200, 9600 - * ElasticSearch: [/databaseindex](http://dbrepo.ossdip.at:9000/databaseindex) + * Docker Image: [elasticsearch](https://hub.docker.com/_/elasticsearch) + * ElasticSearch Index ### Container Service @@ -168,9 +125,8 @@ It is responsible for Docker container lifecycle operations and updating the loc !!! debug "Debug Information" * Port(s): 9091 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9091/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/container-service](http://dbrepo.ossdip.at:9091/v3/api-docs/container-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9091/v3/api-docs.yaml) + * Docker Image: [dbrepo/container-service](https://hub.docker.com/repository/docker/dbrepo/container-service) + * Swagger UI: [/swagger/container](/swagger/container) ### Database Service @@ -180,9 +136,8 @@ support [MariaDB](https://mariadb.org/) images that allow table versioning with !!! debug "Debug Information" * Port(s): 9092 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9092/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/database-service](http://dbrepo.ossdip.at:9092/v3/api-docs/database-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9092/v3/api-docs.yaml) + * Docker Image: [dbrepo/database-service](https://hub.docker.com/repository/docker/dbrepo/database-service) + * Swagger UI: [/swagger/database](/swagger/database) ### Table Service @@ -192,29 +147,20 @@ use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operation !!! debug "Debug Information" * Port(s): 9094 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9094/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/table-service](http://dbrepo.ossdip.at:9094/v3/api-docs/table-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9094/v3/api-docs.yaml) + * Docker Image: [dbrepo/table-service](https://hub.docker.com/repository/docker/dbrepo/table-service) + * Swagger UI: [/swagger/table](/swagger/table) ### Broker Service It holds exchanges and topics responsible for holding AMQP messages for later consumption. We -use [RabbitMQ](https://www.rabbitmq.com/) in the reference implementation. - -For the **HTTP API**, the Broker Service offers an endpoint to manage the AMQP users and their permissions on exchanges -and queues. This endpoint is reachable via the Gateway Service or at port 9098 directly. Internally this service just -parses commands to the [`rabbitmqctl`](https://www.rabbitmq.com/rabbitmqctl.8.html). - -For the **AMQP API**, the Broker Service can declare exchanges and queues. The AMQP endpoint listens to port 5672 for +use [RabbitMQ](https://www.rabbitmq.com/) in the reference implementation. The AMQP endpoint listens to port 5672 for regular declares and offers a management interface at port 15672. !!! debug "Debug Information" * Port(s): 9098, 5672, 15672 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9098/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/broker-service](http://dbrepo.ossdip.at:9098/v3/api-docs/broker-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9098/v3/api-docs.yaml) - * RabbitMQ Management: [/#](http://dbrepo.ossdip.at:15672/#) + * Docker Image: [dbrepo/broker-service](https://hub.docker.com/repository/docker/dbrepo/broker-service) + * RabbitMQ Management Plugin ### Query Service @@ -225,10 +171,8 @@ Service. !!! debug "Debug Information" * Port(s): 9093 - * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9093/swagger-ui/index.html) - * Swagger API .json: [/v3/api-docs/query-service](http://dbrepo.ossdip.at:9093/v3/api-docs/query-service) - * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9093/v3/api-docs.yaml) - + * Docker Image: [dbrepo/query-service](https://hub.docker.com/repository/docker/dbrepo/query-service) + * Swagger UI: [/swagger/query](/swagger/query) ### FAIR Portal It provides a *graphical user interface* (GUI) for a researcher to interact with the database repository's API. @@ -236,7 +180,8 @@ It provides a *graphical user interface* (GUI) for a researcher to interact with !!! debug "Debug Information" * Port(s): 3000 - * GUI: [/#](http://dbrepo.ossdip.at:3000) + * Docker Image: [dbrepo/ui](https://hub.docker.com/repository/docker/dbrepo/ui) + * FAIR Portal ### Analyse Service @@ -248,8 +193,8 @@ numerical columns. !!! debug "Debug Information" * Port(s): 5000 - * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:5000/swagger-ui/) - * Swagger API .json: [/api-analyze.json](http://dbrepo.ossdip.at:5000/api-analyze.json) + * Docker Image: [dbrepo/analyse-service](https://hub.docker.com/repository/docker/dbrepo/analyse-service) + * Swagger UI: [/swagger/analyse](/swagger/analyse) ## Database