diff --git a/.docs/system-services-metadata.md b/.docs/system-services-metadata.md index a44316cc07954d2b24e73bedfcb6d1dfa0a08376..07756f0bdb47f2b80b7de157475732be15bf08a7 100644 --- a/.docs/system-services-metadata.md +++ b/.docs/system-services-metadata.md @@ -32,7 +32,7 @@ This service manages the following topics: ### Databases -The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and +The service handles table operations inside a database. We use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operations. ### Identifiers @@ -47,17 +47,16 @@ This service provides an OAI-PMH endpoint for metadata aggregators. ### Queries -It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query +It provides an interface to insert data into the tables. It also allows for view-only, paginated and versioned query execution to the raw data. ### Semantics The service provides metadata to the table columns in the [Metadata Database](../system-databases-metadata) from -registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of -Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a +registered ontologies like Wikidata [`wd:`](https://wikidata.org), Ontology of Units of +Measurement [`om2:`](https://www.ontology-of-units-of-measure.org/resource/om-2), Friend of a Friend [`foaf:`](http://xmlns.com/foaf/0.1/), the [`prov:`](http://www.w3.org/ns/prov#) namespace, etc. - ### Tables The service manages tables in the [Data Database](../system-databases-data) and manages the metadata of these tables @@ -65,10 +64,38 @@ in the [Metadata Database](../system-databases-metadata). ### Users -The service manages users in the [Data Database](../system-databases-data) +The service manages users in the [Data Database](../system-databases-data) and [Metadata Database](../system-databases-metadata), as well as in the [Broker Service](../system-services-broker) and the [Authentication Service](../system-services-authentication). +The default configuration grants the users only very basic permissions on the databases: + +* `SELECT` +* `CREATE` +* `CREATE VIEW` +* `CREATE ROUTINE` +* `CREATE TEMPORARY TABLES` +* `LOCK TABLES` +* `INDEX` +* `TRIGGER` +* `INSERT` +* `UPDATE` +* `DELETE` + +This configuration is passed as environment variable `GRANT_PRIVILEGES` to the service as comma-separated string. You +can add/remove grants by setting this environment variable, e.g. allow the users to only select data and create +temporary tables: + +```yaml title="docker-compose.yml" +services: + dbrepo-metadata-service: + environment: + GRANT_PRIVILEGES=SELECT,CREATE TEMPORARY TABLES + ... +``` + +A list of all grants is available in the MariaDB documentation for [`GRANT`](https://mariadb.com/kb/en/grant/) + ### Views The service manages views in the [Data Database](../system-databases-data) diff --git a/.docs/system-services-mirror.md b/.docs/system-services-mirror.md deleted file mode 100644 index ab1052ddd8c667b7619fad2ff435664afe1f82a6..0000000000000000000000000000000000000000 --- a/.docs/system-services-mirror.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -author: Martin Weise ---- - -# Mirror Service - -## tl;dr - -!!! debug "Debug Information" - - Image: [`dbrepo/mirror-service:1.4`](https://hub.docker.com/r/dbrepo/mirror-service) - - * Ports: 9050/tcp - * Info: `http://<hostname>:9050/actuator/info` - * Health: `http://<hostname>:9050/actuator/health` - - Readiness: `http://<hostname>:9050/actuator/health/readiness` - - Liveness: `http://<hostname>:9050/actuator/health/liveness` - * Prometheus: `http://<hostname>:9050/actuator/prometheus` - * Swagger UI: `http://<hostname>:9050/swagger-ui/index.html` <a href="../swagger/mirror" target="_blank">:fontawesome-solid-square-up-right: view online</a> - -## Overview - -This service is responsible for synchronizing the [Metadata Database](../system-databases-metadata) with -the [Search Database](../system-databases-search) and the user permissions of databases, tables, etc. with -the [Broker Service](../system-services-broker). - -| Metadata DB | ↦ | Search DB | -|---------------------|:-------:|---------------| -| `mdb_users` | | `/user` | -| `mdb_view` | | `/view` | -| `mdb_databases` | | `/database` | -| `mdb_identifiers` | | `/identifier` | -| `mdb_concepts` | | `/concept` | -| `mdb_columns` | | `/column` | -| `mdb_tables` | | `/table` | -| `mdb_units` | | `/unit` | - -## Limitations - -* No support for cron-job like execution. -* No support for conditional updates in the [Search Database](../system-databases-search), updates occur in defined - intervals. - -!!! question "Do you miss functionality? Do these limitations affect you?" - - We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get - in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming - experience! - -## Security - -(none) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index c484b51fd2740f8cec8e52719336f8bd8cd1f018..398a27b18ee30cb394be27d9c595104bf9b232a8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -62,6 +62,8 @@ build-search-service: build-docker: image: docker.io/docker:24-dind stage: build + before_script: + - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL script: - "cp .env.unix.example .env" - "docker build -t dbrepo-metadata-service:build --target build dbrepo-metadata-service" @@ -468,11 +470,12 @@ release-latest: only: refs: - dev + before_script: + - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL + - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL script: - "ifconfig eth0 mtu 1450 up" - "apk add make" - - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL - - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL - TAG=latest make release release-1.3: @@ -486,11 +489,12 @@ release-1.3: only: refs: - release-v1.3 + before_script: + - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL + - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL script: - "ifconfig eth0 mtu 1450 up" - "apk add make" - - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL - - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL - "TAG=1.3 make release" release-1.4: @@ -504,11 +508,12 @@ release-1.4: only: refs: - release-v1.4 + before_script: + - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL + - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL script: - "ifconfig eth0 mtu 1450 up" - "apk add make" - - echo "$CI_REGISTRY_PASSWORD" | docker login --username "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY_URL - - echo "$CI_REGISTRY2_PASSWORD" | docker login --username "$CI_REGISTRY2_USER" --password-stdin $CI_REGISTRY2_URL - "TAG=1.4 make release" build-api-latest: diff --git a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryBriefDto.java b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryBriefDto.java index 62a347107409b5b8823fa53a445221bbe80a0927..9b48ff9ebfff35806d331541ba6e1333168973f8 100644 --- a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryBriefDto.java +++ b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryBriefDto.java @@ -30,11 +30,9 @@ public class QueryBriefDto { @NotNull(message = "id is required") private Long id; - @NotNull(message = "container id is required") - private Long cid; - @NotNull(message = "database id is required") - private Long dbid; + @JsonProperty("database_id") + private Long databaseId; @JsonIgnore @NotNull(message = "created by is required") diff --git a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryDto.java b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryDto.java index 59533d41b02b06f3c8da108cfea4263b1eceb5fc..593b65ce5bcea34316fc29576847c34e506f20a4 100644 --- a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryDto.java +++ b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/database/query/QueryDto.java @@ -28,11 +28,9 @@ public class QueryDto { @NotNull(message = "id is required") private Long id; - @NotNull(message = "container id is required") - private Long cid; - @NotNull(message = "database id is required") - private Long dbid; + @JsonProperty("database_id") + private Long databaseId; @JsonIgnore @EqualsAndHashCode.Exclude diff --git a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/user/UserDto.java b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/user/UserDto.java index 3ef24c16c1d23c221b44b6148d1833cbe50b6c52..069efaf3ff683f3a169b8f2365f770034a538597 100644 --- a/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/user/UserDto.java +++ b/dbrepo-metadata-service/api/src/main/java/at/tuwien/api/user/UserDto.java @@ -57,9 +57,4 @@ public class UserDto { @NotNull private UserAttributesDto attributes; - @NotNull - @org.springframework.data.annotation.Transient - @Schema(example = "jcarberry@brown.edu") - private String email; - } diff --git a/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/database/View.java b/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/database/View.java index c5540ecf897559e5739da2940433534cc59a1c61..7888bb6fdde986f1e5c73213a811ac6c940fc9f6 100644 --- a/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/database/View.java +++ b/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/database/View.java @@ -2,6 +2,7 @@ package at.tuwien.entities.database; import at.tuwien.entities.database.table.columns.TableColumn; import at.tuwien.entities.identifier.Identifier; +import at.tuwien.entities.user.User; import com.fasterxml.jackson.annotation.JsonFormat; import jakarta.persistence.*; import jakarta.persistence.CascadeType; @@ -47,10 +48,17 @@ public class View { @Column(updatable = false, nullable = false) private Long vdbid; + @ToString.Exclude @JdbcTypeCode(java.sql.Types.VARCHAR) - @Column(name = "createdBy", nullable = false, columnDefinition = "VARCHAR(36)") + @Column(name = "created_by", columnDefinition = "VARCHAR(36)") private UUID createdBy; + @ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.PERSIST) + @JoinColumns({ + @JoinColumn(name = "created_by", referencedColumnName = "ID", insertable = false, updatable = false) + }) + private User creator; + @Column(name = "vname", nullable = false) private String name; diff --git a/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/identifier/Identifier.java b/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/identifier/Identifier.java index 0fb92413730f379f336faa5b514092919be0126f..6b2cba565be171e7108fc2c90732732545bddeee 100644 --- a/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/identifier/Identifier.java +++ b/dbrepo-metadata-service/entities/src/main/java/at/tuwien/entities/identifier/Identifier.java @@ -31,11 +31,11 @@ import java.util.UUID; @EntityListeners(AuditingEntityListener.class) @Table(name = "mdb_identifiers") @NamedQueries({ - @NamedQuery(name = "Identifier.findAllDatabaseIdentifiers", query = "select i from Identifier i where i.type = 'DATABASE'"), - @NamedQuery(name = "Identifier.findAllSubsetIdentifiers", query = "select i from Identifier i where i.type = 'SUBSET'"), - @NamedQuery(name = "Identifier.findDatabaseIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.type = 'DATABASE'"), - @NamedQuery(name = "Identifier.findSubsetIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.queryId = ?2 and i.type = 'SUBSET'"), - @NamedQuery(name = "Identifier.findViewIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.viewId = ?2 and i.type = 'VIEW'"), + @NamedQuery(name = "Identifier.findAllDatabaseIdentifiers", query = "select i from Identifier i where i.type = 'DATABASE' ORDER BY i.id DESC"), + @NamedQuery(name = "Identifier.findAllSubsetIdentifiers", query = "select i from Identifier i where i.type = 'SUBSET' ORDER BY i.id DESC"), + @NamedQuery(name = "Identifier.findDatabaseIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.type = 'DATABASE' ORDER BY i.id DESC"), + @NamedQuery(name = "Identifier.findSubsetIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.queryId = ?2 and i.type = 'SUBSET' ORDER BY i.id DESC"), + @NamedQuery(name = "Identifier.findViewIdentifier", query = "select i from Identifier i where i.databaseId = ?1 and i.viewId = ?2 and i.type = 'VIEW' ORDER BY i.id DESC"), }) public class Identifier implements Serializable { diff --git a/dbrepo-metadata-service/querystore/src/main/java/at/tuwien/querystore/Query.java b/dbrepo-metadata-service/querystore/src/main/java/at/tuwien/querystore/Query.java index e6925b48c5bfa30a1b568d7647358233e5daf2c2..272c03f65fcba81cf36fa80cfbcc12437958da40 100644 --- a/dbrepo-metadata-service/querystore/src/main/java/at/tuwien/querystore/Query.java +++ b/dbrepo-metadata-service/querystore/src/main/java/at/tuwien/querystore/Query.java @@ -6,9 +6,10 @@ import org.hibernate.annotations.GenericGenerator; import org.springframework.data.annotation.CreatedDate; import org.springframework.data.jpa.domain.support.AuditingEntityListener; -import jakarta.persistence.*;; +import jakarta.persistence.*; import java.io.Serializable; import java.time.Instant; +import java.util.UUID; @Data @Entity @@ -61,6 +62,6 @@ public class Query implements Serializable { private Instant executed; @jakarta.persistence.Column(nullable = false) - private String createdBy; + private UUID createdBy; } diff --git a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java index 09839e09f2a18c44c0d818d6504bc081b35aa271..5b3aca08aa9e4dcd5a89846d28e29d04e85233a6 100644 --- a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java +++ b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/QueryMapper.java @@ -48,7 +48,7 @@ import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.stream.Collectors; -@Mapper(componentModel = "spring") +@Mapper(componentModel = "spring", imports = {LinkedList.class}) public interface QueryMapper { org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(QueryMapper.class); @@ -57,12 +57,13 @@ public interface QueryMapper { .withZone(ZoneId.of("UTC")); @Mappings({ - @Mapping(target = "createdBy", ignore = true) + @Mapping(target = "createdBy", ignore = true), + @Mapping(target = "identifiers", expression = "java(new LinkedList())") }) QueryDto queryToQueryDto(Query data); @Mappings({ - @Mapping(target = "createdBy", ignore = true) + @Mapping(target = "identifiers", expression = "java(new LinkedList())") }) QueryBriefDto queryToQueryBriefDto(Query data); @@ -123,14 +124,7 @@ public interface QueryMapper { default void importCsvQuery(Connection connection, Table table, ImportDto csv) throws SQLException { final Statement statement = connection.createStatement(); - final StringBuilder query0 = new StringBuilder("DROP TABLE IF EXISTS `") - .append(table.getDatabase().getInternalName()) - .append("`.`") - .append(table.getInternalName()) - .append("_temporary`;"); - log.trace("mapped drop temporary table statement: {}", query0); - statement.execute(query0.toString()); - final StringBuilder query1 = new StringBuilder("CREATE TABLE `") + final StringBuilder query0 = new StringBuilder("CREATE TABLE `") .append(table.getDatabase().getInternalName()) .append("`.`") .append(table.getInternalName()) @@ -140,13 +134,20 @@ public interface QueryMapper { .append("`.`") .append(table.getInternalName()) .append("`;"); - log.trace("mapped create temporary table statement: {}", query1); + log.trace("mapped create temporary table statement: {}", query0); + statement.execute(query0.toString()); + final String query1 = pathToRawInsertQuery(table, csv); + log.trace("mapped import csv statement: {}", query1); statement.execute(query1.toString()); - final String query2 = pathToRawInsertQuery(table, csv); - log.trace("mapped import csv statement: {}", query2); + final String query2 = generateInsertFromTemporaryTableSQL(table); + log.trace("mapped import table statement: {}", query2); statement.execute(query2.toString()); - final String query3 = generateInsertFromTemporaryTableSQL(table); - log.trace("mapped import table statement: {}", query3); + final StringBuilder query3 = new StringBuilder("DROP TABLE IF EXISTS `") + .append(table.getDatabase().getInternalName()) + .append("`.`") + .append(table.getInternalName()) + .append("_temporary`;"); + log.trace("mapped drop temporary table statement: {}", query3); statement.execute(query3.toString()); } diff --git a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/StoreMapper.java b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/StoreMapper.java index 285d8fd14dd5aed30fbd4e919106395f95256072..cce0c72866eb1df34f68aeb959e6b165322e134f 100644 --- a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/StoreMapper.java +++ b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/StoreMapper.java @@ -12,6 +12,7 @@ import java.time.Instant; import java.time.LocalDateTime; import java.time.ZoneId; import java.time.format.DateTimeFormatter; +import java.util.UUID; @Mapper(componentModel = "spring") public interface StoreMapper { @@ -32,8 +33,8 @@ public interface StoreMapper { } try { final CallableStatement ps = connection.prepareCall(statement); - ps.setString(1, user.getUsername()); - log.trace("param 1={}", user.getUsername()); + ps.setString(1, String.valueOf(user.getId())); + log.trace("param 1={}", user.getId()); ps.setString(2, data.getStatement()); log.trace("param 2={}", data.getStatement()); ps.setTimestamp(3, Timestamp.from(data.getTimestamp())); @@ -99,7 +100,7 @@ public interface StoreMapper { return Query.builder() .id(data.getLong(1)) .created(createdInst) - .createdBy(data.getString(3)) + .createdBy(UUID.fromString(data.getString(3))) .query(data.getString(4)) .queryHash(data.getString(5)) .resultHash(data.getString(6)) diff --git a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/TableMapper.java b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/TableMapper.java index 3e7ce40092cb571330ca8dffbb1ebf6e4dcf28bf..80510416cee75bff784a1199d688b0452386f862 100644 --- a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/TableMapper.java +++ b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/TableMapper.java @@ -44,7 +44,7 @@ import java.util.*; import java.util.regex.Pattern; import java.util.stream.Collectors; -@Mapper(componentModel = "spring", uses = {IdentifierMapper.class}) +@Mapper(componentModel = "spring", uses = {IdentifierMapper.class, UserMapper.class}) public interface TableMapper { org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(TableMapper.class); diff --git a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/ViewMapper.java b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/ViewMapper.java index e8b12e3c38b326dd59124176735f076f935c3891..272275fec6a2ac44ff5748585d0c365510cb5da6 100644 --- a/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/ViewMapper.java +++ b/dbrepo-metadata-service/repositories/src/main/java/at/tuwien/mapper/ViewMapper.java @@ -1,13 +1,9 @@ package at.tuwien.mapper; -import at.tuwien.api.database.DatabaseDto; import at.tuwien.api.database.ViewBriefDto; import at.tuwien.api.database.ViewCreateDto; import at.tuwien.api.database.ViewDto; -import at.tuwien.api.identifier.IdentifierDto; -import at.tuwien.entities.database.Database; import at.tuwien.entities.database.View; -import at.tuwien.entities.identifier.Identifier; import at.tuwien.exception.QueryMalformedException; import org.mapstruct.Mapper; import org.mapstruct.Mapping; @@ -49,9 +45,39 @@ public interface ViewMapper { ViewBriefDto viewToViewBriefDto(View data); + + default PreparedStatement viewToSelectAll(Connection connection, View view, Long page, Long size) throws QueryMalformedException { + log.debug("mapping view query, view.query={}", view.getQuery()); + final StringBuilder statement = new StringBuilder("SELECT "); + final int[] idx = new int[]{0}; + view.getColumns() + .forEach(c -> statement.append(idx[0]++ > 0 ? "," : "") + .append("`") + .append(c.getInternalName()) + .append("`")); + statement.append(" FROM `") + .append(view.getInternalName()) + .append("`"); + /* pagination */ + log.trace("pagination size/limit of {}", size); + statement.append(" LIMIT ") + .append(size); + log.trace("pagination page/offset of {}", page); + statement.append(" OFFSET ") + .append(page * size); + statement.append(";"); + try { + log.trace("mapped view query {} to prepared statement", statement); + return connection.prepareStatement(statement.toString()); + } catch (SQLException e) { + log.error("Failed to prepare statement {}: {}", statement, e.getMessage()); + throw new QueryMalformedException("Failed to prepare statement: " + e.getMessage(), e); + } + } + default PreparedStatement viewToRawDeleteViewQuery(Connection connection, View view) throws QueryMalformedException { - log.debug("mapping delete view query, view={}", view); + log.debug("mapping delete view query, view.name={}", view.getName()); final StringBuilder statement = new StringBuilder("DROP VIEW `") .append(nameToInternalName(view.getName())) .append("`;"); @@ -59,8 +85,8 @@ public interface ViewMapper { log.trace("mapped delete view {} to prepared statement", view.getName()); return connection.prepareStatement(statement.toString()); } catch (SQLException e) { - log.debug("Failed to prepare statement {}, reason: {}", statement, e.getMessage()); - throw new QueryMalformedException("Failed to prepare statement", e); + log.error("Failed to prepare statement {}: {}", statement, e.getMessage()); + throw new QueryMalformedException("Failed to prepare statement: " + e.getMessage(), e); } } @@ -77,8 +103,8 @@ public interface ViewMapper { log.trace("mapped create view {} to prepared statement {}", data.getName(), pstmt); return pstmt; } catch (SQLException e) { - log.error("Failed to prepare statement {}, reason: {}", statement, e.getMessage()); - throw new QueryMalformedException("Failed to prepare statement", e); + log.error("Failed to prepare statement {}: {}", statement, e.getMessage()); + throw new QueryMalformedException("Failed to prepare statement: " + e.getMessage(), e); } } diff --git a/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/IdentifierEndpoint.java b/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/IdentifierEndpoint.java index e1f2cf4a6497089254342b866caa2e87f0e40a62..d5bcd75118ae9d07aa007f0703062300fd5a0708 100644 --- a/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/IdentifierEndpoint.java +++ b/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/IdentifierEndpoint.java @@ -194,7 +194,7 @@ public class IdentifierEndpoint { throw new IdentifierRequestException("Failed to create subset identifier: only parameters database_id & query_id must be present"); } final Query query = storeService.findOne(data.getDatabaseId(), data.getQueryId(), principal); - final User user = userService.findByUsername(query.getCreatedBy()); + final User user = userService.find(query.getCreatedBy()); if (!endpointValidator.validateOnlyMineOrReadAccessOrHasRole(user.getId(), principal, access, "create-foreign-identifier")) { log.error("Failed to create subset identifier: insufficient access or role"); throw new IdentifierRequestException("Failed to create subset identifier: insufficient access or role"); diff --git a/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/StoreEndpoint.java b/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/StoreEndpoint.java index bb5d1287716605286853b81bcc90a3c36123f753..63ff9ed1b6302c9356288edc4a6902330798e483 100644 --- a/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/StoreEndpoint.java +++ b/dbrepo-metadata-service/rest-service/src/main/java/at/tuwien/endpoints/StoreEndpoint.java @@ -5,6 +5,7 @@ import at.tuwien.api.database.query.QueryDto; import at.tuwien.api.database.query.QueryPersistDto; import at.tuwien.api.error.ApiErrorDto; import at.tuwien.api.identifier.IdentifierBriefDto; +import at.tuwien.api.user.UserDto; import at.tuwien.entities.identifier.Identifier; import at.tuwien.exception.*; import at.tuwien.mapper.IdentifierMapper; @@ -38,9 +39,10 @@ import org.springframework.web.bind.annotation.*; import java.security.Principal; import java.util.List; -import java.util.Optional; import java.util.stream.Collectors; +import static org.apache.jena.sparql.vocabulary.VocabTestQuery.query; + @Log4j2 @RestController @RequestMapping("/api/database/{databaseId}/query") @@ -120,18 +122,26 @@ public class StoreEndpoint { endpointValidator.validateOnlyAccessOrPublic(databaseId, principal); /* find all from data database */ final List<Query> queries = storeService.findAll(databaseId, persisted, principal); - /* add identifiers from metadata database */ + /* add identifiers and creator from metadata database */ final List<IdentifierBriefDto> identifiers = identifierService.findAllSubsetIdentifiers() .stream() .map(identifierMapper::identifierToIdentifierBriefDto) .toList(); + final List<UserDto> users = userService.findAll() + .stream() + .map(userMapper::userToUserDto) + .toList(); final List<QueryBriefDto> dto = queries.stream() .map(queryMapper::queryToQueryBriefDto) .peek(q -> { - final List<IdentifierBriefDto> subsetIdentifiers = identifiers.stream() + q.setDatabaseId(databaseId); + users.stream() + .filter(u -> u.getId().equals(q.getCreatedBy())) + .findFirst() + .ifPresentOrElse(q::setCreator, () -> log.warn("Query creator with id {} not found in list of users", q.getCreatedBy())); + q.setIdentifiers(identifiers.stream() .filter(i -> i.getDatabaseId().equals(databaseId) && i.getQueryId().equals(q.getId())) - .toList(); - q.setIdentifiers(subsetIdentifiers); + .toList()); }) .collect(Collectors.toList()); log.trace("find queries resulted in queries {}", dto); @@ -186,7 +196,8 @@ public class StoreEndpoint { /* find */ final Query query = storeService.findOne(databaseId, queryId, principal); final QueryDto dto = queryMapper.queryToQueryDto(query); - dto.setCreator(userMapper.userToUserDto(userService.findByUsername(query.getCreatedBy()))); + dto.setDatabaseId(databaseId); + dto.setCreator(userMapper.userToUserDto(userService.find(query.getCreatedBy()))); final List<Identifier> identifiers = identifierService.findByDatabaseIdAndQueryId(databaseId, queryId); if (!identifiers.isEmpty()) { dto.setIdentifiers(identifiers.stream() @@ -239,22 +250,17 @@ public class StoreEndpoint { @NotNull @Valid @RequestBody QueryPersistDto data, @NotNull Principal principal) throws QueryStoreException, DatabaseNotFoundException, ImageNotSupportedException, - DatabaseConnectionException, UserNotFoundException, QueryNotFoundException, - QueryAlreadyPersistedException, NotAllowedException, AccessDeniedException { - log.debug("endpoint persist query, container, databaseId={}, queryId={}, {}", databaseId, queryId, PrincipalUtil.formatForDebug(principal)); + DatabaseConnectionException, UserNotFoundException, NotAllowedException, AccessDeniedException, + IdentifierAlreadyPublishedException { + log.debug("endpoint persist query, container, databaseId={}, queryId={}, data.persist={}, {}", databaseId, queryId, data.getPersist(), PrincipalUtil.formatForDebug(principal)); /* check */ endpointValidator.validateOnlyAccessOrPublic(databaseId, principal); - final Query check = storeService.findOne(databaseId, queryId, principal); - if (check.getIsPersisted()) { - log.error("Failed to persist, is already persisted"); - throw new QueryAlreadyPersistedException("Failed to persist"); - } /* has access */ accessService.find(databaseId, UserUtil.getId(principal)); /* persist */ final Query query = storeService.persist(databaseId, queryId, data); final QueryDto dto = queryMapper.queryToQueryDto(query); - dto.setCreator(userMapper.userToUserDto(userService.findByUsername(query.getCreatedBy()))); + dto.setCreator(userMapper.userToUserDto(userService.find(query.getCreatedBy()))); log.trace("persist query resulted in query {}", dto); return ResponseEntity.status(HttpStatus.ACCEPTED) .body(dto); diff --git a/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore.sql b/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore.sql index 35808b8eb7ff48df9256a42c2e8361be51fc7f3a..212e262742b7517b3b6e22d319609a0492e8e243 100644 --- a/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore.sql +++ b/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore.sql @@ -1,5 +1,5 @@ CREATE SEQUENCE `qs_queries_seq` NOCACHE; -CREATE TABLE `qs_queries` ( `id` bigint not null primary key default nextval(`qs_queries_seq`), `created` datetime not null default now(), `executed` datetime not null default now(), `created_by` varchar(255) not null, `query` text not null, `query_normalized` text not null, `is_persisted` boolean not null, `query_hash` varchar(255) not null, `result_hash` varchar(255), `result_number` bigint ); +CREATE TABLE `qs_queries` ( `id` bigint not null primary key default nextval(`qs_queries_seq`), `created` datetime not null default now(), `executed` datetime not null default now(), `created_by` varchar(36) not null, `query` text not null, `query_normalized` text not null, `is_persisted` boolean not null, `query_hash` varchar(255) not null, `result_hash` varchar(255), `result_number` bigint ); CREATE PROCEDURE hash_table(IN name VARCHAR(255), OUT hash VARCHAR(255), OUT count BIGINT) BEGIN DECLARE _sql TEXT; SELECT CONCAT('SELECT SHA2(GROUP_CONCAT(CONCAT_WS(\'\',', GROUP_CONCAT(CONCAT('`', column_name, '`') ORDER BY column_name), ') SEPARATOR \',\'), 256) AS hash, COUNT(*) AS count FROM `', name, '` INTO @hash, @count;') FROM `information_schema`.`columns` WHERE `table_schema` = DATABASE() AND `table_name` = name INTO _sql; PREPARE stmt FROM _sql; EXECUTE stmt; DEALLOCATE PREPARE stmt; SET hash = @hash; SET count = @count; END; CREATE PROCEDURE store_query(IN query TEXT, IN executed DATETIME, OUT queryId BIGINT) BEGIN DECLARE _queryhash varchar(255) DEFAULT SHA2(query, 256); DECLARE _username varchar(255) DEFAULT REGEXP_REPLACE(current_user(), '@.*', ''); DECLARE _query TEXT DEFAULT CONCAT('CREATE OR REPLACE TABLE _tmp AS (', query, ')'); PREPARE stmt FROM _query; EXECUTE stmt; DEALLOCATE PREPARE stmt; CALL hash_table('_tmp', @hash, @count); DROP TABLE IF EXISTS `_tmp`; IF @hash IS NULL THEN INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); ELSE INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); END IF; END; CREATE DEFINER = 'root' PROCEDURE _store_query(IN _username VARCHAR(255), IN query TEXT, IN executed DATETIME, OUT queryId BIGINT) BEGIN DECLARE _queryhash varchar(255) DEFAULT SHA2(query, 256); DECLARE _query TEXT DEFAULT CONCAT('CREATE OR REPLACE TABLE _tmp AS (', query, ')'); PREPARE stmt FROM _query; EXECUTE stmt; DEALLOCATE PREPARE stmt; CALL hash_table('_tmp', @hash, @count); DROP TABLE IF EXISTS `_tmp`; IF @hash IS NULL THEN INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); ELSE INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); END IF; END; \ No newline at end of file diff --git a/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore_manual.sql b/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore_manual.sql index 1b22fc218c5579e895b3be1fa328a1f2784ea8ce..037701fa15f2eb4f3520d0aeda79c5f8e92a60fe 100644 --- a/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore_manual.sql +++ b/dbrepo-metadata-service/rest-service/src/main/resources/init/querystore_manual.sql @@ -3,7 +3,7 @@ CREATE TABLE `qs_queries` ( `id` bigint not null primary key default nextval(`qs_queries_seq`), `created` datetime not null default now(), `executed` datetime not null default now(), - `created_by` varchar(255) not null, + `created_by` varchar(36) not null, `query` text not null, `query_normalized` text not null, `is_persisted` boolean not null, diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/config/MariaDbConfig.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/config/MariaDbConfig.java index 995a80b1f6ae80fb81ef603c643ae30864b3fa3e..fab3add8f6bc3b103fd9b59d5cbd162ae5903b80 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/config/MariaDbConfig.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/config/MariaDbConfig.java @@ -38,7 +38,7 @@ public class MariaDbConfig { * @return The generated or retrieved query id. * @throws SQLException The procedure did not succeed. */ - public static Long mockSystemQueryInsert(Database database, String query, String username, String password) + public static Long mockSystemQueryInsert(Database database, String query, String username, UUID userId, String password) throws SQLException { final String jdbc = "jdbc:mariadb://" + database.getContainer().getHost() + ":" + database.getContainer().getPort() + "/" + database.getInternalName(); log.trace("connect to database {}", jdbc); @@ -46,7 +46,7 @@ public class MariaDbConfig { final String call = "{call _store_query(?,?,?,?)}"; log.trace("prepare procedure '{}'", call); final CallableStatement statement = connection.prepareCall(call); - statement.setString(1, username); + statement.setString(1, String.valueOf(userId)); statement.setString(2, query); statement.setTimestamp(3, Timestamp.from(Instant.now())); statement.registerOutParameter(4, Types.BIGINT); @@ -163,7 +163,7 @@ public class MariaDbConfig { public static String getPrivileges(String hostname, Integer port, String database, String username, String password) throws Exception { - final String jdbc = "jdbc:mariadb://" + hostname + ":" + port + (database != null ? "/" + database : ""); + final String jdbc = "jdbc:mariadb://" + hostname + ":" + port + (database != null ? "/" + database : ""); log.trace("connect to database {}", jdbc); try (Connection connection = DriverManager.getConnection(jdbc, username, password)) { final String query = "SHOW GRANTS FOR `" + username + "`;"; @@ -218,16 +218,16 @@ public class MariaDbConfig { * @throws SQLException The procedure did not succeed. */ public static Long mockSystemQueryInsert(Database database, String query) throws SQLException { - return mockSystemQueryInsert(database, query, database.getContainer().getPrivilegedUsername(), database.getContainer().getPrivilegedPassword()); + return mockSystemQueryInsert(database, query, database.getContainer().getPrivilegedUsername(), UUID.randomUUID(), database.getContainer().getPrivilegedPassword()); } - public static void insertQueryStore(Database database, Query query, String username) throws SQLException { + public static void insertQueryStore(Database database, Query query, UUID userId) throws SQLException { final String jdbc = "jdbc:mariadb://" + database.getContainer().getHost() + ":" + database.getContainer().getPort() + "/" + database.getInternalName(); log.trace("connect to database {}", jdbc); try (Connection connection = DriverManager.getConnection(jdbc, database.getContainer().getPrivilegedUsername(), database.getContainer().getPrivilegedPassword())) { final PreparedStatement prepareStatement = connection.prepareStatement( "INSERT INTO qs_queries (created_by, query, query_normalized, is_persisted, query_hash, result_hash, result_number, created, executed) VALUES (?,?,?,?,?,?,?,?,?)"); - prepareStatement.setString(1, username); + prepareStatement.setString(1, String.valueOf(userId)); prepareStatement.setString(2, query.getQuery()); prepareStatement.setString(3, query.getQuery()); prepareStatement.setBoolean(4, query.getIsPersisted()); diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/IdentifierEndpointUnitTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/IdentifierEndpointUnitTest.java index 720736e5707c619310abafa227dd6992712ad7a3..26c180c5e21457d49e6108326e5d077a55c4d565 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/IdentifierEndpointUnitTest.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/IdentifierEndpointUnitTest.java @@ -296,7 +296,7 @@ public class IdentifierEndpointUnitTest extends BaseUnitTest { .when(accessService) .find(databaseId, userId); } - when(userService.findByUsername(USER_1_USERNAME)) + when(userService.find(USER_1_ID)) .thenReturn(USER_1); when(storeService.findOne(databaseId, data.getQueryId(), principal)) .thenReturn(QUERY_1); diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/StoreEndpointUnitTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/StoreEndpointUnitTest.java index 62e2b37824f3d28c731e648e645d6019d7cc5f85..5b1278f33c47ea1553e175b7e387881aa66a0d4d 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/StoreEndpointUnitTest.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/endpoints/StoreEndpointUnitTest.java @@ -13,6 +13,7 @@ import at.tuwien.querystore.Query; import at.tuwien.repository.mdb.UserRepository; import at.tuwien.service.AccessService; import at.tuwien.service.DatabaseService; +import at.tuwien.service.UserService; import at.tuwien.service.impl.StoreServiceImpl; import lombok.extern.log4j.Log4j2; import org.junit.jupiter.api.Test; @@ -53,6 +54,9 @@ public class StoreEndpointUnitTest extends BaseUnitTest { @MockBean private DatabaseService databaseService; + @MockBean + private UserService userService; + @MockBean private AccessService accessService; @@ -172,13 +176,18 @@ public class StoreEndpointUnitTest extends BaseUnitTest { KeycloakRemoteException, AccessDeniedException { /* mock */ - when(userRepository.findByUsername(USER_1_USERNAME)) - .thenReturn(Optional.of(USER_1)); + when(userService.find(USER_1_ID)) + .thenReturn(USER_1); /* test */ final QueryDto response = find_generic(DATABASE_1_ID, DATABASE_1, QUERY_1_ID, QUERY_1, USER_1_PRINCIPAL); + assertNotNull(response.getCreator()); + assertEquals(DATABASE_1_ID, response.getDatabaseId()); assertEquals(QUERY_1_ID, response.getId()); + assertNotNull(response.getIdentifiers()); + assertTrue(response.getIsPersisted()); assertEquals(QUERY_1_STATEMENT, response.getQuery()); + assertNotNull(response.getResultNumber()); } @Test @@ -220,8 +229,8 @@ public class StoreEndpointUnitTest extends BaseUnitTest { @Test @WithMockUser(username = USER_1_USERNAME, authorities = "persist-query") public void persist_ownRead_succeeds() throws UserNotFoundException, QueryStoreException, - NotAllowedException, DatabaseConnectionException, QueryAlreadyPersistedException, QueryNotFoundException, - DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, AccessDeniedException { + NotAllowedException, DatabaseConnectionException, QueryNotFoundException, DatabaseNotFoundException, + ImageNotSupportedException, AccessDeniedException, IdentifierAlreadyPublishedException { /* mock */ when(userRepository.findByUsername(USER_1_USERNAME)) @@ -236,8 +245,8 @@ public class StoreEndpointUnitTest extends BaseUnitTest { @Test @WithMockUser(username = USER_1_USERNAME, authorities = "persist-query") public void persist_ownWriteOwn_succeeds() throws UserNotFoundException, QueryStoreException, - NotAllowedException, DatabaseConnectionException, QueryAlreadyPersistedException, QueryNotFoundException, - DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, AccessDeniedException { + NotAllowedException, DatabaseConnectionException, QueryNotFoundException, DatabaseNotFoundException, + ImageNotSupportedException, AccessDeniedException, IdentifierAlreadyPublishedException { /* mock */ when(userRepository.findByUsername(USER_1_USERNAME)) @@ -252,8 +261,8 @@ public class StoreEndpointUnitTest extends BaseUnitTest { @Test @WithMockUser(username = USER_1_USERNAME, authorities = "persist-query") public void persist_ownWriteAll_succeeds() throws UserNotFoundException, QueryStoreException, - NotAllowedException, DatabaseConnectionException, QueryAlreadyPersistedException, QueryNotFoundException, - DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, AccessDeniedException { + NotAllowedException, DatabaseConnectionException, QueryNotFoundException, DatabaseNotFoundException, + ImageNotSupportedException, AccessDeniedException, IdentifierAlreadyPublishedException { /* mock */ when(userRepository.findByUsername(USER_1_USERNAME)) @@ -268,8 +277,8 @@ public class StoreEndpointUnitTest extends BaseUnitTest { @Test @WithMockUser(username = USER_2_USERNAME, authorities = "persist-query") public void persist_foreignWriteAll_succeeds() throws UserNotFoundException, QueryStoreException, - NotAllowedException, DatabaseConnectionException, QueryAlreadyPersistedException, QueryNotFoundException, - DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, AccessDeniedException { + NotAllowedException, DatabaseConnectionException, QueryNotFoundException, DatabaseNotFoundException, + ImageNotSupportedException, AccessDeniedException, IdentifierAlreadyPublishedException { /* mock */ when(userRepository.findByUsername(USER_1_USERNAME)) @@ -288,7 +297,7 @@ public class StoreEndpointUnitTest extends BaseUnitTest { UUID userId, Principal principal, DatabaseAccess access) throws DatabaseNotFoundException, UserNotFoundException, QueryStoreException, QueryNotFoundException, ImageNotSupportedException, NotAllowedException, DatabaseConnectionException, - QueryAlreadyPersistedException, KeycloakRemoteException, AccessDeniedException { + AccessDeniedException, IdentifierAlreadyPublishedException { final QueryPersistDto request = QueryPersistDto.builder() .persist(true) .build(); @@ -323,19 +332,26 @@ public class StoreEndpointUnitTest extends BaseUnitTest { AccessDeniedException { /* mock */ - doReturn(List.of(QUERY_1)).when(storeService) - .findAll(databaseId, true, principal); + when(storeService.findAll(databaseId, true, principal)) + .thenReturn(List.of(QUERY_1)); when(databaseService.find(databaseId)) .thenReturn(database); + when(userService.findAll()) + .thenReturn(List.of(USER_1)); /* test */ final ResponseEntity<List<QueryBriefDto>> response = storeEndpoint.findAll(databaseId, true, principal); assertEquals(HttpStatus.OK, response.getStatusCode()); assertNotNull(response.getBody()); assertEquals(1, response.getBody().size()); - final QueryBriefDto query = response.getBody().get(0); - assertEquals(QUERY_1_ID, query.getId()); - assertEquals(QUERY_1_STATEMENT, query.getQuery()); + final QueryBriefDto query0 = response.getBody().get(0); + assertNotNull(query0.getCreator()); + assertEquals(databaseId, query0.getDatabaseId()); + assertEquals(QUERY_1_ID, query0.getId()); + assertNotNull(query0.getIdentifiers()); + assertTrue(query0.getIsPersisted()); + assertEquals(QUERY_1_STATEMENT, query0.getQuery()); + assertNotNull(query0.getResultNumber()); } protected QueryDto find_generic(Long databaseId, Database database, Long queryId, Query query, Principal principal) diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/DatabaseServiceIntegrationTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/DatabaseServiceIntegrationTest.java index 2a9553b246c09526f2861faca7b58f255f64f371..91dc9c7d9ca5b69ba3f43999082b799912a1a965 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/DatabaseServiceIntegrationTest.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/DatabaseServiceIntegrationTest.java @@ -35,6 +35,7 @@ import java.sql.SQLException; import java.sql.SQLInvalidAuthorizationSpecException; import java.util.List; import java.util.Optional; +import java.util.UUID; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.ArgumentMatchers.any; @@ -274,7 +275,7 @@ public class DatabaseServiceIntegrationTest extends BaseUnitTest { .thenReturn("SELECT, CREATE, CREATE VIEW, CREATE ROUTINE, CREATE TEMPORARY TABLES, LOCK TABLES, INDEX, TRIGGER, INSERT, UPDATE, DELETE"); /* test */ - generic_system_insert(CONTAINER_1_PRIVILEGED_USERNAME, CONTAINER_1_PRIVILEGED_PASSWORD); + generic_system_insert(CONTAINER_1_PRIVILEGED_USERNAME, UUID.randomUUID(), CONTAINER_1_PRIVILEGED_PASSWORD); } @Test @@ -286,7 +287,7 @@ public class DatabaseServiceIntegrationTest extends BaseUnitTest { /* test */ assertThrows(SQLException.class, () -> { - generic_system_insert("junit1", "junit1"); + generic_system_insert(USER_1_USERNAME, USER_1_ID, USER_1_PASSWORD); }); } @@ -425,13 +426,13 @@ public class DatabaseServiceIntegrationTest extends BaseUnitTest { return response; } - protected void generic_system_insert(String username, String password) throws SQLException, QueryMalformedException { + protected void generic_system_insert(String username, UUID userId, String password) throws SQLException, QueryMalformedException { /* mock */ mariaDbConfig.grantUserPermissions(CONTAINER_1, DATABASE_3, USER_1_USERNAME); /* test */ - final Long queryId = MariaDbConfig.mockSystemQueryInsert(DATABASE_3, QUERY_4_STATEMENT, username, password); + final Long queryId = MariaDbConfig.mockSystemQueryInsert(DATABASE_3, QUERY_4_STATEMENT, username, userId, password); assertEquals(1L, queryId); } diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/QueryServiceIntegrationTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/QueryServiceIntegrationTest.java index eb5c5abb9d622bb1b657906e31a82262cdeecbf9..78298edfea16ae9b62e7f1133e6225d42b754670 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/QueryServiceIntegrationTest.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/QueryServiceIntegrationTest.java @@ -531,12 +531,12 @@ public class QueryServiceIntegrationTest extends BaseUnitTest { .resultNumber(0L) .created(QUERY_1_CREATED) .executed(QUERY_1_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .isPersisted(true) .build(); /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, query, USER_1_USERNAME); + MariaDbConfig.insertQueryStore(DATABASE_1, query, USER_1_ID); doNothing() .when(dataDbSidecarGateway) .exportFile(anyString(), anyInt(), anyString()); diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationModifyTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationModifyTest.java deleted file mode 100644 index 3df28fb663b3b681ef333436aae6e576a728db80..0000000000000000000000000000000000000000 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationModifyTest.java +++ /dev/null @@ -1,355 +0,0 @@ -package at.tuwien.service; - -import at.tuwien.BaseUnitTest; -import at.tuwien.annotations.MockAmqp; -import at.tuwien.annotations.MockOpensearch; -import at.tuwien.api.database.query.ExecuteStatementDto; -import at.tuwien.api.database.query.QueryPersistDto; -import at.tuwien.api.database.query.QueryResultDto; -import at.tuwien.config.MariaDbConfig; -import at.tuwien.config.MariaDbContainerConfig; -import at.tuwien.exception.*; -import at.tuwien.querystore.Query; -import at.tuwien.repository.mdb.*; -import lombok.extern.log4j.Log4j2; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.boot.autoconfigure.EnableAutoConfiguration; -import org.springframework.boot.autoconfigure.amqp.RabbitAutoConfiguration; -import org.springframework.boot.test.context.SpringBootTest; -import org.springframework.test.annotation.DirtiesContext; -import org.springframework.test.context.junit.jupiter.SpringExtension; -import org.testcontainers.containers.MariaDBContainer; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; - -import java.sql.SQLException; -import java.time.Instant; -import java.time.temporal.ChronoUnit; -import java.util.List; -import java.util.Map; - -import static org.junit.jupiter.api.Assertions.*; - -@Log4j2 -@Testcontainers -@DirtiesContext(classMode = DirtiesContext.ClassMode.BEFORE_EACH_TEST_METHOD) -@EnableAutoConfiguration(exclude = RabbitAutoConfiguration.class) -@ExtendWith(SpringExtension.class) -@SpringBootTest -@MockAmqp -@MockOpensearch -public class StoreServiceIntegrationModifyTest extends BaseUnitTest { - - @Autowired - private DatabaseRepository databaseRepository; - - @Autowired - private ImageRepository imageRepository; - - @Autowired - private ContainerRepository containerRepository; - - @Autowired - private LicenseRepository licenseRepository; - - @Autowired - private StoreService storeService; - - @Autowired - private QueryService queryService; - - @Autowired - private UserRepository userRepository; - - @Container - private static MariaDBContainer<?> mariaDBContainer = MariaDbContainerConfig.getContainer(); - - @BeforeEach - public void beforeEach() throws InterruptedException, SQLException { - TABLE_1.setColumns(TABLE_1_COLUMNS); - TABLE_2.setColumns(TABLE_2_COLUMNS); - TABLE_3.setColumns(TABLE_3_COLUMNS); - TABLE_4.setColumns(TABLE_4_COLUMNS); - /* metadata database */ - imageRepository.save(IMAGE_1); - licenseRepository.save(LICENSE_1); - userRepository.saveAll(List.of(USER_1, USER_2, USER_3, USER_4, USER_5)); - containerRepository.save(CONTAINER_1); - databaseRepository.save(DATABASE_1); - MariaDbConfig.dropAllDatabases(CONTAINER_1); - MariaDbConfig.createInitDatabase(CONTAINER_1, DATABASE_1); - } - - @Test - public void insert_same_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, SQLException, KeycloakRemoteException, - AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_2_STATEMENT) - .build(); - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - - /* test */ - final Query response = storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); - log.debug("found queries in query store: {}", MariaDbConfig.selectQuery(DATABASE_1, - "SELECT `query_normalized`, `query_hash`, `result_hash` FROM `qs_queries`", "query_normalized", "query_hash", "result_hash")); - assertEquals(QUERY_1_ID, response.getId()) /* no new query inserted */; - } - - @Test - public void execute_different_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, - QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, - QueryNotFoundException { - final ExecuteStatementDto mock = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .build(); - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_2_STATEMENT) - .build(); - - /* mock */ - queryService.execute(DATABASE_1_ID, mock, USER_1_PRINCIPAL, 0L, 10L, null, null); - - /* test */ - final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - assertEquals(2L, response.getId()) /* new query inserted */; - } - - @Test - public void execute_same_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, QueryMalformedException, - ColumnParseException, KeycloakRemoteException, AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .build(); - - /* mock */ - queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - - - /* test */ - final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - assertEquals(1L, response.getId()) /* no new query inserted */; - } - - @Test - public void execute_notPersisted_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, - QueryMalformedException, ColumnParseException, SQLException, KeycloakRemoteException, - AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .build(); - - - /* test */ - final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - assertEquals(1L, response.getId()) /* no new query inserted */; - assertFalse(Boolean.parseBoolean(MariaDbConfig.listQueryStore(DATABASE_1).get(0).get("is_persisted").toString())); - } - - @Test - public void execute_emptyResult_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, - QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, - QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement("SELECT `id`, `date`, `location`, `mintemp`, `rainfall` FROM `weather_aus` WHERE `location` = 'Vienna'") - .build(); - - /* test */ - final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - assertEquals(1L, response.getId()) /* new query inserted */; - } - - @Test - public void execute_emptyResultTwice_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, - QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, - QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement("SELECT `id`, `date`, `location`, `mintemp`, `rainfall` FROM `weather_aus` WHERE `location` = 'Vienna'") - .build(); - - /* mock */ - queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - - /* test */ - final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - assertEquals(1L, response.getId()) /* no new query inserted */; - } - - @Test - public void execute_dataChangeSameQuery_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, - QueryMalformedException, ColumnParseException, SQLException, KeycloakRemoteException, - AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .build(); - - /* mock */ - queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - MariaDbConfig.execute(DATABASE_1, "INSERT INTO weather_aus (id, `date`, location, mintemp, rainfall) VALUES (4, '2008-12-04', 'Albury', 12.9, 0.2)"); - - /* test */ - storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); - } - - @Test - public void execute_semicolon_fails() { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT + ";") - .build(); - - /* test */ - assertThrows(QueryMalformedException.class, () -> { - queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); - }); - } - - @Test - public void persist_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, SQLException { - final QueryPersistDto request = QueryPersistDto.builder() - .persist(true) - .build(); - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - - /* test */ - final Query response = storeService.persist(DATABASE_1_ID, QUERY_1_ID, request); - assertTrue(response.getIsPersisted()); - } - - @Test - public void persist_alreadyPersisted_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, SQLException { - final QueryPersistDto request = QueryPersistDto.builder() - .persist(true) - .build(); - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_2, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_3, USER_1_USERNAME); - - /* test */ - final Query response = storeService.persist(DATABASE_1_ID, QUERY_3_ID, request); - assertTrue(response.getIsPersisted()); - } - - @Test - public void persist_unPersist_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, SQLException { - final QueryPersistDto request = QueryPersistDto.builder() - .persist(false) - .build(); - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_2, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_3, USER_1_USERNAME); - - /* test */ - final Query response = storeService.persist(DATABASE_1_ID, QUERY_3_ID, request); - assertFalse(response.getIsPersisted()); - } - - @Test - public void insert_timestamp_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, - AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .timestamp(Instant.now().plus(1, ChronoUnit.SECONDS)) - .build(); - - /* test */ - storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); - } - - @Test - public void insert_anonymous_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, - AccessDeniedException, QueryNotFoundException { - final ExecuteStatementDto request = ExecuteStatementDto.builder() - .statement(QUERY_1_STATEMENT) - .build(); - - /* test */ - storeService.insert(DATABASE_1_ID, request, null); - } - - @Test - public void findOne_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - QueryNotFoundException, DatabaseNotFoundException, ImageNotSupportedException, SQLException { - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - - /* test */ - storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); - } - - @Test - public void findOne_notFound_succeeds() { - - /* test */ - assertThrows(QueryNotFoundException.class, () -> { - storeService.findOne(DATABASE_1_ID, 9999L, USER_1_PRINCIPAL); - }); - } - - @Test - public void findOne_notFound_fails() { - - /* test */ - assertThrows(QueryNotFoundException.class, () -> { - storeService.findOne(DATABASE_1_ID, 9999L, USER_1_PRINCIPAL); - }); - } - - @Test - public void deleteStaleQueries_succeeds() throws QueryStoreException, ImageNotSupportedException, SQLException { - final Query queryOk = Query.builder() - .id(QUERY_1_ID) - .query(QUERY_1_STATEMENT) - .queryHash(QUERY_1_QUERY_HASH) - .resultHash(QUERY_1_RESULT_HASH) - .resultNumber(QUERY_1_RESULT_NUMBER) - .created(Instant.now().minus(1, ChronoUnit.HOURS)) - .createdBy(USER_1_USERNAME) - .isPersisted(QUERY_1_PERSISTED) - .executed(Instant.now().minus(1, ChronoUnit.HOURS)) - .build(); - final Query queryDelete = Query.builder() - .id(QUERY_2_ID) - .query(QUERY_2_STATEMENT) - .queryHash(QUERY_2_QUERY_HASH) - .resultHash(QUERY_2_RESULT_HASH) - .resultNumber(QUERY_2_RESULT_NUMBER) - .created(Instant.now().minus(25, ChronoUnit.HOURS)) - .createdBy(USER_2_USERNAME) - .isPersisted(QUERY_2_PERSISTED) - .executed(Instant.now().minus(25, ChronoUnit.HOURS)) - .build(); - - /* mock */ - MariaDbConfig.insertQueryStore(DATABASE_1, queryOk, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, queryDelete, USER_1_USERNAME); - - /* test */ - storeService.deleteStaleQueries(); - final List<Map<String, Object>> response = MariaDbConfig.listQueryStore(DATABASE_1); - assertEquals(1, response.size()); - } - -} diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationReadTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationReadTest.java deleted file mode 100644 index aae9e492a7e888f433adc08c592d00c401c15592..0000000000000000000000000000000000000000 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationReadTest.java +++ /dev/null @@ -1,127 +0,0 @@ -package at.tuwien.service; - -import at.tuwien.BaseUnitTest; -import at.tuwien.annotations.MockAmqp; -import at.tuwien.annotations.MockOpensearch; -import at.tuwien.config.MariaDbConfig; -import at.tuwien.config.MariaDbContainerConfig; -import at.tuwien.exception.*; -import at.tuwien.querystore.Query; -import at.tuwien.repository.mdb.DatabaseRepository; -import lombok.extern.log4j.Log4j2; -import org.apache.http.auth.BasicUserPrincipal; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.boot.autoconfigure.EnableAutoConfiguration; -import org.springframework.boot.autoconfigure.amqp.RabbitAutoConfiguration; -import org.springframework.boot.test.context.SpringBootTest; -import org.springframework.boot.test.mock.mockito.MockBean; -import org.springframework.test.context.junit.jupiter.SpringExtension; -import org.testcontainers.containers.MariaDBContainer; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; - -import java.security.Principal; -import java.sql.SQLException; -import java.util.List; -import java.util.Optional; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.mockito.Mockito.when; - -@Log4j2 -@Testcontainers -@ExtendWith(SpringExtension.class) -@EnableAutoConfiguration(exclude= RabbitAutoConfiguration.class) -@SpringBootTest -@MockAmqp -@MockOpensearch -public class StoreServiceIntegrationReadTest extends BaseUnitTest { - - @MockBean - private DatabaseRepository databaseRepository; - - @Autowired - private StoreService storeService; - - @Container - private static MariaDBContainer<?> mariaDBContainer = MariaDbContainerConfig.getContainer(); - - @BeforeAll - public static void beforeAll() throws SQLException { - /* insert query */ - MariaDbConfig.dropAllDatabases(CONTAINER_1); - MariaDbConfig.createInitDatabase(CONTAINER_1, DATABASE_1); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - } - - @Test - public void findAll_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, TableMalformedException, ContainerNotFoundException { - - /* mock */ - when(databaseRepository.findById(DATABASE_1_ID)) - .thenReturn(Optional.of(DATABASE_1)); - - /* test */ - final List<Query> queries = storeService.findAll(DATABASE_1_ID, null, USER_1_PRINCIPAL); - assertEquals(1, queries.size()); - } - - @Test - public void findAll_filterPersisted_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, TableMalformedException, ContainerNotFoundException { - - /* mock */ - when(databaseRepository.findById(DATABASE_1_ID)) - .thenReturn(Optional.of(DATABASE_1)); - - /* test */ - final List<Query> queries = storeService.findAll(DATABASE_1_ID, true, USER_1_PRINCIPAL); - assertEquals(0, queries.size()); - } - - @Test - public void findOne_succeeds() throws UserNotFoundException, QueryStoreException, - DatabaseConnectionException, QueryNotFoundException, DatabaseNotFoundException, ImageNotSupportedException { - - /* mock */ - when(databaseRepository.findById(DATABASE_1_ID)) - .thenReturn(Optional.of(DATABASE_1)); - - /* test */ - storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); - } - - @Test - public void findOne_notFound_succeeds() { - - /* mock */ - when(databaseRepository.findById(DATABASE_1_ID)) - .thenReturn(Optional.of(DATABASE_1)); - - /* test */ - assertThrows(QueryNotFoundException.class, () -> { - storeService.findOne(DATABASE_1_ID, 9999L, USER_1_PRINCIPAL); - }); - } - - @Test - public void findOne_notFound_fails() { - final Principal principal = new BasicUserPrincipal(USER_1_USERNAME); - - /* mock */ - when(databaseRepository.findById(DATABASE_1_ID)) - .thenReturn(Optional.of(DATABASE_1)); - - /* test */ - assertThrows(QueryNotFoundException.class, () -> { - storeService.findOne(DATABASE_1_ID, QUERY_2_ID, principal); - }); - } - -} diff --git a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationTest.java b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationTest.java index 21da3ad82178f5ae9f750106610b083f782db0ef..e12e7b1b6ba945c72d68f18e5e83e764b94a7956 100644 --- a/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationTest.java +++ b/dbrepo-metadata-service/rest-service/src/test/java/at/tuwien/service/StoreServiceIntegrationTest.java @@ -3,14 +3,19 @@ package at.tuwien.service; import at.tuwien.BaseUnitTest; import at.tuwien.annotations.MockAmqp; import at.tuwien.annotations.MockOpensearch; +import at.tuwien.api.database.query.ExecuteStatementDto; import at.tuwien.api.database.query.QueryPersistDto; +import at.tuwien.api.database.query.QueryResultDto; import at.tuwien.config.MariaDbConfig; import at.tuwien.config.MariaDbContainerConfig; import at.tuwien.exception.*; import at.tuwien.querystore.Query; import at.tuwien.repository.mdb.*; +import com.github.jsonldjava.utils.Obj; import lombok.extern.log4j.Log4j2; +import org.apache.http.auth.BasicUserPrincipal; import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.springframework.beans.factory.annotation.Autowired; @@ -21,8 +26,12 @@ import org.testcontainers.containers.MariaDBContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; +import java.security.Principal; import java.sql.SQLException; +import java.time.Instant; +import java.time.temporal.ChronoUnit; import java.util.List; +import java.util.Map; import static org.junit.jupiter.api.Assertions.*; @@ -47,6 +56,9 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { @Autowired private LicenseRepository licenseRepository; + @Autowired + private QueryService queryService; + @Autowired private UserRepository userRepository; @@ -64,7 +76,7 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { TABLE_4.setColumns(TABLE_4_COLUMNS); /* metadata database */ imageRepository.save(IMAGE_1); - userRepository.save(USER_1); + userRepository.saveAll(List.of(USER_1, USER_2, USER_3, USER_4, USER_5)); licenseRepository.save(LICENSE_1); containerRepository.save(CONTAINER_1); DATABASE_1.setAccesses(List.of()); @@ -72,8 +84,35 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { /* data stuff */ MariaDbConfig.dropAllDatabases(CONTAINER_1); MariaDbConfig.createInitDatabase(CONTAINER_1, DATABASE_1); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_USERNAME); - MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_3, USER_1_USERNAME); + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_ID); + } + + @Test + public void findAll_filterPersisted_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, + DatabaseNotFoundException, ImageNotSupportedException, TableMalformedException, ContainerNotFoundException { + + /* test */ + final List<Query> queries = storeService.findAll(DATABASE_1_ID, true, USER_1_PRINCIPAL); + assertEquals(1, queries.size()); + } + + @Test + public void findOne_notFound_succeeds() { + + /* test */ + assertThrows(QueryNotFoundException.class, () -> { + storeService.findOne(DATABASE_1_ID, 9999L, USER_1_PRINCIPAL); + }); + } + + @Test + public void findOne_notFound_fails() { + final Principal principal = new BasicUserPrincipal(USER_1_USERNAME); + + /* test */ + assertThrows(QueryNotFoundException.class, () -> { + storeService.findOne(DATABASE_1_ID, 9999L, principal); + }); } @Test @@ -83,7 +122,7 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { /* test */ final List<Query> response = storeService.findAll(DATABASE_1_ID, null, USER_1_PRINCIPAL); - assertEquals(2, response.size()); + assertEquals(1, response.size()); } @Test @@ -103,16 +142,7 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { /* test */ final List<Query> response = storeService.findAll(DATABASE_1_ID, false, USER_1_PRINCIPAL); - assertEquals(1, response.size()); - } - - @Test - public void findOne_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, QueryNotFoundException { - - /* test */ - final Query response = storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); - assertNotNull(response); + assertEquals(0, response.size()); } @Test @@ -126,14 +156,15 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { @Test public void persist_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, QueryNotFoundException { + DatabaseNotFoundException, ImageNotSupportedException, QueryNotFoundException, + IdentifierAlreadyPublishedException { final QueryPersistDto request = QueryPersistDto.builder() .persist(true) .build(); /* precondition */ - final Query query3 = storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); - assertFalse(query3.getIsPersisted()); + final Query query1 = storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); + assertTrue(query1.getIsPersisted()); /* test */ final Query response = storeService.persist(DATABASE_1_ID, QUERY_1_ID, request); @@ -142,19 +173,249 @@ public class StoreServiceIntegrationTest extends BaseUnitTest { } @Test - public void persist_unchanged_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, - DatabaseNotFoundException, ImageNotSupportedException, QueryNotFoundException { + public void persist_unPersistUnchanged_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, QueryNotFoundException, + IdentifierAlreadyPublishedException, SQLException { + final Query query = Query.builder() + .id(2L) + .query(QUERY_3_STATEMENT) + .queryHash(QUERY_3_QUERY_HASH) + .resultHash(QUERY_3_RESULT_HASH) + .created(QUERY_3_CREATED) + .executed(QUERY_3_EXECUTION) + .createdBy(USER_1_ID) + .resultNumber(QUERY_3_RESULT_NUMBER) + .isPersisted(false) // <<<<<<< + .build(); final QueryPersistDto request = QueryPersistDto.builder() .persist(false) // <<<<<<< .build(); + /* mock */ + MariaDbConfig.insertQueryStore(DATABASE_1, query, USER_1_ID); + /* precondition */ - final Query query3 = storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); - assertFalse(query3.getIsPersisted()); + final Query query2 = storeService.findOne(DATABASE_1_ID, 2L, USER_1_PRINCIPAL); + assertFalse(query2.getIsPersisted()); /* test */ - final Query response = storeService.persist(DATABASE_1_ID, QUERY_1_ID, request); + final Query response = storeService.persist(DATABASE_1_ID, 2L, request); assertNotNull(response); assertFalse(response.getIsPersisted()); } + + @Test + public void persist_unPersistIdentifierAlreadyAttached_fails () { + final QueryPersistDto request = QueryPersistDto.builder() + .persist(false) + .build(); + + /* test */ + assertThrows(IdentifierAlreadyPublishedException.class, () -> { + storeService.persist(DATABASE_1_ID, QUERY_1_ID, request); + }); + } + + @Test + public void insert_same_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, + DatabaseNotFoundException, ImageNotSupportedException, SQLException, KeycloakRemoteException, + AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_2_STATEMENT) + .build(); + + /* test */ + final Query response = storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); + log.debug("found queries in query store: {}", MariaDbConfig.selectQuery(DATABASE_1, + "SELECT `query_normalized`, `query_hash`, `result_hash` FROM `qs_queries`", "query_normalized", "query_hash", "result_hash")); + assertEquals(QUERY_1_ID, response.getId()) /* no new query inserted */; + } + + @Test + public void execute_differentResult_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, + QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, + QueryNotFoundException, SQLException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .build(); + + /* mock */ + MariaDbConfig.execute(DATABASE_1, "INSERT INTO `weather_aus` (`id`, `date`) VALUES (4, '2024-01-12');"); + + /* test */ + final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + assertEquals(2L, response.getId()) /* new query inserted */; + } + + @Test + @Disabled("not testable") + public void execute_same_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, + TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, QueryMalformedException, + ColumnParseException, KeycloakRemoteException, AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .build(); + + /* test */ + final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + assertEquals(1L, response.getId()) /* no new query inserted */; + } + + @Test + @Disabled("not testable") + public void execute_notPersisted_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, + QueryMalformedException, ColumnParseException, SQLException, KeycloakRemoteException, + AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .build(); + + /* test */ + final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + assertEquals(1L, response.getId()) /* no new query inserted */; + assertFalse(Boolean.parseBoolean(MariaDbConfig.listQueryStore(DATABASE_1).get(0).get("is_persisted").toString())); + } + + @Test + public void execute_emptyResult_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, + QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, + QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement("SELECT `id`, `date`, `location`, `mintemp`, `rainfall` FROM `weather_aus` WHERE `location` = 'Wien'") + .build(); + + /* test */ + final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + assertEquals(2L, response.getId()) /* new query inserted */; + } + + @Test + public void execute_emptyResultTwice_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, + QueryMalformedException, ColumnParseException, KeycloakRemoteException, AccessDeniedException, + QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement("SELECT `location`, `mintemp` FROM `weather_aus` WHERE `rainfall` < 0") + .build(); + + /* mock */ + queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + + /* test */ + final QueryResultDto response = queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + assertEquals(2L, response.getId()) /* no new query inserted */; + } + + @Test + public void execute_dataChangeSameQuery_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, TableMalformedException, DatabaseNotFoundException, ImageNotSupportedException, + QueryMalformedException, ColumnParseException, SQLException, KeycloakRemoteException, + AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .build(); + + /* mock */ + queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + MariaDbConfig.execute(DATABASE_1, "INSERT INTO weather_aus (id, `date`, location, mintemp, rainfall) VALUES (4, '2008-12-04', 'Albury', 12.9, 0.2)"); + + /* test */ + storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); + } + + @Test + public void execute_semicolon_fails() { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT + ";") + .build(); + + /* test */ + assertThrows(QueryMalformedException.class, () -> { + queryService.execute(DATABASE_1_ID, request, USER_1_PRINCIPAL, 0L, 10L, null, null); + }); + } + + @Test + public void persist_alreadyPersisted_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, SQLException, + IdentifierAlreadyPublishedException { + final QueryPersistDto request = QueryPersistDto.builder() + .persist(true) + .build(); + + /* mock */ + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_ID); + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_2, USER_1_ID); + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_3, USER_1_ID); + + /* test */ + final Query response = storeService.persist(DATABASE_1_ID, QUERY_3_ID, request); + assertTrue(response.getIsPersisted()); + } + + @Test + public void persist_unPersist_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, SQLException, + IdentifierAlreadyPublishedException { + final QueryPersistDto request = QueryPersistDto.builder() + .persist(false) + .build(); + + /* mock */ + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_ID); + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_2, USER_1_ID); + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_3, USER_1_ID); + + /* test */ + final Query response = storeService.persist(DATABASE_1_ID, QUERY_3_ID, request); + assertFalse(response.getIsPersisted()); + } + + @Test + public void insert_timestamp_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, + AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .timestamp(Instant.now().plus(1, ChronoUnit.SECONDS)) + .build(); + + /* test */ + storeService.insert(DATABASE_1_ID, request, USER_1_PRINCIPAL); + } + + @Test + public void insert_anonymous_succeeds() throws UserNotFoundException, QueryStoreException, + DatabaseConnectionException, DatabaseNotFoundException, ImageNotSupportedException, KeycloakRemoteException, + AccessDeniedException, QueryNotFoundException { + final ExecuteStatementDto request = ExecuteStatementDto.builder() + .statement(QUERY_1_STATEMENT) + .build(); + + /* test */ + storeService.insert(DATABASE_1_ID, request, null); + } + + @Test + public void findOne_succeeds() throws UserNotFoundException, QueryStoreException, DatabaseConnectionException, + QueryNotFoundException, DatabaseNotFoundException, ImageNotSupportedException, SQLException { + + /* mock */ + MariaDbConfig.insertQueryStore(DATABASE_1, QUERY_1, USER_1_ID); + + /* test */ + storeService.findOne(DATABASE_1_ID, QUERY_1_ID, USER_1_PRINCIPAL); + } + + @Test + public void deleteStaleQueries_succeeds() throws QueryStoreException, ImageNotSupportedException, SQLException { + + /* test */ + storeService.deleteStaleQueries(); + final List<Map<String, Object>> response = MariaDbConfig.listQueryStore(DATABASE_1); + assertEquals(1, response.size()); + } } diff --git a/dbrepo-metadata-service/rest-service/src/test/resources/init/querystore.sql b/dbrepo-metadata-service/rest-service/src/test/resources/init/querystore.sql index 35808b8eb7ff48df9256a42c2e8361be51fc7f3a..212e262742b7517b3b6e22d319609a0492e8e243 100644 --- a/dbrepo-metadata-service/rest-service/src/test/resources/init/querystore.sql +++ b/dbrepo-metadata-service/rest-service/src/test/resources/init/querystore.sql @@ -1,5 +1,5 @@ CREATE SEQUENCE `qs_queries_seq` NOCACHE; -CREATE TABLE `qs_queries` ( `id` bigint not null primary key default nextval(`qs_queries_seq`), `created` datetime not null default now(), `executed` datetime not null default now(), `created_by` varchar(255) not null, `query` text not null, `query_normalized` text not null, `is_persisted` boolean not null, `query_hash` varchar(255) not null, `result_hash` varchar(255), `result_number` bigint ); +CREATE TABLE `qs_queries` ( `id` bigint not null primary key default nextval(`qs_queries_seq`), `created` datetime not null default now(), `executed` datetime not null default now(), `created_by` varchar(36) not null, `query` text not null, `query_normalized` text not null, `is_persisted` boolean not null, `query_hash` varchar(255) not null, `result_hash` varchar(255), `result_number` bigint ); CREATE PROCEDURE hash_table(IN name VARCHAR(255), OUT hash VARCHAR(255), OUT count BIGINT) BEGIN DECLARE _sql TEXT; SELECT CONCAT('SELECT SHA2(GROUP_CONCAT(CONCAT_WS(\'\',', GROUP_CONCAT(CONCAT('`', column_name, '`') ORDER BY column_name), ') SEPARATOR \',\'), 256) AS hash, COUNT(*) AS count FROM `', name, '` INTO @hash, @count;') FROM `information_schema`.`columns` WHERE `table_schema` = DATABASE() AND `table_name` = name INTO _sql; PREPARE stmt FROM _sql; EXECUTE stmt; DEALLOCATE PREPARE stmt; SET hash = @hash; SET count = @count; END; CREATE PROCEDURE store_query(IN query TEXT, IN executed DATETIME, OUT queryId BIGINT) BEGIN DECLARE _queryhash varchar(255) DEFAULT SHA2(query, 256); DECLARE _username varchar(255) DEFAULT REGEXP_REPLACE(current_user(), '@.*', ''); DECLARE _query TEXT DEFAULT CONCAT('CREATE OR REPLACE TABLE _tmp AS (', query, ')'); PREPARE stmt FROM _query; EXECUTE stmt; DEALLOCATE PREPARE stmt; CALL hash_table('_tmp', @hash, @count); DROP TABLE IF EXISTS `_tmp`; IF @hash IS NULL THEN INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); ELSE INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); END IF; END; CREATE DEFINER = 'root' PROCEDURE _store_query(IN _username VARCHAR(255), IN query TEXT, IN executed DATETIME, OUT queryId BIGINT) BEGIN DECLARE _queryhash varchar(255) DEFAULT SHA2(query, 256); DECLARE _query TEXT DEFAULT CONCAT('CREATE OR REPLACE TABLE _tmp AS (', query, ')'); PREPARE stmt FROM _query; EXECUTE stmt; DEALLOCATE PREPARE stmt; CALL hash_table('_tmp', @hash, @count); DROP TABLE IF EXISTS `_tmp`; IF @hash IS NULL THEN INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` IS NULL); ELSE INSERT INTO `qs_queries` (`created_by`, `query`, `query_normalized`, `is_persisted`, `query_hash`, `result_hash`, `result_number`, `executed`) SELECT _username, query, query, false, _queryhash, @hash, @count, executed WHERE NOT EXISTS (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); SET queryId = (SELECT `id` FROM `qs_queries` WHERE `query_hash` = _queryhash AND `result_hash` = @hash); END IF; END; \ No newline at end of file diff --git a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/StoreService.java b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/StoreService.java index d62b959713a273b8ab81334f96e8a7830ca29c9f..5d3c007802c67fdafac7abe9f0111236cf82766d 100644 --- a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/StoreService.java +++ b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/StoreService.java @@ -71,7 +71,7 @@ public interface StoreService { * @throws QueryStoreException The query store raised some error. */ Query persist(Long databaseId, Long queryId, QueryPersistDto data) throws DatabaseNotFoundException, - ImageNotSupportedException, DatabaseConnectionException, QueryStoreException, UserNotFoundException; + ImageNotSupportedException, DatabaseConnectionException, QueryStoreException, UserNotFoundException, IdentifierAlreadyPublishedException; /** * Deletes the stale queries that have not been persisted within 24 hozrs. diff --git a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/MariaDbServiceImpl.java b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/MariaDbServiceImpl.java index 0fa65b87a8f9ace956c9a0bef9c49f4511ac3d67..edae7bfa0e7f8199da7575aad41e9d7a28de3fcc 100644 --- a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/MariaDbServiceImpl.java +++ b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/MariaDbServiceImpl.java @@ -223,7 +223,7 @@ public class MariaDbServiceImpl extends HibernateConnector implements DatabaseSe final Database database = findById(databaseId); final List<Table> diffTables; final List<View> diffViews; - final ComboPooledDataSource dataSource = getPrivilegedDataSource(database.getContainer().getImage(), database.getContainer()); + final ComboPooledDataSource dataSource = getPrivilegedDataSource(database.getContainer().getImage(), database.getContainer(), database); try { final Connection connection = dataSource.getConnection(); final PreparedStatement preparedStatement0 = databaseMapper.databaseToDatabaseMetadata(connection, database); @@ -269,6 +269,8 @@ public class MariaDbServiceImpl extends HibernateConnector implements DatabaseSe } final PreparedStatement preparedStatement2 = queryMapper.databaseToDatabaseConstraintMetadata(connection, table.getDatabase().getInternalName(), table.getInternalName()); table.setConstraints(resultSetTableToObtainedConstraintsMetadata(preparedStatement2.executeQuery(), table)); + final PreparedStatement preparedStatement3 = tableMapper.tableToCreateHistoryViewRawQuery(connection, table); + preparedStatement3.executeUpdate(); database.getTables().add(table); } } catch (SQLException e) { diff --git a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/QueryServiceImpl.java b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/QueryServiceImpl.java index e8f64ab7203dfd7228dee7691f7a9369add43b31..a27845c75c3061f9d7c1a34ebc8716747857014f 100644 --- a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/QueryServiceImpl.java +++ b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/QueryServiceImpl.java @@ -16,6 +16,7 @@ import at.tuwien.entities.database.table.columns.TableColumn; import at.tuwien.exception.*; import at.tuwien.gateway.DataDbSidecarGateway; import at.tuwien.mapper.QueryMapper; +import at.tuwien.mapper.ViewMapper; import at.tuwien.querystore.Query; import at.tuwien.service.DatabaseService; import at.tuwien.service.QueryService; @@ -51,6 +52,7 @@ import java.util.List; public class QueryServiceImpl extends HibernateConnector implements QueryService { private final S3Config s3Config; + private final ViewMapper viewMapper; private final MinioClient minioClient; private final QueryMapper queryMapper; private final StoreService storeService; @@ -59,10 +61,11 @@ public class QueryServiceImpl extends HibernateConnector implements QueryService private final DataDbSidecarGateway dataDbSidecarGateway; @Autowired - public QueryServiceImpl(S3Config s3Config, MinioClient minioClient, QueryMapper queryMapper, + public QueryServiceImpl(S3Config s3Config, ViewMapper viewMapper, MinioClient minioClient, QueryMapper queryMapper, TableService tableService, DatabaseService databaseService, StoreService storeService, DataDbSidecarGateway dataDbSidecarGateway) { this.s3Config = s3Config; + this.viewMapper = viewMapper; this.minioClient = minioClient; this.queryMapper = queryMapper; this.tableService = tableService; @@ -198,8 +201,22 @@ public class QueryServiceImpl extends HibernateConnector implements QueryService @Transactional(readOnly = true) public QueryResultDto viewFindAll(Long databaseId, View view, Long page, Long size, Principal principal) throws DatabaseNotFoundException, QueryMalformedException, TableMalformedException { + /* find */ + final Database database = databaseService.find(databaseId); /* run query */ - return executeNonPersistent(databaseId, view.getQuery(), view.getColumns()); + final ComboPooledDataSource dataSource = getPrivilegedDataSource(database.getContainer().getImage(), + database.getContainer(), database); + try { + final Connection connection = dataSource.getConnection(); + final PreparedStatement preparedStatement = viewMapper.viewToSelectAll(connection, view, page, size); + final ResultSet resultSet = preparedStatement.executeQuery(); + return queryMapper.resultListToQueryResultDto(view.getColumns(), resultSet); + } catch (SQLException e) { + log.error("Failed to map object: {}", e.getMessage()); + throw new TableMalformedException("Failed to map object", e); + } finally { + dataSource.close(); + } } @Override diff --git a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/StoreServiceImpl.java b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/StoreServiceImpl.java index 743529122a43023d200b971e981e53992fc63471..af6d6ee60aa20966006c24a8e0675657c3f25dfd 100644 --- a/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/StoreServiceImpl.java +++ b/dbrepo-metadata-service/services/src/main/java/at/tuwien/service/impl/StoreServiceImpl.java @@ -3,10 +3,12 @@ package at.tuwien.service.impl; import at.tuwien.api.database.query.ExecuteStatementDto; import at.tuwien.api.database.query.QueryPersistDto; import at.tuwien.entities.database.Database; +import at.tuwien.entities.identifier.Identifier; import at.tuwien.entities.user.User; import at.tuwien.exception.*; import at.tuwien.mapper.StoreMapper; import at.tuwien.querystore.Query; +import at.tuwien.repository.mdb.IdentifierRepository; import at.tuwien.service.DatabaseService; import at.tuwien.service.StoreService; import at.tuwien.service.UserService; @@ -28,12 +30,15 @@ public class StoreServiceImpl extends HibernateConnector implements StoreService private final StoreMapper storeMapper; private final UserService userService; private final DatabaseService databaseService; + private final IdentifierRepository identifierRepository; @Autowired - public StoreServiceImpl(StoreMapper storeMapper, UserService userService, DatabaseService databaseService) { + public StoreServiceImpl(StoreMapper storeMapper, UserService userService, DatabaseService databaseService, + IdentifierRepository identifierRepository) { this.storeMapper = storeMapper; this.userService = userService; this.databaseService = databaseService; + this.identifierRepository = identifierRepository; } @Override @@ -146,7 +151,12 @@ public class StoreServiceImpl extends HibernateConnector implements StoreService @Override @Transactional public Query persist(Long databaseId, Long queryId, QueryPersistDto data) throws DatabaseNotFoundException, - ImageNotSupportedException, QueryStoreException { + ImageNotSupportedException, QueryStoreException, IdentifierAlreadyPublishedException { + /* check */ + if (!data.getPersist() && !identifierRepository.findByDatabaseIdAndQueryId(databaseId, queryId).isEmpty()) { + log.error("Failed to de-persist query with id {} in database with id {}: identifier already attached", queryId, databaseId); + throw new IdentifierAlreadyPublishedException("Failed to de-persist query with id " + queryId + " in database with id " + databaseId + ": identifier already attached"); + } /* find */ final Database database = databaseService.find(databaseId); if (!database.getContainer().getImage().getName().equals("mariadb")) { diff --git a/dbrepo-metadata-service/test/src/main/java/at/tuwien/test/BaseTest.java b/dbrepo-metadata-service/test/src/main/java/at/tuwien/test/BaseTest.java index 42a148ba49423e5486766a38f4efb48d6efcc923..51daf587fae7c530227d5f6f5da79b7ab7be3f0e 100644 --- a/dbrepo-metadata-service/test/src/main/java/at/tuwien/test/BaseTest.java +++ b/dbrepo-metadata-service/test/src/main/java/at/tuwien/test/BaseTest.java @@ -302,7 +302,6 @@ public abstract class BaseTest { public final static UserDto USER_1_DTO = UserDto.builder() .id(USER_1_ID) .username(USER_1_USERNAME) - .email(USER_1_EMAIL) .firstname(USER_1_FIRSTNAME) .lastname(USER_1_LASTNAME) .attributes(USER_1_ATTRIBUTES_DTO) @@ -409,7 +408,6 @@ public abstract class BaseTest { public final static UserDto USER_2_DTO = UserDto.builder() .id(USER_2_ID) .username(USER_2_USERNAME) - .email(USER_2_EMAIL) .firstname(USER_2_FIRSTNAME) .lastname(USER_2_LASTNAME) .name(USER_2_NAME) @@ -495,7 +493,6 @@ public abstract class BaseTest { public final static UserDto USER_3_DTO = UserDto.builder() .id(USER_3_ID) .username(USER_3_USERNAME) - .email(USER_3_EMAIL) .firstname(USER_3_FIRSTNAME) .lastname(USER_3_LASTNAME) .name(USER_3_NAME) @@ -572,7 +569,6 @@ public abstract class BaseTest { public final static UserDto USER_4_DTO = UserDto.builder() .id(USER_4_ID) .username(USER_4_USERNAME) - .email(USER_4_EMAIL) .firstname(USER_4_FIRSTNAME) .lastname(USER_4_LASTNAME) .attributes(USER_4_ATTRIBUTES_DTO) @@ -615,7 +611,6 @@ public abstract class BaseTest { public final static UserDto USER_5_DTO = UserDto.builder() .id(USER_5_ID) .username(USER_5_USERNAME) - .email(USER_5_EMAIL) .firstname(USER_5_FIRSTNAME) .lastname(USER_5_LASTNAME) .build(); @@ -661,7 +656,6 @@ public abstract class BaseTest { public final static UserDto USER_6_DTO = UserDto.builder() .id(USER_6_ID) .username(USER_6_USERNAME) - .email(USER_6_EMAIL) .firstname(USER_6_FIRSTNAME) .lastname(USER_6_LASTNAME) .build(); @@ -2424,7 +2418,7 @@ public abstract class BaseTest { public final static String QUERY_1_RESULT_HASH = "8358c8ade4849d2094ab5bb29127afdae57e6bb5acb1db7af603813d406c467a"; public final static Instant QUERY_1_CREATED = Instant.ofEpochSecond(1677648377L); public final static Instant QUERY_1_EXECUTION = Instant.now(); - public final static Boolean QUERY_1_PERSISTED = false; + public final static Boolean QUERY_1_PERSISTED = true; public final static Query QUERY_1 = Query.builder() .id(QUERY_1_ID) @@ -2434,14 +2428,13 @@ public abstract class BaseTest { .resultNumber(QUERY_1_RESULT_NUMBER) .created(QUERY_1_CREATED) .executed(QUERY_1_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .isPersisted(QUERY_1_PERSISTED) .build(); public final static QueryDto QUERY_1_DTO = QueryDto.builder() .id(QUERY_1_ID) - .cid(QUERY_1_CONTAINER_ID) - .dbid(QUERY_1_DATABASE_ID) + .databaseId(QUERY_1_DATABASE_ID) .query(QUERY_1_STATEMENT) .queryHash(QUERY_1_QUERY_HASH) .resultHash(QUERY_1_RESULT_HASH) @@ -2453,8 +2446,7 @@ public abstract class BaseTest { public final static QueryBriefDto QUERY_1_BRIEF_DTO = QueryBriefDto.builder() .id(QUERY_1_ID) - .cid(QUERY_1_CONTAINER_ID) - .dbid(QUERY_1_DATABASE_ID) + .databaseId(QUERY_1_DATABASE_ID) .query(QUERY_1_STATEMENT) .queryHash(QUERY_1_QUERY_HASH) .resultHash(QUERY_1_RESULT_HASH) @@ -2484,20 +2476,20 @@ public abstract class BaseTest { .resultNumber(QUERY_2_RESULT_NUMBER) .created(QUERY_2_CREATED) .executed(QUERY_2_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .isPersisted(QUERY_2_PERSISTED) .build(); public final static QueryDto QUERY_2_DTO = QueryDto.builder() .id(QUERY_2_ID) - .cid(QUERY_2_CONTAINER_ID) - .dbid(QUERY_2_DATABASE_ID) + .databaseId(QUERY_2_DATABASE_ID) .query(QUERY_2_STATEMENT) .queryNormalized(QUERY_2_STATEMENT) .resultNumber(QUERY_2_RESULT_NUMBER) .resultHash(QUERY_2_RESULT_HASH) .lastModified(QUERY_2_LAST_MODIFIED) .created(QUERY_2_CREATED) + .createdBy(USER_1_ID) .queryHash(QUERY_2_QUERY_HASH) .execution(QUERY_2_EXECUTION) .build(); @@ -2521,21 +2513,21 @@ public abstract class BaseTest { .resultHash(QUERY_3_RESULT_HASH) .created(QUERY_3_CREATED) .executed(QUERY_3_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .resultNumber(QUERY_3_RESULT_NUMBER) .isPersisted(QUERY_3_PERSISTED) .build(); public final static QueryDto QUERY_3_DTO = QueryDto.builder() .id(QUERY_3_ID) - .cid(QUERY_3_CONTAINER_ID) - .dbid(QUERY_3_DATABASE_ID) + .databaseId(QUERY_3_DATABASE_ID) .query(QUERY_3_STATEMENT) .queryNormalized(QUERY_3_STATEMENT) .resultNumber(QUERY_3_RESULT_NUMBER) .resultHash(QUERY_3_RESULT_HASH) .lastModified(QUERY_3_LAST_MODIFIED) .created(QUERY_3_CREATED) + .createdBy(USER_1_ID) .queryHash(QUERY_3_QUERY_HASH) .execution(QUERY_3_EXECUTION) .build(); @@ -2562,7 +2554,7 @@ public abstract class BaseTest { .executed(QUERY_4_EXECUTION) .isPersisted(QUERY_4_PERSISTED) .resultNumber(QUERY_4_RESULT_NUMBER) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .build(); public final static List<Map<String, Object>> QUERY_4_RESULT_RESULT = List.of( new HashMap<>() {{ @@ -2593,14 +2585,14 @@ public abstract class BaseTest { public final static QueryDto QUERY_4_DTO = QueryDto.builder() .id(QUERY_4_ID) - .cid(QUERY_4_CONTAINER_ID) - .dbid(QUERY_4_DATABASE_ID) + .databaseId(QUERY_4_DATABASE_ID) .query(QUERY_4_STATEMENT) .queryNormalized(QUERY_4_STATEMENT) .resultNumber(QUERY_4_RESULT_NUMBER) .resultHash(QUERY_4_RESULT_HASH) .lastModified(QUERY_4_LAST_MODIFIED) .created(QUERY_4_CREATED) + .createdBy(USER_1_ID) .queryHash(QUERY_4_QUERY_HASH) .execution(QUERY_4_EXECUTION) .build(); @@ -2624,14 +2616,13 @@ public abstract class BaseTest { .resultHash(QUERY_5_RESULT_HASH) .created(QUERY_5_CREATED) .executed(QUERY_5_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .isPersisted(QUERY_5_PERSISTED) .build(); public final static QueryDto QUERY_5_DTO = QueryDto.builder() .id(QUERY_5_ID) - .cid(QUERY_5_CONTAINER_ID) - .dbid(QUERY_5_DATABASE_ID) + .databaseId(QUERY_5_DATABASE_ID) .query(QUERY_5_STATEMENT) .queryNormalized(QUERY_5_STATEMENT) .resultNumber(QUERY_5_RESULT_NUMBER) @@ -2661,20 +2652,20 @@ public abstract class BaseTest { .resultHash(QUERY_6_RESULT_HASH) .created(QUERY_6_CREATED) .executed(QUERY_6_EXECUTION) - .createdBy(USER_1_USERNAME) + .createdBy(USER_1_ID) .isPersisted(QUERY_6_PERSISTED) .build(); public final static QueryDto QUERY_6_DTO = QueryDto.builder() .id(QUERY_6_ID) - .cid(QUERY_6_CONTAINER_ID) - .dbid(QUERY_6_DATABASE_ID) + .databaseId(QUERY_6_DATABASE_ID) .query(QUERY_6_STATEMENT) .queryNormalized(QUERY_6_STATEMENT) .resultNumber(QUERY_6_RESULT_NUMBER) .resultHash(QUERY_6_RESULT_HASH) .lastModified(QUERY_6_LAST_MODIFIED) .created(QUERY_6_CREATED) + .createdBy(USER_1_ID) .queryHash(QUERY_6_QUERY_HASH) .execution(QUERY_6_EXECUTION) .build(); @@ -6092,6 +6083,7 @@ public abstract class BaseTest { public final static String IDENTIFIER_2_QUERY_HASH = QUERY_1_QUERY_HASH; public final static String IDENTIFIER_2_RESULT_HASH = QUERY_1_RESULT_HASH; public final static String IDENTIFIER_2_QUERY = QUERY_1_STATEMENT; + public final static Long IDENTIFIER_2_QUERY_ID = QUERY_1_ID; public final static String IDENTIFIER_2_NORMALIZED = QUERY_1_STATEMENT; public final static Long IDENTIFIER_2_RESULT_NUMBER = QUERY_1_RESULT_NUMBER; public final static String IDENTIFIER_2_PUBLISHER = "Swedish Government"; @@ -6100,6 +6092,7 @@ public abstract class BaseTest { public final static Identifier IDENTIFIER_2 = Identifier.builder() .id(IDENTIFIER_2_ID) + .queryId(IDENTIFIER_2_QUERY_ID) .databaseId(IDENTIFIER_2_DATABASE_ID) .descriptions(List.of()) .titles(List.of()) @@ -6125,6 +6118,7 @@ public abstract class BaseTest { public final static IdentifierDto IDENTIFIER_2_DTO = IdentifierDto.builder() .id(IDENTIFIER_2_ID) + .queryId(IDENTIFIER_2_QUERY_ID) .databaseId(IDENTIFIER_2_DATABASE_ID) .descriptions(List.of()) .titles(List.of()) @@ -6149,6 +6143,7 @@ public abstract class BaseTest { public final static IdentifierSaveDto IDENTIFIER_2_DTO_REQUEST = IdentifierSaveDto.builder() .databaseId(IDENTIFIER_2_DATABASE_ID) + .queryId(IDENTIFIER_2_QUERY_ID) .descriptions(List.of()) .titles(List.of()) .relatedIdentifiers(List.of()) diff --git a/dbrepo-ui/api/identifier.service.js b/dbrepo-ui/api/identifier.service.js index a214c6d9e7c6a5590bee4ff8fd0be3618ceaf5ce..336137dc2e55d9d729c0d05035f420bfbda5ac31 100644 --- a/dbrepo-ui/api/identifier.service.js +++ b/dbrepo-ui/api/identifier.service.js @@ -17,10 +17,6 @@ class IdentifierService { }) } - find (id) { - return this.findAccept(id, 'application/json') - } - retrieve (url) { return new Promise((resolve, reject) => { if (url === null) { diff --git a/dbrepo-ui/api/query.service.js b/dbrepo-ui/api/query.service.js index 5029b562ae80013b9f8e7a6a6ca9800183eac679..c2b456ddc37331c1f878c0ffdbcc288ffa03b6c9 100644 --- a/dbrepo-ui/api/query.service.js +++ b/dbrepo-ui/api/query.service.js @@ -30,9 +30,9 @@ class QueryService { }) } - persist (databaseId, queryId) { + persist (databaseId, queryId, persist) { return new Promise((resolve, reject) => { - api.put(`/api/database/${databaseId}/query/${queryId}`, { persist: true }, { headers: { Accept: 'application/json' } }) + api.put(`/api/database/${databaseId}/query/${queryId}`, { persist }, { headers: { Accept: 'application/json' } }) .then((response) => { const query = response.data console.debug('response query', query) diff --git a/dbrepo-ui/components/UserBadge.vue b/dbrepo-ui/components/UserBadge.vue new file mode 100644 index 0000000000000000000000000000000000000000..130b47636d54045ce9150f919c9e0037eccfdddc --- /dev/null +++ b/dbrepo-ui/components/UserBadge.vue @@ -0,0 +1,63 @@ +<template> + <p class="mb-0"> + <OrcidIcon v-if="hasOrcid" :orcid="orcid" /> + <span v-if="isSelf"> + <v-badge inline content="you" color="code">{{ creatorName }}</v-badge> + </span> + <span v-else v-text="creatorName" /> + </p> +</template> +<script> +import OrcidIcon from '@/components/icons/OrcidIcon.vue' +import UserMapper from '@/api/user.mapper' + +export default { + components: { + OrcidIcon + }, + props: { + user: { + type: Object, + default () { + return { + id: null, + attributes: { + orcid: null + } + } + } + }, + otherUser: { + type: Object, + default () { + return { + id: null + } + } + } + }, + computed: { + hasOrcid () { + if (!this.user || !this.user.attributes || !this.user.attributes.orcid) { + return false + } + return true + }, + orcid () { + if (!this.hasOrcid) { + return false + } + return this.user.attributes.orcid + }, + creatorName () { + return UserMapper.userToFullName(this.user) + }, + isSelf () { + if (!this.otherUser) { + return false + } + return this.user.id === this.otherUser.id + } + } +} +</script> diff --git a/dbrepo-ui/components/query/Results.vue b/dbrepo-ui/components/query/Results.vue index 3249f99e18ba8a3889862865a95024fd48f16ff5..56856893753bf2b3a942dfcbbf5dd5d7938dd1a4 100644 --- a/dbrepo-ui/components/query/Results.vue +++ b/dbrepo-ui/components/query/Results.vue @@ -1,11 +1,14 @@ <template> - <v-data-table - flat - :headers="headers" - :items="result.rows" - :loading="loading > 0" - :options.sync="options" - :server-items-length="total" /> + <div> + <v-data-table + flat + :headers="headers" + :items="result.rows" + :loading="loading > 0" + :options.sync="options" + :footer-props="footerProps" + :server-items-length="total" /> + </div> </template> <script> @@ -36,6 +39,10 @@ export default { page: 1, itemsPerPage: 10 }, + footerProps: { + showFirstLastPage: true, + itemsPerPageOptions: [10, 25, 50, 100] + }, total: -1 } }, @@ -71,7 +78,7 @@ export default { statement: sql, timestamp } - QueryService.execute(this.$route.params.database_id, payload, 0, this.options.itemsPerPage) + QueryService.execute(this.$route.params.database_id, payload, this.options.page - 1, this.options.itemsPerPage) .then((result) => { this.mapResults(result) parent.resultId = result.id @@ -86,7 +93,7 @@ export default { } this.loading++ if (this.type === 'query') { - QueryService.reExecuteQuery(this.$route.params.database_id, id, 0, this.options.itemsPerPage) + QueryService.reExecuteQuery(this.$route.params.database_id, id, this.options.page - 1, this.options.itemsPerPage) .then((result) => { this.mapResults(result) this.id = id @@ -95,7 +102,7 @@ export default { this.loading-- }) } else { - QueryService.reExecuteView(this.$route.params.database_id, id, 0, this.options.itemsPerPage) + QueryService.reExecuteView(this.$route.params.database_id, id, this.options.page - 1, this.options.itemsPerPage) .then((result) => { this.mapResults(result) this.id = id diff --git a/dbrepo-ui/components/query/SubsetToolbar.vue b/dbrepo-ui/components/query/SubsetToolbar.vue index 0aa8590a0058faa05238e712f57ec7b18394f97b..3f6f76a030bdc54c7af28edd82283793c406a80b 100644 --- a/dbrepo-ui/components/query/SubsetToolbar.vue +++ b/dbrepo-ui/components/query/SubsetToolbar.vue @@ -12,6 +12,9 @@ <v-btn v-if="canPersistQuery" :loading="loadingSave" class="mb-1" @click.stop="save"> <v-icon left>mdi-content-save-outline</v-icon> Save </v-btn> + <v-btn v-if="canForgetQuery" :loading="loadingSave" class="mb-1" @click.stop="forget"> + <v-icon left>mdi-trash-can-outline</v-icon> Soft-Delete + </v-btn> <v-btn v-if="result_visibility && subset && subset.result_number" class="mb-1" :loading="downloadLoading" @click.stop="downloadSubset"> <v-icon left>mdi-download</v-icon> Data .csv </v-btn> @@ -51,7 +54,6 @@ export default { loading: false, loadingSave: false, downloadLoading: false, - identifier: null, subset: null } }, @@ -75,7 +77,21 @@ export default { if (!this.database || !this.database.subsets || this.database.subsets.length === 0) { return [] } - return this.database.subsets + return this.database.subsets.filter(s => s.query_id === Number(this.$route.params.query_id)) + }, + identifier () { + /* mount pid */ + if (this.pid) { + const filter = this.identifiers.filter(i => i.id === Number(this.pid)) + if (filter.length > 0) { + const identifier = filter[0] + console.debug('identifier set according to route pid', identifier) + return identifier + } + } + const identifier = this.identifiers[0] + console.debug('defaulted to latest identifier', identifier) + return identifier }, canPersistQuery () { if (this.loading || !this.subset || this.subset.is_persisted) { @@ -83,6 +99,15 @@ export default { } return UserUtils.hasReadAccess(this.access) }, + canForgetQuery () { + if (this.loading || !this.subset || !this.subset.is_persisted) { + return false + } + if (this.subset.identifiers.length > 0) { + return false + } + return UserUtils.hasReadAccess(this.access) + }, executionUTC () { if (!this.subset) { return null @@ -90,7 +115,7 @@ export default { return formatTimestampUTCLabel(this.subset.created) }, result_visibility () { - if (!this.database || this.database.is_public === null) { + if (!this.database) { return false } if (this.database.is_public) { @@ -112,17 +137,6 @@ export default { } }, mounted () { - /* mount pid */ - if (this.pid) { - const filter = this.identifiers.filter(i => i.id === Number(this.pid)) - if (filter.length > 0) { - this.identifier = filter[0] - console.debug('identifier set according to route pid', this.identifier) - return - } - } - this.identifier = this.identifiers[0] - console.debug('defaulted to latest identifier', this.identifier) /* load subset metadata */ if (!this.subset) { this.loadSubset() @@ -131,7 +145,20 @@ export default { methods: { save () { this.loadingSave = true - QueryService.persist(this.$route.params.database_id, this.$route.params.query_id) + QueryService.persist(this.$route.params.database_id, this.$route.params.query_id, true) + .then((subset) => { + this.subset = subset + }) + .catch(() => { + this.loadingSave = false + }) + .finally(() => { + this.loadingSave = false + }) + }, + forget () { + this.loadingSave = true + QueryService.persist(this.$route.params.database_id, this.$route.params.query_id, false) .then((subset) => { this.subset = subset }) diff --git a/dbrepo-ui/components/view/ViewToolbar.vue b/dbrepo-ui/components/view/ViewToolbar.vue index b210ef696e03a3e855ea71849d014a34c7b8142d..f4e4efbf783f168a32a372d47a581a1caca1dc3a 100644 --- a/dbrepo-ui/components/view/ViewToolbar.vue +++ b/dbrepo-ui/components/view/ViewToolbar.vue @@ -6,15 +6,13 @@ <v-icon left>mdi-arrow-left</v-icon> </v-btn> </v-toolbar-title> - <v-toolbar-title> - <span v-if="view.name">{{ view.name }}</span> - </v-toolbar-title> + <v-toolbar-title v-text="title" /> <v-spacer /> <v-toolbar-title> - <v-btn v-if="canDeleteView" :loading="loadingDelete" color="error" class="mb-1" @click="deleteView"> + <v-btn v-if="canDeleteView" class="mb-1" :loading="loadingDelete" color="error" @click="deleteView"> <v-icon left>mdi-delete</v-icon> Delete </v-btn> - <v-btn v-if="canCreatePid" class="mb-1 ml-2" color="primary" :to="`/database/${$route.params.database_id}/view/${$route.params.view_id}/persist`"> + <v-btn v-if="canCreatePid" class="mb-1" color="primary" :to="`/database/${$route.params.database_id}/view/${$route.params.view_id}/persist`"> <v-icon left>mdi-content-save-outline</v-icon> Get PID </v-btn> </v-toolbar-title> @@ -33,6 +31,7 @@ <script> import UserUtils from '@/api/user.utils' import DatabaseService from '@/api/database.service' +import IdentifierMapper from '@/api/identifier.mapper' export default { components: { @@ -45,6 +44,9 @@ export default { } }, computed: { + pid () { + return this.$route.query.pid + }, database () { return this.$store.state.database }, @@ -74,6 +76,35 @@ export default { }, roles () { return this.$store.state.roles + }, + identifiers () { + if (!this.view) { + return [] + } + return this.view.identifiers.filter(s => s.view_id === Number(this.$route.params.view_id)) + }, + identifier () { + /* mount pid */ + if (this.pid) { + const filter = this.identifiers.filter(i => i.id === Number(this.pid)) + if (filter.length > 0) { + const identifier = filter[0] + console.debug('identifier set according to route pid', identifier) + return identifier + } + } + const identifier = this.identifiers[0] + console.debug('defaulted to latest identifier', identifier) + return identifier + }, + title () { + if (!this.view) { + return null + } + if (!this.identifier) { + return this.view.name + } + return IdentifierMapper.identifierPreferEnglishTitle(this.identifier) } }, methods: { diff --git a/dbrepo-ui/pages/database/_database_id/info.vue b/dbrepo-ui/pages/database/_database_id/info.vue index 09bd91759efa1561e238817cfe317c253c7c7100..1d9eb4d45730af3c9277876331f2d3ff373f6f78 100644 --- a/dbrepo-ui/pages/database/_database_id/info.vue +++ b/dbrepo-ui/pages/database/_database_id/info.vue @@ -19,7 +19,7 @@ <v-list-item-title> Database Visibility </v-list-item-title> - <v-list-item-content v-if="!loading" v-text="`${database.is_public ? 'Public' : 'Private'}`" /> + <v-list-item-content v-if="!loading && database" v-text="`${database.is_public ? 'Public' : 'Private'}`" /> <v-list-item-title class="mt-2"> Database Internal Name </v-list-item-title> @@ -27,11 +27,8 @@ <v-list-item-title class="mt-2"> Database Owner </v-list-item-title> - <v-list-item-content v-if="!loading"> - <p class="mb-0"> - <OrcidIcon v-if="database.owner.attributes.orcid" :orcid="database.owner.attributes.orcid" /> - <span v-text="owner" /> - </p> + <v-list-item-content> + <UserBadge v-if="database" :user="database.owner" :other-user="user" /> </v-list-item-content> <v-list-item-title class="mt-2"> Database Creation @@ -57,11 +54,8 @@ <v-list-item-title v-if="contact" class="mt-2"> Database Contact </v-list-item-title> - <v-list-item-content v-if="contact"> - <p class="mb-0"> - <OrcidIcon v-if="database.contact.attributes.orcid" :orcid="database.contact.attributes.orcid" /> - <span v-text="contact" /> - </p> + <v-list-item-content> + <UserBadge v-if="database.contact" :user="database.contact" :other-user="user" /> </v-list-item-content> </v-list-item-content> </v-list-item> @@ -108,14 +102,14 @@ import { formatTimestampUTCLabel } from '@/utils' import DatabaseMapper from '@/api/database.mapper' import Summary from '@/components/identifier/Summary' import Select from '@/components/identifier/Select' -import OrcidIcon from '@/components/icons/OrcidIcon.vue' +import UserBadge from '@/components/UserBadge.vue' export default { components: { - OrcidIcon, DatabaseToolbar, Summary, - Select + Select, + UserBadge }, data () { return { diff --git a/dbrepo-ui/pages/database/_database_id/query/_query_id/data.vue b/dbrepo-ui/pages/database/_database_id/query/_query_id/data.vue index edcfa2bca50dd1f333047a08eafefa590dd341fd..9e1f8e3c5da3e88b8f6f11f951f8fd2020d8a828 100644 --- a/dbrepo-ui/pages/database/_database_id/query/_query_id/data.vue +++ b/dbrepo-ui/pages/database/_database_id/query/_query_id/data.vue @@ -20,11 +20,13 @@ <script> import QueryResults from '@/components/query/Results' +import SubsetToolbar from '@/components/query/SubsetToolbar' import QueryService from '@/api/query.service' import { formatTimestampUTCLabel } from '@/utils' export default { components: { + SubsetToolbar, QueryResults }, data () { diff --git a/dbrepo-ui/pages/database/_database_id/query/_query_id/info.vue b/dbrepo-ui/pages/database/_database_id/query/_query_id/info.vue index 04cd96275f29511ad331898b45eb3eaeacf92817..69ab9fe5e6ce6df3881898bcc7a07b252bd2e52e 100644 --- a/dbrepo-ui/pages/database/_database_id/query/_query_id/info.vue +++ b/dbrepo-ui/pages/database/_database_id/query/_query_id/info.vue @@ -14,40 +14,53 @@ <v-list dense> <v-list-item> <v-list-item-content> + <v-list-item-title> + Subset Visibility + </v-list-item-title> + <v-list-item-content> + <v-skeleton-loader v-if="!database" type="text" class="skeleton-small" /> + <span v-if="database" v-text="database.is_public ? 'Public' : 'Private'" /> + </v-list-item-content> + <v-list-item-title> + Subset Creator + </v-list-item-title> + <v-list-item-content> + <v-skeleton-loader v-if="!subset" type="text" class="skeleton-small" /> + <UserBadge v-if="subset" :user="subset.creator" :other-user="user" /> + </v-list-item-content> <v-list-item-title> Subset Query </v-list-item-title> <v-list-item-content> - <pre>{{ subset.query }}</pre> + <v-skeleton-loader v-if="!subset" type="text" /> + <pre v-if="subset">{{ subset.query }}</pre> </v-list-item-content> <v-list-item-title class="mt-2"> Subset Query Hash </v-list-item-title> <v-list-item-content> - <pre>sha256:{{ subset.query_hash }}</pre> + <v-skeleton-loader v-if="!subset" type="text" /> + <pre v-if="subset">sha256:{{ subset.query_hash }}</pre> </v-list-item-content> <v-list-item-title class="mt-2"> Subset Creation </v-list-item-title> <v-list-item-content> + <v-skeleton-loader v-if="!executionUTC" type="text" class="skeleton-small" /> <span v-if="executionUTC">{{ executionUTC }}</span> </v-list-item-content> - <v-list-item-title> - Subset Visibility - </v-list-item-title> - <v-list-item-content> - {{ database.is_public ? 'Public' : 'Private' }} - </v-list-item-content> <v-list-item-title class="mt-2"> Subset Hash </v-list-item-title> <v-list-item-content> + <v-skeleton-loader v-if="!subset" type="text" /> <pre v-if="subset">{{ result_hash }}</pre> </v-list-item-content> <v-list-item-title class="mt-2"> Subset Count </v-list-item-title> <v-list-item-content> + <v-skeleton-loader v-if="!subset" type="text" class="skeleton-xsmall" /> <span v-if="subset">{{ subset.result_number }}</span> </v-list-item-content> </v-list-item-content> @@ -88,13 +101,16 @@ import SubsetToolbar from '@/components/query/SubsetToolbar.vue' import QueryService from '@/api/query.service' import Select from '@/components/identifier/Select' import { formatTimestampUTCLabel } from '@/utils' +import UserMapper from '@/api/user.mapper' +import UserBadge from '@/components/UserBadge.vue' export default { name: 'QueryShow', components: { Select, Summary, - SubsetToolbar + SubsetToolbar, + UserBadge }, data () { return { @@ -112,10 +128,7 @@ export default { downloadLoading: false, error: false, promises: [], - subset: { - query: null, - query_hash: null - } + subset: null } }, computed: { @@ -135,7 +148,7 @@ export default { if (!this.database || !this.database.subsets || this.database.subsets.length === 0) { return [] } - return this.database.subsets + return this.database.subsets.filter(s => s.query_id === Number(this.$route.params.query_id)) }, hasIdentifier () { return this.identifiers.length > 0 @@ -194,6 +207,15 @@ export default { .finally(() => { this.loadingSubset = false }) + }, + isCreator (subset) { + if (!this.user) { + return false + } + return subset.creator.id === this.user.id + }, + formatCreator (creator) { + return UserMapper.userToFullName(creator) } } } diff --git a/dbrepo-ui/pages/database/_database_id/table/_table_id/data.vue b/dbrepo-ui/pages/database/_database_id/table/_table_id/data.vue index 15c007a5453498fa60e67820535c5b2825e05527..c663a38827699e4d6bb262d9e4db8979ae6e2298 100644 --- a/dbrepo-ui/pages/database/_database_id/table/_table_id/data.vue +++ b/dbrepo-ui/pages/database/_database_id/table/_table_id/data.vue @@ -66,7 +66,8 @@ export default { editTupleDialog: false, total: -1, footerProps: { - 'items-per-page-options': [10, 20, 30, 40, 50] + showFirstLastPage: true, + itemsPerPageOptions: [10, 25, 50, 100] }, downloadLoading: false, dateMenu: false, diff --git a/dbrepo-ui/pages/database/_database_id/table/_table_id/info.vue b/dbrepo-ui/pages/database/_database_id/table/_table_id/info.vue index 4dc128c2a10895063848a9ed2e7dd66d43b1544b..d4b0620414afd29196250e01d1c8b150fa1b7b67 100644 --- a/dbrepo-ui/pages/database/_database_id/table/_table_id/info.vue +++ b/dbrepo-ui/pages/database/_database_id/table/_table_id/info.vue @@ -28,7 +28,8 @@ Table Owner </v-list-item-title> <v-list-item-content> - <span v-if="table && table.creator">{{ formatCreator(table.owner) }} <span v-if="is_owner(table)" style="flex:none;"> (you)</span></span> + <v-skeleton-loader v-if="!table" type="text" class="skeleton-small" /> + <UserBadge v-if="table" :user="table.creator" :other-user="user" /> </v-list-item-content> <v-list-item-title v-if="table && table.created" class="mt-2"> Table Creation @@ -106,16 +107,17 @@ </template> <script> import TableToolbar from '@/components/table/TableToolbar.vue' -import UserMapper from '@/api/user.mapper' import Select from '@/components/identifier/Select' import Summary from '@/components/identifier/Summary' import { formatTimestampUTCLabel } from '@/utils' +import UserBadge from '@/components/UserBadge.vue' export default { components: { Summary, Select, - TableToolbar + TableToolbar, + UserBadge }, data () { return { @@ -226,15 +228,6 @@ export default { } }, methods: { - formatCreator (creator) { - return UserMapper.userToFullName(creator) - }, - is_owner (table) { - if (!this.user) { - return false - } - return table.owner.id === this.user.id - }, amqpBadgeText (port) { if (port === 5672) { return 'insecure' @@ -264,4 +257,16 @@ export default { .v-card__text { font-size: initial; } +.skeleton-large > div { + width: 400px !important; +} +.skeleton-medium > div { + width: 200px !important; +} +.skeleton-small > div { + width: 100px !important; +} +.skeleton-xsmall > div { + width: 50px !important; +} </style> diff --git a/dbrepo-ui/pages/database/_database_id/view/_view_id/info.vue b/dbrepo-ui/pages/database/_database_id/view/_view_id/info.vue index 79e9edae2d1e570d004af01dc57b5e3dfaf6be9b..60b74bc0a03e69b2abc2b18c92a60b6c5ae5a676 100644 --- a/dbrepo-ui/pages/database/_database_id/view/_view_id/info.vue +++ b/dbrepo-ui/pages/database/_database_id/view/_view_id/info.vue @@ -20,29 +20,29 @@ Query Statement </v-list-item-title> <v-list-item-content> - <v-skeleton-loader v-if="!view.query" type="text" /> - <v-skeleton-loader v-if="!view.query" type="text" class="skeleton-large" /> - <pre v-if="view.query">{{ view.query }}</pre> + <v-skeleton-loader v-if="!view" type="text" class="skeleton-large" /> + <pre v-if="view">{{ view.query }}</pre> </v-list-item-content> - <v-list-item-title v-if="canViewView" class="mt-2"> + <v-list-item-title class="mt-2"> View Creator </v-list-item-title> - <v-list-item-content v-if="canViewView"> - <span v-if="creator">{{ creator }}</span> + <v-list-item-content> + <v-skeleton-loader v-if="!view" type="text" class="skeleton-small" /> + <UserBadge v-if="view" :user="view.creator" :other-user="user" /> </v-list-item-content> <v-list-item-title class="mt-2"> View Creation </v-list-item-title> <v-list-item-content> - <v-skeleton-loader v-if="!view.created" type="text" class="skeleton-medium" /> + <v-skeleton-loader v-if="!view" type="text" class="skeleton-medium" /> <span v-if="view.created">{{ formatUTC(view.created) }}</span> </v-list-item-content> <v-list-item-title> View Visibility </v-list-item-title> <v-list-item-content> - <v-skeleton-loader v-if="view.is_public === null" type="text" class="skeleton-xsmall" /> - <span v-if="view.is_public !== null">{{ view.is_public ? 'Public' : 'Private' }}</span> + <v-skeleton-loader v-if="!view" type="text" class="skeleton-xsmall" /> + <span v-if="view" v-text="view.is_public ? 'Public' : 'Private'" /> </v-list-item-content> </v-list-item-content> </v-list-item> @@ -85,15 +85,16 @@ import { formatTimestampUTCLabel } from '@/utils' import ViewToolbar from '@/components/view/ViewToolbar.vue' import UserMapper from '@/api/user.mapper' -import UserUtils from '@/api/user.utils' import Summary from '@/components/identifier/Summary.vue' import Select from '@/components/identifier/Select.vue' +import UserBadge from '@/components/UserBadge.vue' export default { components: { Select, Summary, - ViewToolbar + ViewToolbar, + UserBadge }, data () { return { @@ -151,10 +152,6 @@ export default { pid () { return this.$route.query.pid }, - canViewView () { - /* is private */ - return UserUtils.hasReadAccess(this.access) - }, hasIdentifier () { return this.identifiers.length > 0 }, @@ -162,6 +159,7 @@ export default { if (!this.view) { return null } + console.debug('====>', this.view) return UserMapper.userToFullName(this.view.creator) } }, @@ -189,4 +187,16 @@ pre { #back-btn::before { opacity: 0; } +.skeleton-large > div { + width: 400px !important; +} +.skeleton-medium > div { + width: 200px !important; +} +.skeleton-small > div { + width: 100px !important; +} +.skeleton-xsmall > div { + width: 50px !important; +} </style> diff --git a/dbrepo-ui/plugins/axios.js b/dbrepo-ui/plugins/axios.js index b2d7170ddf72f7e07c8e1ba3cefe21d013be2e42..6405d7bd358b87fa9be5ae83502126945d007e8a 100644 --- a/dbrepo-ui/plugins/axios.js +++ b/dbrepo-ui/plugins/axios.js @@ -21,13 +21,6 @@ api.interceptors.request.use(config => if (AuthenticationMapper.isExpiredToken(refreshToken)) { console.warn('Refresh token is expired: trigger logout of user') store().dispatch('logout') - .then(() => { - resolve(config) - }) - .catch((error) => { - console.error('Failed to logout', error) - reject(error) - }) resolve(config) } AuthenticationService.authenticateToken(refreshToken) @@ -39,8 +32,10 @@ api.interceptors.request.use(config => resolve(config) }) .catch((error) => { - console.error('Failed to refresh token', error) - resolve(config) + if (error.response.data.error === 'invalid_grant') { + store().dispatch('logout') + } + reject(error) }) } else { config.headers.Authorization = `Bearer ${store().state.token}` diff --git a/mkdocs.yml b/mkdocs.yml index b4dfba732c38d719639ace21b964457fde77f84e..f4ee5e8f5be1810a4e98a80334a92de0b3ac8dd2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,7 +18,6 @@ nav: - Gateway Service: system-services-gateway.md - Data Service: system-services-data.md - Metadata Service: system-services-metadata.md - - Mirror Service: system-services-mirror.md - Upload Service: system-services-upload.md - Storage Service: system-services-storage.md - Databases: