Merge branch '437-ldap' into 'dev'

Resolve "LDAP"

See merge request !295

.docs/api/broker-service.md

@@ -19,23 +19,18 @@ It holds exchanges and topics responsible for holding AMQP messages for later co
 use [RabbitMQ](https://www.rabbitmq.com/) in the implementation. By default, the endpoint listens to the insecure port `5672` for incoming 
 AMQP tuples and insecure port `15672` for the management UI.
 
-The default configuration creates a user with administrative privileges on the default virtual host `dbrepo`:
+The default configuration allows any user in the `cn=system,ou=users,dc=dbrepo,dc=at` from the 
+[Identity Service](../identity-service) to access the Broker Service as user with `administrator` role, i.e. the
+`cn=admin,dc=dbrepo,dc=at` user that is created by default.
 
-* Username: `fda`
-* Password: `fda`
-* Roles: `["administrator"]`
+The Broker Service allows two ways of authentication for AMQP tuples:
 
-The Broker Service allows two ways of authentication:
+1. LDAP
+2. Plain (RabbitMQ's internal authentication)
 
-1. Plain
-2. OAuth2
-
-For detailed examples how to authenticate with the Broker Service see 
-the [usage](/usage-broker) page.
-
-The architecture of the Broker Service is very simple. There is only one durable, topic exchange `dbrepo` and one quorum
-queue `dbrepo`, connected with a binding of `dbrepo.#` which routes all tuples with routing key prefix `dbrepo.` (mind 
-the dot!) to this queue.
+The queue architecture of the Broker Service is very simple. There is only one durable, topic exchange `dbrepo` and one
+quorum queue `dbrepo`, connected with a binding of `dbrepo.#` which routes all tuples with routing key prefix `dbrepo.`
+to this queue.
 
 <figure markdown>
    ![Data ingest](../images/queue-quorum.png)
@@ -64,10 +59,10 @@ The consumer takes care of writing it to the correct table in the [Data Service]
 
 For a secure deployment it is necessary to configure the Broker Service as follows:
 
-1. Download the [`rabbitmq.conf`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/dev/dbrepo-broker-service/rabbitmq.conf.secure) and
-   change the `default_user` and `default_pass` lines before mounting it to `/etc/rabbitmq/rabbitmq.conf`.
-2. Mount your previously generated certificate and RSA public key pair (PEM-encoded) to `/app/cert.pem` 
+1. Once you change the admin password of the [Identity Service](../identity-service), you need to change it in the
+   `rabbitmq.conf` as well: `auth_ldap.dn_lookup_bind.password=newpassword`.
+2. Enable TLS and mount your previously generated certificate and RSA public key pair (PEM-encoded) to `/app/cert.pem` 
    and `/app/pubkey.pem`. Note that these are *not* used for TLS encryption, but only for authentication of users. It
-   is not recommended to use "real" TLS certificates, self-signed certificates with *sufficient keylength* are best-practice.
-3. Mount your TLS certificate authority file into `/etc/rabbitmq/cacert.crt` and your TLS certificate / private key pair
-   into `/etc/tls/tls.crt` and `/etc/tls/tls.key`.
+   is not recommended to use "real" TLS certificates, self-signed certificates with *sufficient keylength* are 
+   best-practice. Mount your TLS certificate authority file into `/etc/rabbitmq/cacert.crt` and your TLS certificate 
+   / private key pair into `/etc/tls/tls.crt` and `/etc/tls/tls.key`.

.docs/api/identity-service.md

+---
+author: Martin Weise
+---
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+    Image: [`docker.io/openldap:2.6.8-debian-12-r1`](https://hub.docker.com/r/openldap)
+
+    * Ports: 1389/tcp, 1636/tcp
+
+## Overview
+
+This service holds the user identities which we simply call identities in the following. It is integrated into the 
+[Auth Service](../auth-service) through an LDAP federation, allowing any identity to authenticate through the Auth
+Service. The LDAP protocol is not used for authentication. You can use your own identity provider, e.g. Active
+Directory.
+
+## Identities
+
+Any identity is identified by its `entryUUID` by default in the Auth Service. Note that Keycloak (the software running
+the Auth Service) may assign a different UUID to a user. DBRepo **always** uses the UUID provided through the Identity
+Service.
+
+## Limitations
+
+* Limited support for scaling in Kubernetes, see the
+  [guide](https://github.com/jp-gouin/helm-openldap?tab=readme-ov-file#scaling-your-cluster) of the chart developers.
+
+!!! question "Do you miss functionality? Do these limitations affect you?"
+
+    We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
+    in [contact](../contact) with us, we happily answer requests for collaboration with attached CV and your programming 
+    experience!
+
+## Security
+
+1. By default, no ingress is enabled. If you need ingress on LTP Password and phpLDAPadmin, configure the ingress
+   to use your TLS secret `tls-cert-secret` containing the `tls.crt` and `tls.key`, e.g.:
+
+   ```yaml title="values.yaml"
+   identityservice:
+     ltb-passwd:
+       ingress:
+         enabled: true
+         tls:
+           - secretName: tls-cert-secret
+             hosts:
+               - example.com
+     phpldapadmin:
+       ingress:
+         enabled: true
+         tls:
+           - secretName: tls-cert-secret
+             hosts:
+               - example.com
+   ```
\ No newline at end of file

.docs/installation.md

@@ -147,6 +147,12 @@ In case the deployment is unsuccessful, we have explanations on their origin and
 :   *Origin*:   Your deployment machine (e.g. laptop, virtual machine) appears to not have enough RAM assigned.
 :   *Solution*: Assign more RAM to the deployment machine (e.g. add vRAM to the virtual machine).
 
+**HTTP access denied: user 'admin' - invalid credentials**
+
+:   *Origin*:   The broker service cannot bind to the identity service due to wrong configuration.
+:   *Solution*: This is very likely due to a wrong `auth_ldap.dn_lookup_bind.password` in `rabbitmq.conf`. The error
+                indicates that LDAP check is not even attempted.
+
 ## Next Steps
 
 You should now be able to view the front end at [http://localhost](http://localhost).

.gitlab-ci.yml

@@ -18,6 +18,7 @@ cache:
 
 stages:
   - build
+  - lint
   - test
   - docs
   - release
@@ -140,6 +141,18 @@ verify-install-script:
     - bash install.sh
     - exit 0
 
+lint-helm:
+  image: docker.io/docker:24-dind
+  stage: lint
+  except:
+    refs:
+      - /^release-.*/
+  needs:
+    - build-metadata-service
+  script:
+    - apk add sed helm curl
+    - helm lint ./helm/dbrepo
+
 test-metadata-service:
   image: maven:3-openjdk-17
   stage: test

dbrepo-broker-service/README.md

+# Broker Service
+
+## Advanced Config
+
+https://www.rabbitmq.com/docs/ldap
\ No newline at end of file

dbrepo-broker-service/advanced.config

+[
+  {
+    rabbitmq_auth_backend_ldap,
+    [
+      {
+        tag_queries, [
+          {
+            administrator, {in_group_nested, "cn=system,ou=users,dn=dbrepo,dn=at", "member"}
+          },
+          {
+            management, {constant, true}
+          }
+        ]
+      }
+    ]
+  }
+].
\ No newline at end of file

dbrepo-broker-service/definitions.json

@@ -21,15 +21,7 @@
   ],
   "global_parameters": [],
   "parameters": [],
-  "permissions": [
-    {
-      "configure": ".*",
-      "read": ".*",
-      "user": "fda",
-      "vhost": "dbrepo",
-      "write": ".*"
-    }
-  ],
+  "permissions": [],
   "policies": [],
   "queues": [
     {
@@ -46,17 +38,7 @@
   "rabbit_version": "3.10.25",
   "rabbitmq_version": "3.10.25",
   "topic_permissions": [],
-  "users": [
-    {
-      "hashing_algorithm": "rabbit_password_hashing_sha256",
-      "limits": {},
-      "name": "fda",
-      "password_hash": "7e3Pa0qgP4kvQmCecg6mfFLDWuBEtKagLcNvPcgCd1XCr3sR",
-      "tags": [
-        "administrator"
-      ]
-    }
-  ],
+  "users": [],
   "vhosts": [
     {
       "limits": [],

dbrepo-broker-service/enabled_plugins

-[rabbitmq_prometheus,rabbitmq_auth_backend_oauth2,rabbitmq_auth_mechanism_ssl,rabbitmq_management].
\ No newline at end of file
+[rabbitmq_prometheus,rabbitmq_auth_backend_ldap,rabbitmq_auth_mechanism_ssl,rabbitmq_management].
\ No newline at end of file

dbrepo-broker-service/rabbitmq.conf

@@ -18,20 +18,15 @@ log.console.level = warning
 
 # Obviously your authentication server cannot vouch for itself, so you'll need another backend with at least one user in
 # it. You should probably use the internal database
-#auth_backends.1 = rabbit_auth_backend_oauth2
-auth_backends.1 = rabbit_auth_backend_internal
+auth_backends.1.authn = ldap
+auth_backends.1.authz = ldap
+auth_backends.2 = internal
 
-# management.oauth_enabled = true
-# management.oauth_client_id = rabbitmq-client
-# management.oauth_client_secret = JEC2FexxrX4N65fLeDGukAl6R3Lc9y0u
-# management.oauth_scopes = openid
-# management.oauth_provider_url = http://localhost/api/auth/realms/dbrepo
-
-# OAuth 2.0 files
-#auth_oauth2.resource_server_id = rabbitmq
-#auth_oauth2.preferred_username_claims.1 = client_id
-#auth_oauth2.default_key = t2OCeCheJ9uwoBbNQjG_nN6WKiLcceTIAZmiTbGODFM
-#auth_oauth2.signing_keys.t2OCeCheJ9uwoBbNQjG_nN6WKiLcceTIAZmiTbGODFM = /app/cert.pem
-#auth_oauth2.signing_keys.id2 = /app/pubkey.pem
-#auth_oauth2.algorithms.1 = HS256
-#auth_oauth2.algorithms.2 = RS256
+# LDAP
+auth_ldap.servers.1 = identity-service
+auth_ldap.port = 1389
+auth_ldap.user_dn_pattern = ${username}
+auth_ldap.dn_lookup_base = ou=users,dc=dbrepo,dc=at
+auth_ldap.dn_lookup_attribute = uid
+auth_ldap.dn_lookup_bind.user_dn = cn=admin,dc=dbrepo,dc=at
+auth_ldap.dn_lookup_bind.password = admin

dbrepo-broker-service/rabbitmq.conf.secure

-# user
-default_vhost = dbrepo
-default_user = fda
-default_pass = fda
-default_user_tags.administrator = true
-default_permissions.configure = .*
-default_permissions.read = .*
-default_permissions.write = .*
-
-# enable http outside localhost
-listeners.tcp.1 = 0.0.0.0:5672
-listeners.ssl.2 = 0.0.0.0:5671
-
-# management prefix (https://www.rabbitmq.com/management.html#path-prefix)
-management.path_prefix = /admin/broker
-management.load_definitions = /app/definitions.json
-
-# logging
-log.console = true
-log.console.level = warning
-ssl_options.cacertfile = /etc/rabbitmq/cacert.crt
-ssl_options.certfile = /etc/tls/tls.crt
-ssl_options.keyfile = /etc/tls/tls.key
-ssl_options.verify = verify_peer
-ssl_options.fail_if_no_peer_cert = true
-
-# Obviously your authentication server cannot vouch for itself, so you'll need another backend with at least one user in
-# it. You should probably use the internal database
-auth_backends.1 = rabbit_auth_backend_oauth2
-auth_backends.2 = rabbit_auth_backend_internal
-
-# OAuth 2.0 files
-auth_oauth2.resource_server_id = rabbitmq
-#auth_oauth2.additional_scopes_key = my_custom_scope_key
-auth_oauth2.preferred_username_claims.1 = client_id
-auth_oauth2.default_key = t2OCeCheJ9uwoBbNQjG_nN6WKiLcceTIAZmiTbGODFM
-auth_oauth2.signing_keys.t2OCeCheJ9uwoBbNQjG_nN6WKiLcceTIAZmiTbGODFM = /app/cert.pem
-auth_oauth2.signing_keys.id2 = /app/pubkey.pem
-auth_oauth2.algorithms.1 = HS256
-auth_oauth2.algorithms.2 = RS256

dbrepo-data-service/rest-service/src/main/resources/application-local.yml

@@ -9,8 +9,8 @@ spring:
   rabbitmq:
     host: localhost
     virtual-host: dbrepo
-    password: guest
-    username: guest
+    password: admin
+    username: admin
     port: 5672
   jpa:
     show-sql: false

dbrepo-data-service/rest-service/src/main/resources/application.yml

@@ -10,8 +10,8 @@ spring:
   rabbitmq:
     host: "${BROKER_HOST:broker-service}"
     virtual-host: "${BROKER_VIRTUALHOST:dbrepo}"
-    password: "${BROKER_PASSWORD:fda}"
-    username: "${BROKER_USERNAME:fda}"
+    password: "${BROKER_PASSWORD:admin}"
+    username: "${BROKER_USERNAME:admin}"
     port: ${BROKER_PORT:5672}
   jpa:
     show-sql: false

dbrepo-data-service/rest-service/src/test/java/at/tuwien/service/StorageServiceIntegrationTest.java

@@ -15,12 +15,10 @@ import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.beans.factory.annotation.Value;
 import org.springframework.boot.test.context.SpringBootTest;
 import org.springframework.test.context.DynamicPropertyRegistry;
 import org.springframework.test.context.DynamicPropertySource;
 import org.springframework.test.context.junit.jupiter.SpringExtension;
-import org.testcontainers.containers.MariaDBContainer;
 import org.testcontainers.containers.MinIOContainer;
 import org.testcontainers.junit.jupiter.Container;
 import org.testcontainers.junit.jupiter.Testcontainers;
@@ -32,7 +30,6 @@ import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import java.io.File;
 import java.io.InputStream;
 import java.sql.SQLException;
-import java.util.List;
 
 import static org.junit.jupiter.api.Assertions.*;
 

dbrepo-data-service/services/src/main/java/at/tuwien/auth/AuthTokenFilter.java

@@ -74,8 +74,8 @@ public class AuthTokenFilter extends OncePerRequestFilter {
         final DecodedJWT jwt = verifier.verify(token);
         final RealmAccessDto realmAccess = jwt.getClaim("realm_access").as(RealmAccessDto.class);
         return UserDetailsDto.builder()
-                .id(jwt.getSubject())
-                .username(jwt.getClaim("client_id").asString())
+                .id(jwt.getClaim("uid").asString())
+                .username(jwt.getClaim("preferred_username").asString())
                 .authorities(Arrays.stream(realmAccess.getRoles()).map(SimpleGrantedAuthority::new).collect(Collectors.toList()))
                 .build();
     }

dbrepo-data-service/services/src/main/java/at/tuwien/auth/BasicAuthenticationProvider.java

@@ -3,14 +3,11 @@ package at.tuwien.auth;
 import at.tuwien.api.keycloak.TokenDto;
 import at.tuwien.api.user.UserDetailsDto;
 import at.tuwien.config.GatewayConfig;
-import at.tuwien.exception.RemoteUnavailableException;
-import at.tuwien.exception.ServiceConnectionException;
-import at.tuwien.exception.ServiceException;
+import at.tuwien.exception.*;
 import at.tuwien.gateway.KeycloakGateway;
 import jakarta.servlet.ServletException;
 import lombok.extern.log4j.Log4j2;
 import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.beans.factory.annotation.Value;
 import org.springframework.security.authentication.AuthenticationManager;
 import org.springframework.security.authentication.BadCredentialsException;
 import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;

dbrepo-data-service/services/src/main/java/at/tuwien/config/RabbitConfig.java

 package at.tuwien.config;
 
+import at.tuwien.listener.DefaultListener;
 import lombok.Getter;
 import lombok.extern.log4j.Log4j2;
 import org.springframework.amqp.core.AcknowledgeMode;
@@ -7,6 +8,8 @@ import org.springframework.amqp.rabbit.config.SimpleRabbitListenerContainerFacto
 import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;
 import org.springframework.amqp.rabbit.connection.ConnectionFactory;
 import org.springframework.amqp.rabbit.core.RabbitTemplate;
+import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;
+import org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
@@ -53,34 +56,25 @@ public class RabbitConfig {
     private Integer connectionTimeout;
 
     @Bean
-    public SimpleRabbitListenerContainerFactory getSimpleRabbitListenerContainerFactory() {
-        log.debug("container factory settings: concurrentConsumers={}, maxConcurrentConsumers={}, acknowledgeMode={}, requeueRejected={}",
-                minConcurrent, maxConcurrent, AcknowledgeMode.AUTO, requeueRejected);
-        final SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
-        factory.setConnectionFactory(getConnectionFactory());
-        factory.setConcurrentConsumers(minConcurrent);
-        factory.setMaxConcurrentConsumers(maxConcurrent);
-        factory.setConsecutiveActiveTrigger(1);
-        factory.setAcknowledgeMode(AcknowledgeMode.AUTO);
-        factory.setDefaultRequeueRejected(requeueRejected);
-        return factory;
+    public SimpleMessageListenerContainer container(ConnectionFactory connectionFactory,
+                                             MessageListenerAdapter listenerAdapter) {
+        SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
+        container.setConnectionFactory(connectionFactory);
+        container.setQueueNames(queueName);
+        container.setMessageListener(listenerAdapter);
+        container.setConcurrentConsumers(minConcurrent);
+        container.setMaxConcurrentConsumers(maxConcurrent);
+        return container;
     }
 
     @Bean
-    public ConnectionFactory getConnectionFactory() {
-        log.debug("rabbitmq endpoint: amqp://{}:{}/{}", host, port, virtualHost);
-        final CachingConnectionFactory factory = new CachingConnectionFactory();
-        factory.setAddresses(host);
-        factory.setPort(port);
-        factory.setUsername(username);
-        factory.setPassword(password);
-        factory.setVirtualHost(virtualHost);
-        return factory;
+    public MessageListenerAdapter listenerAdapter(DefaultListener listener) {
+        return new MessageListenerAdapter(listener, "onMessage");
     }
 
     @Bean
-    public RabbitTemplate rabbitTemplate() {
-        return new RabbitTemplate(getConnectionFactory());
+    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
+        return new RabbitTemplate(connectionFactory);
     }
 
 }

dbrepo-data-service/services/src/main/java/at/tuwien/config/WebSecurityConfig.java

@@ -44,7 +44,8 @@ public class WebSecurityConfig {
 
     @Bean
     public SecurityFilterChain filterChain(HttpSecurity http, KeycloakGateway keycloakGateway,
-                                           GatewayConfig gatewayConfig) throws Exception {
+                                           GatewayConfig gatewayConfig)
+            throws Exception {
         final OrRequestMatcher internalEndpoints = new OrRequestMatcher(
                 new AntPathRequestMatcher("/actuator/**", "GET"),
                 new AntPathRequestMatcher("/v3/api-docs.yaml"),

dbrepo-data-service/services/src/main/java/at/tuwien/listener/DefaultListener.java

@@ -12,7 +12,6 @@ import lombok.extern.log4j.Log4j2;
 import org.springframework.amqp.core.Message;
 import org.springframework.amqp.core.MessageListener;
 import org.springframework.amqp.core.MessageProperties;
-import org.springframework.amqp.rabbit.annotation.RabbitListener;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
@@ -23,7 +22,6 @@ import java.util.Map;
 
 @Log4j2
 @Component
-@RabbitListener(queues = "dbrepo")
 public class DefaultListener implements MessageListener {
 
     private final ObjectMapper objectMapper;

dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/QueueServiceRabbitMqImpl.java

@@ -46,6 +46,7 @@ public class QueueServiceRabbitMqImpl extends HibernateConnector implements Queu
                 dataMapper.prepareStatementWithColumnTypeObject(preparedStatement, optional.get().getColumnType(), idx[0]++,
                         entry.getValue());
             }
+            preparedStatement.executeUpdate();
             log.trace("successfully inserted tuple");
         } finally {
             dataSource.close();