Added data ingest

54fea249 · Martin Weise · 031ba8d7 · 54fea249 · 54fea249 · 54fea249
Unverified Commit 54fea249 authored Oct 12, 2023 by Martin Weise
--- a/docs/deployment-kubernetes-azure.md
+++ b/docs/deployment-kubernetes-azure.md
@@ -49,6 +49,6 @@ in your Kubernetes Cluster).

 We recommend to create 
 a [Container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers) for the
-[Upload Service](/system-services-upload/) to deposit files and mount the BLOB storage
+[Upload Service](../system-services-upload) to deposit files and mount the BLOB storage
 via CSI drivers into the *Deployment*. It greatly increases the available interfaces (see below) for file uploads and
 provides a highly-available filesystem for the many deployments that need to use the files.
--- a/docs/deployment-kubernetes-minikube.md
+++ b/docs/deployment-kubernetes-minikube.md
@@ -83,9 +83,9 @@ in the cluster. In our test deployment we obtained a certificate from Let's Encr

 ### Fileshare

-Since the Upload Service uses a shared filesystem with the [Analyst Service](/system-services-analyse/),
-[Metadata Service](/system-services-metadata/) and
-[Data Database](/system-databases-data/), the dynamic provision of the *PersistentVolume* 
+Since the Upload Service uses a shared filesystem with the [Analyst Service](../system-services-analyse),
+[Metadata Service](../system-services-metadata) and
+[Data Database](../system-databases-data), the dynamic provision of the *PersistentVolume* 
 by the *PersistentVolumeClaim* 
 of [`pvc.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/blob/master/charts/dbrepo-core/templates/upload-service/pvc.yaml)
 needs to happen statically. You can make use of the host's filesystem and mount it in each of those deployments.

--- a/docs/images/data-ingest.svg
+++ b/docs/images/data-ingest.svg
--- a/docs/system-databases-search.md
+++ b/docs/system-databases-search.md
@@ -14,12 +14,15 @@ author: Martin Weise

 ## Overview

-It processes search requests from the Gateway Service for full-text lookups in the metadata database. We use
-[Elasticsearch](https://www.elastic.co/) in the reference implementation. The search database implements Elastic Search
-and creates a retrievable index on all databases that is getting updated with each save operation on databases in the
-metadata database.
+It processes search requests from the Gateway Service for full-text lookups in 
+the [Metadata Database](../system-databases-metadata). We use [OpenSearch](https://opensearch.org/) in the default 
+configuration and create a searchable index on all databases that is updated regularly by 
+the [Mirror Service](../system-services-mirror).

-All requests need to be authenticated, by default the credentials `elastic:elastic` are used.
+All requests need to be authenticated, by default the credentials `admin:admin` are used.
+
+Please see the [Search Database Dashboard](../system-other-search-dashboard) for information how to inspect the Search
+Database more efficient.

 ## Limitations


--- a/docs/system-other-search-dashboard.md
+++ b/docs/system-other-search-dashboard.md
+---
+author: Martin Weise
+---
+
+# Search Database Dashboard
+
+## tl;dr
+
+!!! debug "Debug Information"
+
+    * Ports: 5601/tcp
+    * UI: `http://:5601/`
+
+## Overview
+
+It provides a *graphical user interface* (GUI) for an administrator to interact with 
+the [Search Database](../system-databases-search).
+
+## Limitations
+
+(none)
+
+## Security
+
+(none)
--- a/docs/system-other-ui.md
+++ b/docs/system-other-ui.md
@@ -4,15 +4,27 @@ author: Martin Weise

 # UI

+## tl;dr
+
 !!! debug "Debug Information"

    * Ports: 3000/tcp, 9100/tcp
    * Prometheus: `http://:9100/metrics`
    * UI: `http://:3000/`

+## Overview
+
 It provides a *graphical user interface* (GUI) for a researcher to interact with the database repository's API.

 <figure markdown>
 ![UI microservice architecture detailed](images/architecture-ui.png)
 <figcaption>Architecture of the UI microservice</figcaption>
 </figure>
+
+## Limitations
+
+(none)
+
+## Security
+
+(none)
--- a/docs/system.md
+++ b/docs/system.md
@@ -11,3 +11,9 @@ author: Martin Weise
 We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by
 sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise.

+## Data Ingest
+
+<figure markdown>
+![Data ingest](images/data-ingest.svg)
+<figcaption>Modes of data ingest</figcaption>
+</figure>
--- a/misc/architecture.drawio
+++ b/misc/architecture.drawio
-<mxfile host="Electron" modified="2023-10-12T09:52:54.322Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.1.2 Chrome/106.0.5249.199 Electron/21.4.3 Safari/537.36" etag="0r1qt_yaWFpHhzOEASeC" version="21.1.2" type="device" pages="7">
+<mxfile host="Electron" modified="2023-10-12T12:06:15.344Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.1.2 Chrome/106.0.5249.199 Electron/21.4.3 Safari/537.36" etag="TZYPGr9GyEHSU3c5VvbN" version="21.1.2" type="device" pages="7">
  <diagram id="mvBsv1rP8O80Qe3yGnn_" name="docker-compose">
    <mxGraphModel dx="1434" dy="822" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
      <root>
@@ -2404,26 +2404,11 @@
    </mxGraphModel>
  </diagram>
  <diagram id="e2fuorb2m_XjboQ7R7_y" name="Data Ingest">
-    <mxGraphModel dx="717" dy="411" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
+    <mxGraphModel dx="1195" dy="685" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
      <root>
        <mxCell id="0" />
        <mxCell id="1" parent="0" />
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-15" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;startArrow=classic;startFill=1;endArrow=none;endFill=0;dashed=1;" edge="1" parent="1" source="JkB3rZmVi6YrdEwAjtGV-17">
-          <mxGeometry relative="1" as="geometry">
-            <mxPoint x="276" y="203" as="targetPoint" />
-            <mxPoint x="157.5" y="384" as="sourcePoint" />
-            <Array as="points">
-              <mxPoint x="153" y="202" />
-              <mxPoint x="276" y="202" />
-            </Array>
-          </mxGeometry>
-        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-16" value="&lt;u&gt;443&lt;/u&gt;/tcp" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="JkB3rZmVi6YrdEwAjtGV-15">
-          <mxGeometry x="-0.2154" y="-1" relative="1" as="geometry">
-            <mxPoint as="offset" />
-          </mxGeometry>
-        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-1" value="&lt;u&gt;5671&lt;/u&gt;/tcp&lt;br&gt;5672/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;strokeColor=#000000;startArrow=classic;startFill=1;dashed=1;" parent="1" target="0VJCIkSjTVNNat6pGbHc-3" edge="1">
+        <mxCell id="0VJCIkSjTVNNat6pGbHc-1" value="5672/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;strokeColor=#000000;startArrow=none;startFill=0;dashed=1;" parent="1" target="0VJCIkSjTVNNat6pGbHc-3" edge="1">
          <mxGeometry x="-0.028" relative="1" as="geometry">
            <mxPoint x="448" y="220" as="sourcePoint" />
            <mxPoint as="offset" />
@@ -2432,7 +2417,7 @@
        <mxCell id="0VJCIkSjTVNNat6pGbHc-2" value="&lt;b&gt;AMQP&lt;/b&gt;" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" parent="1" vertex="1">
          <mxGeometry x="432.5" y="137" width="30" height="60" as="geometry" />
        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-30" value="&lt;u&gt;5671&lt;/u&gt;/tcp&lt;br&gt;5672/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="0VJCIkSjTVNNat6pGbHc-3" target="0VJCIkSjTVNNat6pGbHc-29" edge="1">
+        <mxCell id="0VJCIkSjTVNNat6pGbHc-30" value="5672/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="0VJCIkSjTVNNat6pGbHc-3" target="0VJCIkSjTVNNat6pGbHc-29" edge="1">
          <mxGeometry x="-0.1905" y="-1" relative="1" as="geometry">
            <mxPoint as="offset" />
          </mxGeometry>
@@ -2465,7 +2450,7 @@
            <mxPoint as="offset" />
          </mxGeometry>
        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-3" value="NGINX Ingress Controller" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
+        <mxCell id="0VJCIkSjTVNNat6pGbHc-3" value="Gateway Service" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="382.5" y="278" width="130" height="40" as="geometry" />
        </mxCell>
        <mxCell id="0VJCIkSjTVNNat6pGbHc-7" value="&lt;b&gt;JDBC&lt;/b&gt;" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" parent="1" vertex="1">
@@ -2478,10 +2463,18 @@
            <mxPoint x="600" y="250" as="targetPoint" />
          </mxGeometry>
        </mxCell>
+        <mxCell id="SPLHNWc1rHttcZ-O3PPf-2" value="&lt;font color=&quot;#b3b3b3&quot;&gt;(not used in&lt;br&gt;the default&lt;br&gt;config)&lt;/font&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;endArrow=cross;endFill=0;dashed=1;strokeColor=#B3B3B3;" edge="1" parent="1" source="0VJCIkSjTVNNat6pGbHc-9" target="JkB3rZmVi6YrdEwAjtGV-5">
+          <mxGeometry relative="1" as="geometry">
+            <Array as="points">
+              <mxPoint x="217" y="200" />
+              <mxPoint x="217" y="414" />
+            </Array>
+          </mxGeometry>
+        </mxCell>
        <mxCell id="0VJCIkSjTVNNat6pGbHc-9" value="&lt;b&gt;API&lt;/b&gt;" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" parent="1" vertex="1">
          <mxGeometry x="280" y="170" width="30" height="60" as="geometry" />
        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-10" value="&lt;u&gt;443&lt;/u&gt;/tcp&lt;br&gt;80/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#000000;startArrow=classic;startFill=1;dashed=1;" parent="1" edge="1">
+        <mxCell id="0VJCIkSjTVNNat6pGbHc-10" value="80/tcp" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;strokeColor=#000000;startArrow=classic;startFill=1;dashed=1;" parent="1" edge="1">
          <mxGeometry x="0.2319" relative="1" as="geometry">
            <mxPoint x="295" y="250" as="sourcePoint" />
            <mxPoint as="offset" />
@@ -2491,10 +2484,7 @@
            </Array>
          </mxGeometry>
        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-11" value="TLS/SSL secured channels are marked with &lt;u&gt;underline&lt;/u&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;" parent="1" vertex="1">
-          <mxGeometry x="292.5" y="540" width="310" height="30" as="geometry" />
-        </mxCell>
-        <mxCell id="0VJCIkSjTVNNat6pGbHc-13" value="data-db" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;" parent="1" vertex="1">
+        <mxCell id="0VJCIkSjTVNNat6pGbHc-13" value="data-db" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#fff2cc;strokeColor=#000000;" parent="1" vertex="1">
          <mxGeometry x="575" y="400" width="60" height="80" as="geometry" />
        </mxCell>
        <mxCell id="0VJCIkSjTVNNat6pGbHc-37" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="0VJCIkSjTVNNat6pGbHc-26" target="0VJCIkSjTVNNat6pGbHc-13" edge="1">
@@ -2530,7 +2520,7 @@
        <mxCell id="0VJCIkSjTVNNat6pGbHc-33" value="Data Service&lt;br&gt;(Spring AMQP)" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="316" y="480" width="96" height="40" as="geometry" />
        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-6" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.107;entryY=0.991;entryDx=0;entryDy=0;entryPerimeter=0;startArrow=classic;startFill=1;endArrow=none;endFill=0;" edge="1" parent="1" source="JkB3rZmVi6YrdEwAjtGV-5" target="0VJCIkSjTVNNat6pGbHc-3">
+        <mxCell id="JkB3rZmVi6YrdEwAjtGV-6" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.107;entryY=0.991;entryDx=0;entryDy=0;entryPerimeter=0;startArrow=classic;startFill=1;endArrow=none;endFill=0;" parent="1" source="JkB3rZmVi6YrdEwAjtGV-5" target="0VJCIkSjTVNNat6pGbHc-3" edge="1">
          <mxGeometry relative="1" as="geometry">
            <Array as="points">
              <mxPoint x="285" y="330" />
@@ -2538,29 +2528,16 @@
            </Array>
          </mxGeometry>
        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-7" value="80/tcp" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="JkB3rZmVi6YrdEwAjtGV-6">
+        <mxCell id="JkB3rZmVi6YrdEwAjtGV-7" value="80/tcp" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" parent="JkB3rZmVi6YrdEwAjtGV-6" vertex="1" connectable="0">
          <mxGeometry x="-0.5492" y="-2" relative="1" as="geometry">
            <mxPoint as="offset" />
          </mxGeometry>
        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-14" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="JkB3rZmVi6YrdEwAjtGV-5" target="JkB3rZmVi6YrdEwAjtGV-17">
-          <mxGeometry relative="1" as="geometry">
-            <mxPoint x="192.5" y="414" as="targetPoint" />
-          </mxGeometry>
-        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-5" value="Upload Service&lt;br&gt;(TUS)" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">
-          <mxGeometry x="237" y="394" width="96" height="40" as="geometry" />
-        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-18" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;dashed=1;strokeColor=#B3B3B3;" edge="1" parent="1" source="JkB3rZmVi6YrdEwAjtGV-17" target="0VJCIkSjTVNNat6pGbHc-33">
+        <mxCell id="SPLHNWc1rHttcZ-O3PPf-1" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="JkB3rZmVi6YrdEwAjtGV-5" target="0VJCIkSjTVNNat6pGbHc-33">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-19" value="PersistentVolumeClaim" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];fontColor=#B3B3B3;" vertex="1" connectable="0" parent="JkB3rZmVi6YrdEwAjtGV-18">
-          <mxGeometry x="0.0766" y="2" relative="1" as="geometry">
-            <mxPoint as="offset" />
-          </mxGeometry>
-        </mxCell>
-        <mxCell id="JkB3rZmVi6YrdEwAjtGV-17" value="Azure BLOB&lt;br&gt;Container" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="1">
-          <mxGeometry x="113" y="374" width="80" height="80" as="geometry" />
+        <mxCell id="JkB3rZmVi6YrdEwAjtGV-5" value="Upload Service&lt;br&gt;(TUS)" style="rounded=1;whiteSpace=wrap;html=1;" parent="1" vertex="1">
+          <mxGeometry x="237" y="394" width="96" height="40" as="geometry" />
        </mxCell>
      </root>
    </mxGraphModel>

--- a/mkdocs.yml
+++ b/mkdocs.yml
 site_name: Database Repository
 repo_url: https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services
 repo_name: fda-services
-site_url: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/latest/
 site_author: Research Unit Data Science, Technische Universit&auml;t Wien
 copyright: CC-BY 4.0 Technische Universit&auml;t Wien & Universit&auml;t Wien
 nav:
@@ -30,6 +29,7 @@ nav:
      - Search Database: system-databases-search.md
    - Other:
      - User Interface: system-other-ui.md
+      - Search Database Dashboard: system-other-search-dashboard.md
  - Usage:
    - Overview: usage-overview.md
    - Authentication Service: usage-auth.md