Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
DBRepo
Manage
Activity
Members
Labels
Plan
External wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Deploy
Releases
Package registry
Model registry
Operate
Terraform modules
Analyze
Contributor analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
FAIR Data Austria DB Repository
DBRepo
Merge requests
!399
Added scheduled scraping that removes stale datasets
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
Added scheduled scraping that removes stale datasets
513-scrape-old-datasets
into
dev
Overview
0
Commits
1
Pipelines
0
Changes
12
Merged
Martin Weise
requested to merge
513-scrape-old-datasets
into
dev
2 months ago
Overview
0
Commits
1
Pipelines
0
Changes
12
Expand
Signed-off-by: Martin Weise
martin.weise@tuwien.ac.at
Closes #513
0
0
Merge request reports
Compare
dev
dev (base)
and
latest version
latest version
9b7ed3cb
1 commit,
2 months ago
12 files
+
109
−
10
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
Files
12
Search (e.g. *.vue) (Ctrl+P)
.docs/api/data-service.md
+
7
−
1
Options
@@ -46,12 +46,18 @@ everytime e.g. a sensor measurement is inserted. By default, this information is
administrators can disable this behavior by setting
`CREDENTIAL_CACHE_TIMEOUT=0`
(cache is deleted after 0 seconds).
##
Upload
##
Storage
The Data Service also is capable to upload files to the S3 backend. The default limit
of
[
`Tomcat`
](
https://spring.io/guides/gs/uploading-files#_tuning_file_upload_limits
)
in Spring Boot is configured to be
`2GB`
. You can provide your own limit with setting
`MAX_UPLOAD_SIZE`
.
By default, the Data Service removes datasets older than 24 hours on a regular basis every 60 minutes. You can set the
`MAX_AGE`
(in seconds) and
`S3_STALE_CRON`
to fit your use-case. You can disable this feature by setting
`S3_STALE_CRON`
to
`-`
, this may lead to storage issues as no space will be available inevitably. Note
that
[
Spring Boot uses its own flavor
](
https://spring.io/blog/2020/11/10/new-in-spring-5-3-improved-cron-expressions#usage
)
of cron syntax.
## Limitations
*
Views in DBRepo can only have 63-character length (it is assumed only internal views have the maximum length of 64
Loading