Skip to content
Snippets Groups Projects
Unverified Commit 75911770 authored by Martin Weise's avatar Martin Weise
Browse files

Updated feature extract

parent 3a771a1d
No related branches found
No related tags found
2 merge requests!81New stable release,!80Multiple features connected with user management and ownership of databases
Show whitespace changes
Inline Side-by-side
.invenio/feature_extract.ipynb
+ 11
− 58
View file @ 75911770
%% Cell type:markdown id: tags:
# Feature Extraction & Deposit
In this notebook we define an example of creating a database from a .csv and perform feature extraction of audio files. The APIs are auto-generated from the Swagger Endpoint documentations using [`generate.sh`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-docs/-/blob/master/swagger/generate.sh). Steps we perform:
1. Download a music file from a public repository
2. Perform feature extraction
3. Create an account at DBRepo
4. Obtain an authentication token
5. Create a mariadb container
6. Start the mariadb container
7. Create a database within the mariadb container
8. Import the feature .csv (manually)
3. Obtain an authentication token
4. Create a mariadb container
5. Start the mariadb container
6. Create a database within the mariadb container
7. Import the feature .csv (manually)
%% Cell type:code id: tags:
``` python
import os.path
import uuid
import time
import re
import csv
import requests as rq
from api_authentication.api.authentication_endpoint_api import AuthenticationEndpointApi
from api_authentication.api.user_endpoint_api import UserEndpointApi
from api_container.api.container_endpoint_api import ContainerEndpointApi
from api_database.api.container_database_endpoint_api import ContainerDatabaseEndpointApi
from api_table.api.table_endpoint_api import TableEndpointApi
authentication = AuthenticationEndpointApi()
user = UserEndpointApi()
container = ContainerEndpointApi()
database = ContainerDatabaseEndpointApi()
table = TableEndpointApi()
doi = "10.5281/zenodo.5649276"
email = "some@example.com"
```
%% Cell type:markdown id: tags:
1. Download wav
8. Download wav
Resolve the DOI to URI
%% Cell type:code id: tags:
``` python
response = rq.get("https://doi.org/" + doi)
id = re.findall("/([a-z0-9-]+)$", response.url)[0]
host = re.findall("^https?:\/\/([a-z0-9]+\.[a-z]+)", response.url)[0]
print("Resolved DOI to", host, "and record id", id)
```
%% Output
Resolved DOI to zenodo.org and record id 5649276
%% Cell type:markdown id: tags:
2. Perform feature extraction
%% Cell type:code id: tags:
``` python
response = rq.get("https://" + host + "/api/records/" + id)
record = response.json()
with open(os.path.expanduser("~/features.csv"), "w") as f:
writer = csv.writer(f)
writer.writerow(["key", "size", "link"])
for file in record["files"]:
rq.get(file["links"]["self"])
print("... feature extract from", file["links"]["self"])
writer.writerow([file["key"], file["size"], file["links"]["self"]])
print("Generated a feature .csv in your home directory")
```
%% Output
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0044_20200518133554_1_m4a_1.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0044_20200518133554_2_m4a_1.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200611134530_1_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200611134530_2_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200612072315_1_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200612072315_2_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200613082517_1_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200613082517_2_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200614080017_1_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200614080017_2_m4a_0.wav
... feature extract from https://zenodo.org/api/files/22d69a63-2aff-47ae-b818-be78a23e9889/colive.0066_20200615070238_1_m4a_0.wav
Generated a feature .csv in your home directory
%% Cell type:markdown id: tags:
3. Create an account at DBRepo
%% Cell type:code id: tags:
``` python
response = user.register({
"username": "user",
"password": "user",
"email": email
})
print(response)
```
%% Output
{'authorities': [{'authority': 'ROLE_RESEARCHER'}],
'containers': None,
'databases': None,
'email': 'martinweiseat@gmail.com',
'firstname': None,
'id': 2,
'identifiers': None,
'lastname': None,
'titles_after': None,
'titles_before': None,
'username': 'user'}
%% Cell type:markdown id: tags:
4. Obtain an authentication token
3. Obtain an authentication token
%% Cell type:code id: tags:
``` python
response = authentication.authenticate_user1({
"username": "user",
"password": "user"
})
container.api_client.default_headers = {"Authorization": "Bearer " + response.token}
database.api_client.default_headers = {"Authorization": "Bearer " + response.token}
table.api_client.default_headers = {"Authorization": "Bearer " + response.token}
```
%% Cell type:markdown id: tags:
5. Create a mariadb container
4. Create a mariadb container
%% Cell type:code id: tags:
``` python
response = container.create1({
"name": "MIR " + str(uuid.uuid1()),
"repository": "mariadb",
"tag": "10.5"
})
container_id = response.id
print(response)
```
%% Output
{'hash': 'f5a649a71aae3748e62228721c44627ffc866f665d677bb890c37b9111590ffa',
'id': 2,
'internal_name': 'fda-userdb-mir-1010b964-f6fa-11ec-9f77-64bc58900b78',
'is_public': None,
'name': 'MIR 1010b964-f6fa-11ec-9f77-64bc58900b78'}
%% Cell type:markdown id: tags:
6. Start the mariadb container
5. Start the mariadb container
%% Cell type:code id: tags:
``` python
response = container.modify({
"action": "START"
}, container_id)
time.sleep(5)
print(response)
```
%% Output
{'hash': 'f5a649a71aae3748e62228721c44627ffc866f665d677bb890c37b9111590ffa',
'id': 2,
'internal_name': 'fda-userdb-mir-1010b964-f6fa-11ec-9f77-64bc58900b78',
'is_public': None,
'name': 'MIR 1010b964-f6fa-11ec-9f77-64bc58900b78'}
%% Cell type:markdown id: tags:
7. Create a database within the mariadb container
6. Create a database within the mariadb container
%% Cell type:code id: tags:
``` python
response = database.create({
"name": "MIR " + str(uuid.uuid1()),
"description": "Music Information Retrieval",
"is_public": True
}, container_id)
database_id = response.id
```
%% Cell type:markdown id: tags:
8. Import the feature .csv
7. Import the feature .csv
Now open [http://localhost:3000/](http://localhost:3000/) and import the .csv file by clicking the database. After successful creation of the table, come back here.
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment