diff --git a/Python/QA-015-Using Pipelines for building.md b/Python/QA-015-Using Pipelines for building.md new file mode 100644 index 0000000000000000000000000000000000000000..124ceb17e1084afda9e561e2cd7e969b38d0f947 --- /dev/null +++ b/Python/QA-015-Using Pipelines for building.md @@ -0,0 +1,55 @@ +# Using a Gitlab pipeline for building and publishing a python package + +## Steps + +## 1. Prepare Your Python Package + +Ensure your Python package is properly structured and includes the following: + +A pyproject.toml file for package configuration. + +A README.md file for documentation. + +## 2. Create a .gitlab-ci.yml File + +This file defines the pipeline stages and jobs. Below is an example .gitlab-ci.yml for building and publishing a Python package: +```sh +default: + image: python:3.13 + cache: + paths: + - .pip-cache/ + before_script: + - python --version + - pip install --upgrade pip + - pip install build twine + +stages: + - build + - publish + +variables: + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache" + +build: + stage: build + script: + - python -m build + artifacts: + paths: + - dist/ + + +publish: + stage: publish + script: + - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/* + rules: + - if: $CI_COMMIT_TAG +``` + + +## 3.Push Your Code and Tags + +Push your code to the GitLab repository. +To trigger the publish stage, create and push a tag \ No newline at end of file diff --git a/Python/QA-016-Migrate-from-Anaconda.ipynb b/Python/QA-016-Migrate-from-Anaconda.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..beafbbe53ac5196bcf1c54f9f1f90e80808a5765 --- /dev/null +++ b/Python/QA-016-Migrate-from-Anaconda.ipynb @@ -0,0 +1,127 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Migrating from Anaconda to Micromamba/Mamba/Miniforge\n", + "\n", + "To migrate from Anacondo to Micromamba follow the steps below.\n", + "\n", + "1. Export the installed Packages into a yml\n", + "\n", + "```sh\n", + "conda env export -n <env-name> > <env-name>.yml\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Make sure your yml doesn't use the defaults channel but conda-forge instead\n", + "\n", + "```yaml\n", + "name: test\n", + "channels:\n", + "- pkgs/main # replace with conda-forge\n", + "dependencies:\n", + "- ...\n", + "```\n", + "\n", + "**Open the yml file you just created and remove the channel default.**\n", + "Add the channel `conda-forge`. Using both at the same time can lead to problems with the environment.\n", + "\n", + "2. Create the new environment from the yml file\n", + "\n", + "```sh\n", + "# using micromamba, available on all IMGW servers\n", + "micromamba create -n <env-name> --file <env-name>.yml\n", + "\n", + "# or using mamba\n", + "mamba env create -n my_new_env -f environment.yml\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Remember that you should also remove the conda part in your `~.bashrc` file.** e.g.:\n", + "\n", + "```sh\n", + "# >>> conda initialize >>>\n", + "# !! Contents within this block are managed by 'conda init' !!\n", + "__conda_setup=\"$('/path/to/conda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)\"\n", + "if [ $? -eq 0 ]; then\n", + " eval \"$__conda_setup\"\n", + "else\n", + " if [ -f \"/path/to/conda/etc/profile.d/conda.sh\" ]; then\n", + " . \"/path/to/conda/etc/profile.d/conda.sh\"\n", + " else\n", + " export PATH=\"/path/to/conda/bin:$PATH\"\n", + " fi\n", + "fi\n", + "unset __conda_setup\n", + "# <<< conda initialize <<<\n", + "```\n", + "Remove this! You can add the micromamba shell init if you wish by running: `micromamba shell init`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Migrating from Anaconda to venv\n", + "\n", + "1. Export the packages into a requirements.txt:\n", + "\n", + "```sh\n", + "# activate conda environment or use -n or -p options.\n", + "conda list -n <env_name> -p <env_path> --export > requirements.txt\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "2. Run the following command to create a new virtual environment:\n", + "\n", + "```sh\n", + "python -m venv <env-name>\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Activate the virtual environment\n", + "\n", + "`<env-name>\\Scripts\\activate`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that your virtual environment is active, you can install the packages from your Conda environment.\n", + "\n", + "```sh\n", + "pip install -r requirements.txt\n", + "```\n", + "\n", + "finished." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/Python/README.md b/Python/README.md index bab09fcd097d9c8998ec30fec4ecb3dc270d4284..ef9918fa25da841facbbd6083162962088b48bfd 100644 --- a/Python/README.md +++ b/Python/README.md @@ -168,4 +168,12 @@ $ python -c 'import sys;print("\n".join(sys.path))' /jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages # unset the variable again unset PYTHONNOUSERSITE -``` \ No newline at end of file +``` + +## Q: How to use pipelines for building and publishing packages? +[Pipelines for building](QA-015-Using Pipelines for building.md) + +Use a .gitlab-ci.yml file to automate building and publishing of your package. + +## Q: How do I migrate from Anaconda to Venv or Micromamba +[Migrate away from Anaconda](QA-016-Migrate-from-Anaconda.ipynb) \ No newline at end of file