Local development environment

Important:

The most common environment developers and maintainers of Artemis use is Fedora, of a reasonably recent version, 31+. The "howto" described below is written with respect to this fact, and it is highly likely it would be a bit harder to follow on Debian or other Linux distributions. Please, bear with us.

Any patches to extend this text to non-Fedora realms would be very welcomed.

There are two possible way how to prepare and use a local environment for developing with Artemis:

  • using Minishift and Skaffold. This way, Artemis code is wrapped by containers, and runs side to side with other required services on top of a Minishift cluster, which is very similar to how Artemis is actually deployed.

    Important:

    Currently, the most recent versions of Skaffold are not working properly for our use case, use version 1.5 instead.

    Important:

    Currently, the most recent versions of Minishift are not working properly for our use case, use version v1.29.0 instead.

  • using Docker Compose for required services, and running Artemis directly on your workstation. This way is much more flexible than the first choice, and allows for very quick development, but it not how Artemis is deployed after all, and in some corner cases, things may behave differently.

    Warning:

    This way Artemis source runs in the same environment as the rest of the processes of your workstation. If you do not want to allow that, and you wish to keep Artemis isolated from the rest of your workstation, do not use this method.

Create configuration

No matter which way you chose, you will need a configuration for your Artemis server. Create the development configuration in the server/configuration/ directory.

You can use samples and server/configuration/gen.sh script which will setup Artemis to use your existing OpenStack project:

  • copy the env.yml.sample file:

    $ cd server/configuration
$ cp env.yml.sample env.yml
  • edit this new env.yml file, provide details of the cloud accounts you would like to use with Artemis (removing configuration for those you don’t need) and an SSH key pair to be used by Artemis as its master-key.

  • create an SSH keypair for Artemis.

    In case of OpenStack or AWS clouds you should also create a keypair with the public key in the OpenStack tenant or AWS account, the keypair name should match the openstack.vms.keyname or aws.vms.keyname in your env.yml, respectively. For example, if you set openstack.vms.keyname to artemis:

    $ openstack keypair create --public-key path/to/your-new-ssh-key.pub artemis

    In case of Beaker you should add the public key to the SSH Keys section of the User Preferences. This way you will be able to login as root without providing a password.

  • run gen.sh to generate Artemis configuration.

    $ bash ./gen.sh

Follow-up steps

After you complete the preferred way to setup your local development environment, there are additional steps to complete the whole process and make your environment ready.

Install development tools

Artemis packaging points to several tools that can help with the development and testing of your patches - pre-commit, linters.

Warning:

Technically, you do not need to install these tools to develop Artemis patches. However, some of them are mandatory part of Gitlab CI process testing each submitted patch (pre-commit) and other can be very helpful when integrated with your IDE (mypy, flake8, etc.).

  • install the Artemis sources and all requirements:

    $ cd server/
$ poetry install -v --no-root

  • install pre-commit hook:

    $ poetry run pre-commit install

To integrate the tools with your IDE, you will need provide IDE with path to the virtual environment managed by Poetry. Run following command:

$ poetry env info

Virtualenv
Python:         3.7.7
Implementation: CPython
Path:           /foo/bar/artemis/server/.venv 1
Valid:          True
...
  1. this is the path you are looking for.

Then follow your IDE’s documentation - it is often possible to point the IDE to particular linters by their path, e.g. /foo/bar/artemis/server/.venv/bin/mypy.

Minishift and Scaffold

Install requirements

Docker

Currently, Docker is required to run the development environment.

Important:

If you are running Fedora 31+, where Docker does not work out of the box, follow this guide to get it working.

Follow this guide to install Docker CE on Fedora.

Skaffold

Skaffold is a development tool for local Kubernetes development. To install it, follow this guide.

Important:

Currently, the most recent versions of Skaffold are not working properly for our use case, use version 1.5 instead.

Minishift

Minishift is a local OpenShift cluster running in a KVM VM on your workstation.

Important:

Currently, the most recent versions of Minishift are not working properly for our use case, use version v1.29.0 instead.

To install it, follow this guide.

After installation, follow this guide to set up the virtualization environment.

Minishift makes requests to the GitHub API to download an image. Sometimes, the GitHub limits the request from IP addresses - to solve this limitation, follow this guide.

Openshift Client Tools

Openshift Client Tools are required to interact with a Minishift cluster. To install them, follow this guide.

Create configuration

Follow the shared Create configuration instructions.

Start the development environment

Start the development environment by sourcing the develop.sh script:

$ source develop.sh

The first execution of the script will take some time, as it needs to start Minishift and build Artemis containers for the first time.

Interaction with the development environment

Don’t start Minishift

If you want to interact with the Minishfit and Skaffold, you can source develop.sh together with the -s option. Such use will not start Minishift nor Skaffold - the script will prepare the environment, but won’t progress further.

$ source develop.sh -s

Trigger redeployment

If you want to redeploy Artemis you made changes, press Enter in the terminal where you sourced the develop.sh script. This is called a manual deployment trigger, and it is a bit saner then the default trigger which redeploys automatically if any of the files changed.

Details of local development services

  • RabbitMQ Management Console
    • hostname: $(oc get route artemis-api)
    • user: guest
    • password: guest
  • RabbitMQ Management Console
    • hostname: $(oc get route artemis-rabbitmq-management)
    • user: guest
    • password: guest
  • PostgreSQL:
    • user: artemis
    • password: artemis
    • database: artemis

Docker Compose

This is the most lightweight development setup: it spawns the necessary requirements - RabbitMQ, PostgreSQL and Redis - in Docker containers on your workstation, while Artemis services - API server, dispatcher, scheduler and workers - are launched as daemons.

  • Artemis service lives in server directory:

    $ cd server/

  • create a local installation of Artemis:

    $ poetry install

    Poetry will take care of creating a dedicated virtual environment, installing requirements, and makes it accessible via poetry run or poetry shell.

  • follow the shared Create configuration instructions.

  • launch RabbitMQ, PostgreSQL and Redis containers:

    $ docker-compose up

  • start Artemis:

    $ bash nominishift-develop.sh
  • To stop Artemis, use a well-known Ctrl+C combination.

  • To stop the RabbitMQ, PostgreSQL and Redis containers, either use Ctrl+C as well, or, if you started them in a detached mode, use docker-compose:

    $ docker-compose down

Environment variables

Besides the variables accepted by Artemis itself, the nominishift-develop.sh accepts additional variables:

  • ONLY_DB_INIT: When set to any value, the nominishift-develop.sh script would perform the database initialization and quit immediately after that, without running any Artemis components.
  • SKIP_DB_INIT: When set to any value, the nominishift-develop.sh script would skip the database initialization.