📄 Configuration

WfExS software needs at least two and sometimes three different input confguration files to run your workflow:

• Local configuration file (required!)

• Workflow staging configuration file (required!)

• Security contexts file (optional)

Keep in mind that any workflow/situation is different, so we cannot give you an exact manual on what to do here. Configuration files will need to be filled adapting to each workflow execution needs.

Local configuration

WfExS needs a local installation configuration file. This is a YAML (.yaml / .yml) formatted file which describes the local setup of the backend. JSON Schema describing the format (and used for validation) is available at wfexs_backend/schemas/config.json. Automatically generated documentation is also available (see config_schema.md).

An initial local configuration can be generated by running the following command in an empty directory:

mkdir ${CONFIG_DIR}
python WfExS-backend.py -L ${CONFIG_DIR}/local_config_${USER}.yaml --cache-dir ${CACHE_DIR} init

init

This command is used to initialize a WfExS installation. It takes a local configuration file through -L parameter, and it can both generate crypt4gh paired keys for installation work and identification purposes and update the path to them in case they are not properly defined. Those keys are needed to decrypt encrypted working directories, and in the future to decrypt secure requests and encrypt secure results.

This will genarte the local configuration file as well as the encryption keys. The first time you run WfExS the new encryption keys will be made in those files.

Note

python WfExS-backend.py --init will generate default local configuration file and ecryption keys in the current directory. wfexs_config.yml, wfexs_config.yml.key, wfexs_config.yml.pub

An example of a local configuration file:

cacheDir: ../wfexs-backend-test
crypt4gh:
   key: local_config.yaml.key
   passphrase: strive backyard dividing gumball
   pub: local_config.yaml.pub
tools:
   dockerCommand: docker
encrypted_fs:
   type: encfs
   command: encfs
engineMode: local
gitCommand: git
javaCommand: java
singularityCommand: singularity
staticBashCommand: bash-linux-x86_64
workDir: ../wfexs-backend-test_WorkDir

Note

Relative paths in this configuration file use as reference the directory where the local configuration file is living.

More extensive examples are available at workflow_examples/.

Plese fill the local configuration file according to your workflow needs.

Main local configuration parameters

Description of main local configuration parameters. More detailed information can be found here.

cacheDir	The path in this key sets up the place where all the contents which can be cached are hold. It contains downloaded RO-Crate, workflow git repositories, workflow engines. It is recommended to have it outside /tmp directory when Singularity is being used, due undesirable side interactions with the way workflow engines use Singularity.
workDir	The path in this key sets up the place where all the executions are going to store both intermediate and final results, having a separate directory for each execution. It is recommended to have it outside /tmp directory when Singularity is being used, due undesirable side interactions with the way workflow engines use Singularity.
crypt4gh.key	The path to the secret key used in this installation. It is paired to crypt4gh.pub.
crypt4gh.pub	The path to the public key used in this installation. It is paired to crypt4gh.key.
crypt4gh.passphrase	The passphrase needed to decrypt the contents of crypt4gh.key.
tools.engineMode	Currently, local mode only.
tools.containerType	Currently, singularity, docker or podman.
tools.gitCommand	Path to git command (only used when needed).
tools.dockerCommand	Path to docker command (only used when needed).
tools.singularityCommand	Path to singularity command (only used when needed).
tools.javaCommand	Path to java command (only used when needed).

Workflow staging configuration

Here, we describe how to make your own workflow configuration file. This is a YAML formatted file which describes the workflow staging before being executed: where inputs are located and can be fetched, the security contexts to be used on specific inputs to get those controlled access resources, the parameters, the outputs to capture, etc.

• JSON Schema describing the format and valid keys (and used for validation),

is available at wfexs_backend/schemas/stage-definition.json and there is also automatically generated documentation (see stage-definition_schema.md).

Generate a staging config file:

touch ${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.stage

Minimum example template to fill in your ${WORKFLOW_NAME}_wfex.stage file:

workflow_id: #URL to workflow
workflow_config:
    secure: true / false
    containerType: docker / podman / singularity / none
cacheDir: /path/to/chacheDir
crypt4gh:
    key: /path/to/private-key
    passphrase: four random words here
    pub: /path/to/public-key
params:
...
outputs:
...

The lines after params are used to describe the input files. The outputs map to the expected files / directories that come out at the end of the workflow execution.

Warning

Only URLS can be used to define your workflow (i.e. workflow available through WorkflowHub), input files and references, local files are not yet supported by WfExS.

Plese fill the workflow staging configuration file according to your workflow needs. You need to know the specific steps which are performed in the workflow you are going to execute.

• Define your input files and the references which need to be used for each step.

• Define your outputs files. It can be described what the output file type is (this is predefined by the workflow itself) and its preferred name.

You can find additional general examples of workflow configuration files on the WfExS GitHub page in workflow_examples folder. You’ll find examples of workflow configuration files (files ending with .stage) tailored for both CWL and Nextflow workflows.

Security contexts file

Some websites require credentials in order to download or access input/ reference files. Usernames and passwords credentials for accessing secured files required in some steps of the workflow should be added to the security contexts file. This is a YAML formatted file which holds the user/ password pairs, security tokens or keys needed on different steps, like input fetching.

touch ${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.credentials.ctxt

An example of a security contexts file (${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.credentials.ctxt):

public_broad:
    username: gsapubftp-anonymous
    password: ""

More examples of security context files can be found here

JSON Schema describing the format and valid keys (and used for validation), is available at wfexs_backend/schemas/security-context.json and there is also automatically generated documentation (see security-context_schema.md).