CLI directives

CLI directives#

WfExS (workflow execution service) backend 1.0.0rc2

usage: WfExS-backend.py [-h] [--log-file LOGFILENAME] [-q] [-v] [-d]
                        [-L LOCALCONFIGFILENAME] [--cache-dir CACHEDIR] [-V]
                        [--full-help]
                        {init,populate-side-caches,cache,staged-workdir,export,list-fetchers,list-exporters,list-container-factories,list-workflow-engines,list-licences,config-validate,try-stage,stage,re-stage,import,mount-workdir,export-stage,offline-execute,execute,export-results,export-crate}
                        ...

Named Arguments#

--log-file

Store messages in a file instead of using standard error and standard output

-q, --quiet

Only show engine warnings and errors

-v, --verbose

Show verbose (informational) messages

-d, --debug

Show debug messages (use with care, as it can disclose passphrases and passwords)

-L, --local-config

Local installation configuration file (can also be set up through WFEXS_CONFIG_FILE environment variable)

Default: '/home/docs/checkouts/readthedocs.org/user_builds/wfexs-backend/checkouts/latest/docs/source/wfexs_config.yml'

--cache-dir

Caching directory

-V, --version

show program’s version number and exit

--full-help

It returns full help

Default: False

commands#

Command to run. It must be one of these

command: Possible choices: init, populate-side-caches, cache, staged-workdir, export, list-fetchers, list-exporters, list-container-factories, list-workflow-engines, list-licences, config-validate, try-stage, stage, re-stage, import, mount-workdir, export-stage, offline-execute, execute, export-results, export-crate

Sub-commands#

init#

Init local setup

WfExS-backend.py init [-h]

populate-side-caches#

Populate read-only side caches which live in XDG_CACHE_HOME (shared by all the WfExS instances of the very same user)

WfExS-backend.py populate-side-caches [-h]

cache#

Cache handling subcommands

WfExS-backend.py cache [-h] [-r] [--cascade] [-g]
                       {ls,status,inject,fetch,rm,validate}
                       {input,ro-crate,ga4gh-trs,workflow}
                       [cache_command_args ...]

Positional Arguments#

cache_command

Possible choices: ls, status, inject, fetch, rm, validate

Cache command to perform

ls List the cache entries status Show the cache entries metadata inject Inject a new entry in the cache fetch Fetch a new cache entry, giving as input both the URI and optionally both a security context file and a security context name rm Remove an entry from the cache validate Validate the consistency of the cache

cache_type

Possible choices: input, ro-crate, ga4gh-trs, workflow

Cache type to perform the cache command

input Cached or injected inputs ro-crate Cached RO-Crates (usually from WorkflowHub) ga4gh-trs Cached files from tools described at GA4GH TRS repositories workflow Cached workflows, which come from a git repository

cache_command_args

Optional cache element names

Named Arguments#

-r

Try doing the operation recursively (i.e. both metadata and data)

Default: False

--cascade

Try doing the operation in cascade (including the URIs which resolve to other URIs)

Default: False

-g, --glob

Given cache element names are globs

Default: False

staged-workdir#

Staged working directories handling subcommands

WfExS-backend.py staged-workdir [-h] [--orcid ORCIDS]
                                [--private-key-file PRIVATE_KEY_FILE]
                                [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                                [--inputs] [--outputs] [--workflow]
                                [--containers] [--prospective] [--full]
                                [--licence LICENCES] [--crate-pid CRATE_PID]
                                [-g]
                                {offline-exec,offline-queue,ls,mount,rm,shell,status,create-staged-crate,create-prov-crate}
                                [staged_workdir_command_args ...]

Positional Arguments#

staged_workdir_command

Possible choices: offline-exec, offline-queue, ls, mount, rm, shell, status, create-staged-crate, create-prov-crate

Staged working directory command to perform

offline-exec Offline execute the staged instances which match the input pattern offline-queue Queue offline execution about the staged instances which match the input pattern ls List the staged instances

It shows the instance id, nickname, encryption and whether they are damaged

mount Mount the staged instances which match the input pattern rm Removes the staged instances which match the input pattern shell Launches a command in the workdir

First parameter is either the staged instance id or the nickname. It launches the command specified after the id. If there is no additional parameters, it launches a shell in the mounted working directory of the instance

status Shows staged instances status create-staged-crateIt creates an RO-Crate from the prospective provenance create-prov-crateIt creates an RO-Crate from the retrospective provenance (after a workflow execution)

staged_workdir_command_args

Optional staged working directory element names

Named Arguments#

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

-g, --glob

Given staged workflow names are globs

Default: False

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

ro-crate-payload#

What to include in the RO-Crate

--inputs

Should the RO-Crate contain a inputs copy (of everything)?

Default: []

--outputs

Should the RO-Crate contain a outputs copy (of everything)?

Default: []

--workflow

Should the RO-Crate contain a workflow copy (of everything)?

Default: []

--containers

Should the RO-Crate contain a containers copy (of everything)?

Default: []

--prospective

Should the RO-Crate contain a prospective copy (of everything)?

Default: []

--full

Should the RO-Crate contain a full copy (of everything)?

Default: []

--licence

Licence(s) to attach to the generated RO-Crate

Default: []

--crate-pid

Permanent identifier to embed within the generated RO-Crate metadata, like a pre-generated DOI

export#

Staged working directories export subcommands

WfExS-backend.py export [-h] [-Z SECURITYCONTEXTSCONFIGFILENAME]
                        [-E EXPORTSCONFIGFILENAME] [--orcid ORCIDS]
                        [--public-key-file PUBLIC_KEY_FILES]
                        [--private-key-file PRIVATE_KEY_FILE]
                        [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                        -J WORKFLOWWORKINGDIRECTORY [--licence LICENCES]
                        {ls,run} [export_contents_command_args ...]

Positional Arguments#

export_contents_command

Possible choices: ls, run

Export operations from staged working directory to perform

ls List the public identifiers obtained from previous export actions run Run the different export actions, pushing the exported content and gathering the obtained permanent / public identifiers

export_contents_command_args

Optional export names

Named Arguments#

-Z, --creds-config

Configuration file, describing security contexts, which hold credentials and similar

-E, --exports-config

Configuration file, describing exports which can be done

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

--public-key-file

This parameter switches on secure processing. Path to the public key(s) to be used to encrypt the working directory

Default: []

-J, --staged-job-dir

Already staged job directory

--licence

Licence(s) to attach to the exported contents

Default: []

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

list-fetchers#

List the supported fetchers / schemes

WfExS-backend.py list-fetchers [-h]

list-exporters#

List the supported export plugins

WfExS-backend.py list-exporters [-h]

list-container-factories#

List the supported container factories

WfExS-backend.py list-container-factories [-h]

list-workflow-engines#

List the supported workflow engines

WfExS-backend.py list-workflow-engines [-h]

list-licences#

List the documented licences, both embedded and fetched from SPDX release 3.23

WfExS-backend.py list-licences [-h]

config-validate#

Validate the configuration files to be used for staging and execution

WfExS-backend.py config-validate [-h] -W WORKFLOWCONFIGFILENAME
                                 [-Z SECURITYCONTEXTSCONFIGFILENAME]

Named Arguments#

-W, --workflow-config: Configuration file, describing workflow and inputs
-Z, --creds-config: Configuration file, describing security contexts, which hold credentials and similar

try-stage#

Prepare a workflow in a new staging (working) directory for workflow execution, fetching dependencies but no input

WfExS-backend.py try-stage [-h] --workflow-uri WORKFLOWURI
                           [-Z SECURITYCONTEXTSCONFIGFILENAME]
                           [-n NICKNAME_PREFIX]

Named Arguments#

--workflow-uri: URI of the workflow to be tried
-Z, --creds-config: Configuration file, describing security contexts, which hold credentials and similar
-n, --nickname-prefix: Nickname prefix to be used on staged workdir creation

stage#

Prepare the staging (working) directory for workflow execution, fetching dependencies and contents

WfExS-backend.py stage [-h] -W WORKFLOWCONFIGFILENAME [--paranoid]
                       [-Z SECURITYCONTEXTSCONFIGFILENAME]
                       [-n NICKNAME_PREFIX] [--orcid ORCIDS]
                       [--public-key-file PUBLIC_KEY_FILES]
                       [--private-key-file PRIVATE_KEY_FILE]
                       [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]

Named Arguments#

-W, --workflow-config

Configuration file, describing workflow and inputs

--paranoid

Force secured working directory

Default: False

-Z, --creds-config

Configuration file, describing security contexts, which hold credentials and similar

-n, --nickname-prefix

Nickname prefix to be used on staged workdir creation

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

--public-key-file

This parameter switches on secure processing. Path to the public key(s) to be used to encrypt the working directory

Default: []

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

re-stage#

Prepare a new staging (working) directory for workflow execution, repeating the fetch of dependencies and contents

WfExS-backend.py re-stage [-h] [-W WORKFLOWCONFIGFILENAME] [-s] [-S]
                          [--strict-reproducibility]
                          [--no-strict-reproducibility]
                          [--reproducibility-level {1,2,3}]
                          [-Z SECURITYCONTEXTSCONFIGFILENAME]
                          [-n NICKNAME_PREFIX] [--orcid ORCIDS]
                          [--public-key-file PUBLIC_KEY_FILES]
                          [--private-key-file PRIVATE_KEY_FILE]
                          [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                          -J WORKFLOWWORKINGDIRECTORY

Named Arguments#

-W, --workflow-config

Optional configuration file, describing some inputs which will replace the base, original ones

-s, --no-secure

Make unsecured working directory

Default: True

-S, --secure

Make secured working directory (default)

Default: False

--strict-reproducibility

Strict reproducibility

Default: False

--no-strict-reproducibility

Permissive reproducibility

Default: True

--reproducibility-level

Possible choices: 1, 2, 3

Max reproducibility level to be tried

Default: 2

-Z, --creds-config

Configuration file, describing security contexts, which hold credentials and similar

-n, --nickname-prefix

Nickname prefix to be used on staged workdir creation

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

--public-key-file

This parameter switches on secure processing. Path to the public key(s) to be used to encrypt the working directory

Default: []

-J, --staged-job-dir

Already staged job directory

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

import#

Workflow Run RO-Crate import into a new staged working directory

WfExS-backend.py import [-h] -R WORKFLOWROCRATEFILENAMEORURI
                        [--ignore-retrospective-provenance]
                        [--prefer-retrospective-provenance]
                        [-W WORKFLOWCONFIGFILENAME] [-s] [-S]
                        [--strict-reproducibility]
                        [--no-strict-reproducibility]
                        [--reproducibility-level {1,2,3}]
                        [-Z SECURITYCONTEXTSCONFIGFILENAME]
                        [-n NICKNAME_PREFIX] [--orcid ORCIDS]
                        [--public-key-file PUBLIC_KEY_FILES]
                        [--private-key-file PRIVATE_KEY_FILE]
                        [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]

Named Arguments#

-R, --workflow-rocrate

Workflow Run RO-Crate describing a previous workflow execution. It can be either a local path or an URI resolvable from WfExS with no authentication

--ignore-retrospective-provenance

Retrospective provenance is ignored

Default: True

--prefer-retrospective-provenance

Retrospective provenance is first inspected

Default: False

-W, --workflow-config

Optional configuration file, describing some inputs which will replace the base, original ones

-s, --no-secure

Make unsecured working directory

Default: True

-S, --secure

Make secured working directory (default)

Default: False

--strict-reproducibility

Strict reproducibility

Default: False

--no-strict-reproducibility

Permissive reproducibility

Default: True

--reproducibility-level

Possible choices: 1, 2, 3

Max reproducibility level to be tried

Default: 2

-Z, --creds-config

Configuration file, describing security contexts, which hold credentials and similar

-n, --nickname-prefix

Nickname prefix to be used on staged workdir creation

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

--public-key-file

This parameter switches on secure processing. Path to the public key(s) to be used to encrypt the working directory

Default: []

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

mount-workdir#

Mount the encrypted staging directory on secure staging scenarios

WfExS-backend.py mount-workdir [-h] [--private-key-file PRIVATE_KEY_FILE]
                               [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                               -J WORKFLOWWORKINGDIRECTORY

Named Arguments#

-J, --staged-job-dir: Already staged job directory

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

export-stage#

Export the staging directory as an RO-Crate

WfExS-backend.py export-stage [-h] [--orcid ORCIDS]
                              [--private-key-file PRIVATE_KEY_FILE]
                              [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                              -J WORKFLOWWORKINGDIRECTORY [--inputs]
                              [--outputs] [--workflow] [--containers]
                              [--prospective] [--full] [--licence LICENCES]
                              [--crate-pid CRATE_PID]

Named Arguments#

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

-J, --staged-job-dir

Already staged job directory

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

ro-crate-payload#

What to include in the RO-Crate

--inputs

Should the RO-Crate contain a inputs copy (of everything)?

Default: []

--outputs

Should the RO-Crate contain a outputs copy (of everything)?

Default: []

--workflow

Should the RO-Crate contain a workflow copy (of everything)?

Default: []

--containers

Should the RO-Crate contain a containers copy (of everything)?

Default: []

--prospective

Should the RO-Crate contain a prospective copy (of everything)?

Default: []

--full

Should the RO-Crate contain a full copy (of everything)?

Default: []

--licence

Licence(s) to attach to the generated RO-Crate

Default: []

--crate-pid

Permanent identifier to embed within the generated RO-Crate metadata, like a pre-generated DOI

offline-execute#

Execute an already prepared workflow in the staging directory

WfExS-backend.py offline-execute [-h] [--private-key-file PRIVATE_KEY_FILE]
                                 [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                                 -J WORKFLOWWORKINGDIRECTORY

Named Arguments#

-J, --staged-job-dir: Already staged job directory

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

execute#

Execute the stage + offline-execute + export steps

WfExS-backend.py execute [-h] -W WORKFLOWCONFIGFILENAME [--paranoid]
                         [-Z SECURITYCONTEXTSCONFIGFILENAME]
                         [-E EXPORTSCONFIGFILENAME] [-n NICKNAME_PREFIX]
                         [--orcid ORCIDS] [--public-key-file PUBLIC_KEY_FILES]
                         [--private-key-file PRIVATE_KEY_FILE]
                         [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                         [--inputs] [--outputs] [--workflow] [--containers]
                         [--prospective] [--full] [--licence LICENCES]
                         [--crate-pid CRATE_PID]

Named Arguments#

-W, --workflow-config

Configuration file, describing workflow and inputs

--paranoid

Force secured working directory

Default: False

-Z, --creds-config

Configuration file, describing security contexts, which hold credentials and similar

-E, --exports-config

Configuration file, describing exports which can be done

-n, --nickname-prefix

Nickname prefix to be used on staged workdir creation

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

--public-key-file

This parameter switches on secure processing. Path to the public key(s) to be used to encrypt the working directory

Default: []

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

ro-crate-payload#

What to include in the RO-Crate

--inputs

Should the RO-Crate contain a inputs copy (of everything)?

Default: []

--outputs

Should the RO-Crate contain a outputs copy (of everything)?

Default: []

--workflow

Should the RO-Crate contain a workflow copy (of everything)?

Default: []

--containers

Should the RO-Crate contain a containers copy (of everything)?

Default: []

--prospective

Should the RO-Crate contain a prospective copy (of everything)?

Default: []

--full

Should the RO-Crate contain a full copy (of everything)?

Default: []

--licence

Licence(s) to attach to the generated RO-Crate

Default: []

--crate-pid

Permanent identifier to embed within the generated RO-Crate metadata, like a pre-generated DOI

export-results#

Export the results to a remote location, gathering their public ids

WfExS-backend.py export-results [-h] [--orcid ORCIDS]
                                [--private-key-file PRIVATE_KEY_FILE]
                                [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                                -J WORKFLOWWORKINGDIRECTORY
                                [--licence LICENCES]

Named Arguments#

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

-J, --staged-job-dir

Already staged job directory

--licence

Licence(s) to attach to the exported contents

Default: []

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

export-crate#

Export an already executed workflow in the staging directory as an RO-Crate

WfExS-backend.py export-crate [-h] [--orcid ORCIDS]
                              [--private-key-file PRIVATE_KEY_FILE]
                              [--private-key-passphrase-envvar PRIVATE_KEY_PASSPHRASE_ENVVAR]
                              -J WORKFLOWWORKINGDIRECTORY [--inputs]
                              [--outputs] [--workflow] [--containers]
                              [--prospective] [--full] [--licence LICENCES]
                              [--crate-pid CRATE_PID]

Named Arguments#

--orcid

ORCID(s) of the person(s) staging, running or exporting the workflow scenario

Default: []

-J, --staged-job-dir

Already staged job directory

secure workdir arguments#

Private key and passphrase to access secured working directories

--private-key-file

This parameter passes the name of the file containing the private key needed to unlock an encrypted working directory.

--private-key-passphrase-envvar

This parameter passes the name of the environment variable containing the passphrase needed to decrypt the private key needed to unlock an encrypted working directory.

Default: ''

ro-crate-payload#

What to include in the RO-Crate

--inputs

Should the RO-Crate contain a inputs copy (of everything)?

Default: []

--outputs

Should the RO-Crate contain a outputs copy (of everything)?

Default: []

--workflow

Should the RO-Crate contain a workflow copy (of everything)?

Default: []

--containers

Should the RO-Crate contain a containers copy (of everything)?

Default: []

--prospective

Should the RO-Crate contain a prospective copy (of everything)?

Default: []

--full

Should the RO-Crate contain a full copy (of everything)?

Default: []

--licence

Licence(s) to attach to the generated RO-Crate

Default: []

--crate-pid

Permanent identifier to embed within the generated RO-Crate metadata, like a pre-generated DOI

CLI directives

Contents

CLI directives#

Named Arguments#

commands#

Sub-commands#

init#

populate-side-caches#

cache#

Positional Arguments#

Named Arguments#

staged-workdir#

Positional Arguments#

Named Arguments#

secure workdir arguments#

ro-crate-payload#

export#

Positional Arguments#

Named Arguments#

secure workdir arguments#

list-fetchers#

list-exporters#

list-container-factories#

list-workflow-engines#

list-licences#

config-validate#

Named Arguments#

try-stage#

Named Arguments#

stage#

Named Arguments#

secure workdir arguments#

re-stage#

Named Arguments#

secure workdir arguments#

import#

Named Arguments#

secure workdir arguments#

mount-workdir#

Named Arguments#

secure workdir arguments#

export-stage#

Named Arguments#

secure workdir arguments#

ro-crate-payload#

offline-execute#

Named Arguments#

secure workdir arguments#

execute#

Named Arguments#

secure workdir arguments#

ro-crate-payload#

export-results#

Named Arguments#

secure workdir arguments#

export-crate#

Named Arguments#

secure workdir arguments#

ro-crate-payload#