.. _configuration: 📄 Configuration ================ WfExS software needs at least two and sometimes three different input confguration files to run your workflow: - :ref:`Local configuration` file (**required!**) - :ref:`Workflow staging configuration` file (**required!**) - :ref:`Security contexts` file (optional) .. index:: single: configuration; local-config Keep in mind that any workflow/situation is different, so we cannot give you an exact manual on what to do here. Configuration files will need to be filled adapting to each workflow execution needs. .. _local_config: Local configuration ------------------- WfExS needs a local installation configuration file. This is a YAML (``.yaml`` / ``.yml``) formatted file which describes the local setup of the backend. JSON Schema describing the format (and used for validation) is available at `wfexs_backend/schemas/config.json `_. Automatically generated documentation is also available (see `config_schema.md `_). An initial local configuration can be generated by running the following command in an empty directory: .. code-block:: bash mkdir ${CONFIG_DIR} python WfExS-backend.py -L ${CONFIG_DIR}/local_config_${USER}.yaml init --cache-dir .... .. list-table:: * - ``init`` - This command is used to initialize a WfExS installation. It takes a local configuration file through **-L** parameter, and it can both generate crypt4gh paired keys for installation work and identification purposes and update the path to them in case they are not properly defined. Those keys are needed to decrypt encrypted working directories, and in the future to decrypt secure requests and encrypt secure results. This will genarte the local configuration file as well as the encryption keys. The first time you run WfExS the new encryption keys will be made in those files. .. note:: ``python WfExS-backend.py --init`` will generate default local configuration file and ecryption keys in the current directory. ``wfexs_config.yml``, ``wfexs_config.yml.key``, ``wfexs_config.yml.pub`` An example of a local configuration file: .. code-block:: bash cacheDir: ../wfexs-backend-test crypt4gh: key: local_config.yaml.key passphrase: strive backyard dividing gumball pub: local_config.yaml.pub tools: dockerCommand: docker encrypted_fs: type: encfs command: encfs engineMode: local gitCommand: git javaCommand: java singularityCommand: singularity staticBashCommand: bash-linux-x86_64 workDir: ../wfexs-backend-test_WorkDir .. note:: Relative paths in this configuration file use as reference the directory where the local configuration file is living. More extensive examples are available at `workflow_examples/ `_. Plese fill the local configuration file according to your workflow needs. Main local configuration parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Description of main local configuration parameters. More detailed information can be found `here `_. .. list-table:: * - ``cacheDir`` - The path in this key sets up the place where all the contents which can be cached are hold. It contains downloaded RO-Crate, workflow git repositories, workflow engines. It is recommended to have it outside /tmp directory when Singularity is being used, due undesirable side interactions with the way workflow engines use Singularity. * - ``workDir`` - The path in this key sets up the place where all the executions are going to store both intermediate and final results, having a separate directory for each execution. It is recommended to have it outside /tmp directory when Singularity is being used, due undesirable side interactions with the way workflow engines use Singularity. * - crypt4gh.key - The path to the secret key used in this installation. It is paired to ``crypt4gh.pub``. * - crypt4gh.pub - The path to the public key used in this installation. It is paired to ``crypt4gh.key``. * - crypt4gh.passphrase - The passphrase needed to decrypt the contents of ``crypt4gh.key``. * - tools.engineMode - Currently, local mode only. * - tools.containerType - Currently, singularity, docker or podman. * - tools.gitCommand - Path to ``git`` command (only used when needed). * - tools.dockerCommand - Path to ``docker`` command (only used when needed). * - tools.singularityCommand - Path to ``singularity`` command (only used when needed). * - tools.javaCommand - Path to ``java`` command (only used when needed). .. index:: single: configuration; wf-config .. _wf_stage_config: Workflow staging configuration ------------------------------- Here, we describe how to make your own workflow configuration file. This is a ``YAML`` formatted file which describes the workflow staging before being executed: where inputs are located and can be fetched, the security contexts to be used on specific inputs to get those controlled access resources, the parameters, the outputs to capture, etc. * ``JSON`` Schema describing the format and valid keys (and used for validation), is available at `wfexs_backend/schemas/stage-definition.json `_ and there is also automatically generated documentation (see `stage-definition_schema.md `_). Generate a staging config file: .. code-block:: bash touch ${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.stage Minimum example template to fill in your ``${WORKFLOW_NAME}_wfex.stage`` file: .. code-block:: bash workflow_id: #URL to workflow workflow_config: secure: true / false containerType: docker / podman / singularity / none cacheDir: /path/to/chacheDir crypt4gh: key: /path/to/private-key passphrase: four random words here pub: /path/to/public-key params: ... outputs: ... The lines after ``params`` are used to describe the input files. The ``outputs`` map to the expected ``files`` / ``directories`` that come out at the end of the workflow execution. .. warning:: Only URLS can be used to define your workflow (i.e. workflow available through `WorkflowHub `_), input files and references, local files are not yet supported by WfExS. Plese fill the workflow staging configuration file according to your workflow needs. You need to know the specific steps which are performed in the workflow you are going to execute. - Define your ``input`` files and the ``references`` which need to be used for each step. - Define your ``outputs`` files. It can be described what the output file type is (this is predefined by the workflow itself) and its preferred name. You can find additional general examples of workflow configuration files on the WfExS GitHub page in `workflow_examples `_ folder. You'll find examples of workflow configuration files (files ending with ``.stage``) tailored for both CWL and Nextflow workflows. .. index:: single: configuration; security-config .. _secure_config: Security contexts file ---------------------- Some websites require credentials in order to download or access ``input``/ ``reference`` files. Usernames and passwords credentials for accessing secured files required in some steps of the workflow should be added to the ``security contexts file``. This is a ``YAML`` formatted file which holds the `user`/ `password` pairs, security tokens or keys needed on different steps, like input fetching. .. code-block:: bash touch ${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.credentials.ctxt An example of a security contexts file (``${CONFIG_DIR}/${WORKFLOW_NAME}_wfex.credentials.ctxt``): .. code-block:: bash public_broad: username: gsapubftp-anonymous password: "" More examples of security context files can be found `here `_ ``JSON`` Schema describing the format and valid keys (and used for validation), is available at `wfexs_backend/schemas/security-context.json `_ and there is also automatically generated documentation (see `security-context_schema.md `_).