wfexs_backend.common

Contents

wfexs_backend.common#

Module Contents#

Classes#

EngineMode

ContentKind

ContainerType

ContainerTaggedName

origTaggedName: Symbolic name or identifier of the container

(including tag) which appears in the workflow.

type: Compatible container type with this symbolic name

Container factories have to decide whether they bear with it.

AttributionRole

The valid roles come from CASRAI CRediT, and can be visited through http://credit.niso.org/contributor-roles/{term}/

Attribution

LicenceDescription

This tuple is used to describe licences

LicensedURI

uri: The uri licence: The licence associated to the dataset behind this URI attributions: The attributions associated to the dataset pointed out by this URI secContext: The optional, security context to use when the uri has to be accessed. This is useful for use cases like DRS, where it can provide the authentication metadata

URIWithMetadata

uri: The uri, which can be either a raw or an annotated one metadata: A dictionary with the metadata associated to that URI. preferredName: A pretty way to name this resource. Workflow

ResolvedORCID

A resolved ORCID

MaterializedContent

local: Local absolute path of the content which was materialized. It

can be either a path in the cached inputs directory, or an absolute path in the inputs directory of the execution

licensed_uri: Either an URL or a CURIE of the content which was materialized,

needed for the provenance

prettyFilename: The preferred filename to use in the inputs directory

of the execution environment

fingerprint: If it is available, propagate the computed fingerprint

from the cache.

clonable: If it is true, copies of this materialized content can be

performed. Otherwise, content should remain in the original place represented by “local”.

ContentWithURIsDesc

encodingFormat: the kind of content with URIs (currently implemented is text/csv => tabular) setup: The dictionary describing the setup

MaterializedInput

name: Name of the input values: list of associated values, which can be literal ones or

ExpectedOutput

name: Name of the output. If the workflow engine allows using

symbolic names attached to the outputs, this name must match that. Otherwise, a matching pattern must be defined.

AbstractGeneratedContent

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

GeneratedContent

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

GeneratedDirectoryContent

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

values: The list of contents of the directory, which are either

GeneratedContent or GeneratedDirectoryContent

MaterializedOutput

name: Name of the output. It should be a public identifier whenever it is possible expectedCardinality: Whether it was expected to be optional, a single value or

LocalWorkflow

dir: The path to the directory where the checkout was applied relPath: Inside the checkout, the relative path to the workflow definition effectiveCheckout: hex hash of the materialized checkout langVersion: workflow language version / revision relPathFiles: files and directories composing the workflow, which can be either local or remote ones (i.e. CWL)

StagedSetup

ExecutionStatus

The status of a workflow execution

MarshallingStatus

ArgTypeMixin

StrDocEnum

ArgsDefaultWithRawHelpFormatter

CacheType

ExportItemType

Types of items which can be exported as such

CratableItem

What can be materialized in the RO-Crate

Functions#

Data#

API#

wfexs_backend.common.create_augmented_context(purpose: ssl.Purpose = ssl.Purpose.SERVER_AUTH, *, cafile: str | None = None, capath: str | None = None, cadata: str | bytes | None = None) ssl.SSLContext#
wfexs_backend.common.DEFAULT_DOCKER_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_SINGULARITY_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_APPTAINER_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_PODMAN_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_JAVA_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_FUSERMOUNT_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_DOT_CMD = 'cast(...)'#
wfexs_backend.common.DEFAULT_PROGS: wfexs_backend.common.ProgsMapping = None#
class wfexs_backend.common.EngineMode(*args, **kwds)#

Bases: enum.Enum

Local = 'local'#
Docker = 'docker'#
wfexs_backend.common.DEFAULT_ENGINE_MODE = None#
class wfexs_backend.common.ContentKind(*args, **kwds)#

Bases: enum.Enum

File = 'file'#
Directory = 'dir'#
Value = 'val'#
ContentWithURIs = 'luris'#
class wfexs_backend.common.ContainerType(*args, **kwds)#

Bases: enum.Enum

Singularity = 'singularity'#
Apptainer = 'singularity'#
Docker = 'docker'#
UDocker = 'udocker'#
Podman = 'podman'#
Conda = 'conda'#
NoContainer = 'none'#
class wfexs_backend.common.ContainerTaggedName#
origTaggedName: Symbolic name or identifier of the container

(including tag) which appears in the workflow.

type: Compatible container type with this symbolic name

Container factories have to decide whether they bear with it.

registries: an optional mapping from container type to registry, to be used by different container materialization solutions.

origTaggedName: str = None#
type: wfexs_backend.common.ContainerType = None#
registries: Mapping[ContainerType, str] | None = None#
class wfexs_backend.common.AttributionRole(*args, **kwds)#

Bases: enum.Enum

The valid roles come from CASRAI CRediT, and can be visited through http://credit.niso.org/contributor-roles/{term}/

Initialization

Conceptualization = 'conceptualization'#
DataCuration = 'data-curation'#
FormalAnalysis = 'formal-analysis'#
FundingAcquisition = 'funding-acquisition'#
Investigation = 'investigation'#
Methodology = 'methodology'#
ProjectAdministration = 'project-administration'#
Resources = 'resources'#
Software = 'software'#
Supervision = 'supervision'#
Validation = 'validation'#
Visualization = 'visualization'#
WritingOriginalDraft = 'writing-original-draft'#
WritingReviewEditing = 'writing-review-editing'#
class wfexs_backend.common.Attribution#

Bases: typing.NamedTuple

name: str = None#
pid: wfexs_backend.common.URIType = None#
roles: Sequence[AttributionRole] = []#
classmethod ParseRawAttribution(rawAttribution: Mapping[str, Any]) wfexs_backend.common.Attribution#
classmethod ParseRawAttributions(rawAttributions: Sequence[Mapping[str, Any]] | None) Sequence[Attribution]#
wfexs_backend.common.NoLicenceShort: Final[str] = 'notspecified'#
wfexs_backend.common.NoLicence: Final[URIType] = 'cast(...)'#
wfexs_backend.common.DefaultNoLicenceTuple: Tuple[URIType, ...] = ()#
class wfexs_backend.common.LicenceDescription#

Bases: typing.NamedTuple

This tuple is used to describe licences

short: str = None#
uris: Sequence[URIType] = None#
description: str = None#
is_spdx: bool = True#
get_uri() wfexs_backend.common.URIType#
wfexs_backend.common.NoLicenceDescription: Final[LicenceDescription] = 'LicenceDescription(...)'#
wfexs_backend.common.CC_BY_40_LICENCE: Final[str] = 'CC-BY-4.0'#
wfexs_backend.common.CC_BY_40_LicenceDescription: Final[LicenceDescription] = 'LicenceDescription(...)'#
class wfexs_backend.common.LicensedURI#

Bases: typing.NamedTuple

uri: The uri licence: The licence associated to the dataset behind this URI attributions: The attributions associated to the dataset pointed out by this URI secContext: The optional, security context to use when the uri has to be accessed. This is useful for use cases like DRS, where it can provide the authentication metadata

uri: wfexs_backend.common.URIType = None#
licences: Tuple[URIType | LicenceDescription, ...] = None#
attributions: Sequence[Attribution] = []#
secContext: SecurityContextConfig | None = None#
class wfexs_backend.common.URIWithMetadata#

Bases: typing.NamedTuple

uri: The uri, which can be either a raw or an annotated one metadata: A dictionary with the metadata associated to that URI. preferredName: A pretty way to name this resource. Workflow

execution can decide whether to honour it or not

uri: wfexs_backend.common.URIType = None#
metadata: Mapping[str, Any] = None#
preferredName: RelPath | None = None#
class wfexs_backend.common.ResolvedORCID#

Bases: typing.NamedTuple

A resolved ORCID

orcid: The resolved ORCID id url: The URL of the ORCID profile record: The fetched, public ORCID record record_fetch_metadata: Metadata about the resolution process

orcid: str = None#
url: wfexs_backend.common.URIType = None#
record: wfexs_backend.common.ORCIDPublicRecord = None#
record_fetch_metadata: Sequence[URIWithMetadata] = None#
class wfexs_backend.common.MaterializedContent#

Bases: typing.NamedTuple

local: Local absolute path of the content which was materialized. It

can be either a path in the cached inputs directory, or an absolute path in the inputs directory of the execution

licensed_uri: Either an URL or a CURIE of the content which was materialized,

needed for the provenance

prettyFilename: The preferred filename to use in the inputs directory

of the execution environment

fingerprint: If it is available, propagate the computed fingerprint

from the cache.

clonable: If it is true, copies of this materialized content can be

performed. Otherwise, content should remain in the original place represented by “local”.

local: wfexs_backend.common.PathlibLike = None#
licensed_uri: wfexs_backend.common.LicensedURI = None#
prettyFilename: wfexs_backend.common.RelPath = None#
kind: wfexs_backend.common.ContentKind = None#
metadata_array: Sequence[URIWithMetadata] | None = None#
extrapolated_local: PathlibLike | None = None#
fingerprint: Fingerprint | None = None#
clonable: bool = True#
classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any]#
classmethod _key_fixes() Mapping[str, str]#
class wfexs_backend.common.ContentWithURIsDesc#

Bases: typing.NamedTuple

encodingFormat: the kind of content with URIs (currently implemented is text/csv => tabular) setup: The dictionary describing the setup

encodingFormat: str = None#
setup: wfexs_backend.common.ContentWithURIsSetup = None#
class wfexs_backend.common.MaterializedInput#

Bases: typing.NamedTuple

name: Name of the input values: list of associated values, which can be literal ones or

instances from MaterializedContent

name: wfexs_backend.common.SymbolicParamName = None#
values: wfexs_backend.common.MaterializedInputValues = None#
secondaryInputs: Sequence[MaterializedContent] | None = None#
autoFilled: bool = False#
implicit: bool = False#
contentWithURIs: ContentWithURIsDesc | None = None#
disclosable: bool = True#
class wfexs_backend.common.ExpectedOutput#

Bases: typing.NamedTuple

name: Name of the output. If the workflow engine allows using

symbolic names attached to the outputs, this name must match that. Otherwise, a matching pattern must be defined.

kind: The kind of output. Either an atomic value. preferredFilename: Relative “pretty” name which is going to be used

to export the file to external storage.

cardinality: Whether it is expected to be optional, a single value or

multiple ones.

glob: When the workflow engine does not use symbolic

names to label the outputs, this is the filename pattern to capture the local path, based on the output / working directory.

syntheticOutput: It is true for outputs which do not really exist either as parameter or explicit outputs.

name: wfexs_backend.common.SymbolicOutputName = None#
kind: wfexs_backend.common.ContentKind = None#
preferredFilename: RelPath | None = None#
cardinality: Tuple[int, int] = None#
fillFrom: SymbolicParamName | None = None#
glob: GlobPattern | None = None#
syntheticOutput: bool | None = None#
_marshall() MutableMapping[str, Any]#
classmethod _unmarshall(**obj: Any) wfexs_backend.common.ExpectedOutput#
class wfexs_backend.common.AbstractGeneratedContent#

Bases: abc.ABC

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

signature: Computed checksum from the file preferredFilename: The preferred relative filename to use when it is

uploaded from the computational environment

local: wfexs_backend.common.PathlibLike = None#
signature: Fingerprint | None = None#
uri: LicensedURI | None = None#
preferredFilename: RelPath | None = None#
classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any]#
class wfexs_backend.common.GeneratedContent#

Bases: wfexs_backend.common.AbstractGeneratedContent

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

signature: Computed checksum from the file preferredFilename: The preferred relative filename to use when it is

uploaded from the computational environment

secondaryFiles: Sequence[AbstractGeneratedContent] | None = None#
class wfexs_backend.common.GeneratedDirectoryContent#

Bases: wfexs_backend.common.AbstractGeneratedContent

local: Local absolute path of the content which was generated. It

is an absolute path in the outputs directory of the execution.

uri: A putative URL or a CURIE of the content which was generated,

needed for the provenance and upload matters.

values: The list of contents of the directory, which are either

GeneratedContent or GeneratedDirectoryContent

signature: Optional computed checksum from the directory preferredFilename: The preferred relative filename to use when it is

uploaded from the computational environment

values: Sequence[AbstractGeneratedContent] | None = None#
secondaryFiles: Sequence[AbstractGeneratedContent] | None = None#
class wfexs_backend.common.MaterializedOutput#

Bases: typing.NamedTuple

name: Name of the output. It should be a public identifier whenever it is possible expectedCardinality: Whether it was expected to be optional, a single value or

multiple ones.

local: Local absolute path of the output prettyFilename: Relative “pretty” name to be used in provenance

name: wfexs_backend.common.SymbolicOutputName = None#
kind: wfexs_backend.common.ContentKind = None#
expectedCardinality: Tuple[int, int] = None#
values: Sequence[bool] | Sequence[str] | Sequence[int] | Sequence[float] | Sequence[AbstractGeneratedContent] = None#
syntheticOutput: bool | None = None#
filledFrom: str | None = None#
glob: GlobPattern | None = None#
class wfexs_backend.common.LocalWorkflow#

Bases: typing.NamedTuple

dir: The path to the directory where the checkout was applied relPath: Inside the checkout, the relative path to the workflow definition effectiveCheckout: hex hash of the materialized checkout langVersion: workflow language version / revision relPathFiles: files and directories composing the workflow, which can be either local or remote ones (i.e. CWL)

dir: wfexs_backend.common.PathlibLike = None#
relPath: RelPath | None = None#
effectiveCheckout: RepoTag | None = None#
langVersion: EngineVersion | WFLangVersion | None = None#
relPathFiles: Sequence[RelPath | URIType] | None = None#
classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any]#
class wfexs_backend.common.StagedSetup#

Bases: typing.NamedTuple

instance_id: wfexs_backend.common.WfExSInstanceId = None#
container_type: wfexs_backend.common.ContainerType = None#
nickname: str | None = None#
creation: datetime.datetime = None#
workflow_config: Mapping[str, Any] | None = None#
engine_tweaks_dir: pathlib.Path | None = None#
raw_work_dir: pathlib.Path = None#
work_dir: pathlib.Path | None = None#
workflow_dir: pathlib.Path | None = None#
consolidated_workflow_dir: pathlib.Path | None = None#
inputs_dir: pathlib.Path | None = None#
extrapolated_inputs_dir: pathlib.Path | None = None#
outputs_dir: pathlib.Path | None = None#
intermediate_dir: pathlib.Path | None = None#
containers_dir: pathlib.Path | None = None#
meta_dir: pathlib.Path | None = None#
temp_dir: pathlib.Path = None#
secure_exec: bool = None#
allow_other: bool = None#
is_encrypted: bool = None#
is_damaged: bool = None#
class wfexs_backend.common.ExecutionStatus(*args, **kwds)#

Bases: enum.Enum

The status of a workflow execution

Initialization

Queued = 'queued'#
Running = 'running'#
Finished = 'finished'#
Died = 'died'#
class wfexs_backend.common.MarshallingStatus#

Bases: typing.NamedTuple

pid: str | None = None#
workflow_type: str | None = None#
container_type: ContainerType | None = None#
config: bool | datetime.datetime | None = None#
stage: bool | datetime.datetime | None = None#
execution: bool | datetime.datetime | None = None#
export: bool | datetime.datetime | None = None#
execution_stats: Sequence[Tuple[str, ExecutionStatus, datetime.datetime | None, datetime.datetime, datetime.datetime, ExitVal]] | None = None#
export_stamps: Sequence[datetime.datetime] | None = None#
__repr__() str#
wfexs_backend.common.DEFAULT_CONTAINER_TYPE = None#
wfexs_backend.common.META_JSON_POSTFIX: Final[str] = '_meta.json'#
exception wfexs_backend.common.AbstractWfExSException#

Bases: Exception

class wfexs_backend.common.ArgTypeMixin(*args, **kwds)#

Bases: enum.Enum

classmethod argtype(s: str) enum.Enum#
__str__() str#
class wfexs_backend.common.StrDocEnum#

Bases: str, wfexs_backend.common.ArgTypeMixin

description: str = None#
__new__(value: Any, description: str = '') wfexs_backend.common.StrDocEnum#
__str__() str#
class wfexs_backend.common.ArgsDefaultWithRawHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)#

Bases: argparse.ArgumentDefaultsHelpFormatter

_split_lines(text: str, width: int) List[str]#

Formats the given text by splitting the lines at ‘

‘.

Overrides argparse.HelpFormatter._split_lines function.

param text:

help text passed by ArgumentParser.HelpFormatter

param width:

console width passed by argparse.HelpFormatter

return:

argparse.HelpFormatter._split_lines function

with new split text argument.

class wfexs_backend.common.CacheType#

Bases: wfexs_backend.common.StrDocEnum

Input = ('input', 'Cached or injected inputs')#
ROCrate = ('ro-crate', 'Cached RO-Crates (usually from WorkflowHub)')#
TRS = ('ga4gh-trs', 'Cached files from tools described at GA4GH TRS repositories')#
Workflow = ('workflow', 'Cached workflows, which come from a git repository')#
class wfexs_backend.common.ExportItemType(*args, **kwds)#

Bases: enum.Enum

Types of items which can be exported as such

Initialization

Param = 'param'#
Environment = 'envvar'#
Output = 'output'#
WorkingDirectory = 'working-directory'#
StageCrate = 'stage-rocrate'#
ProvenanceCrate = 'provenance-rocrate'#
class wfexs_backend.common.CratableItem#

Bases: enum.IntFlag

What can be materialized in the RO-Crate

Initialization

Initialize self. See help(type(self)) for accurate signature.

Workflow = 'auto(...)'#
Containers = 'auto(...)'#
Inputs = 'auto(...)'#
Outputs = 'auto(...)'#
ProspectiveProvenance = None#
RetrospectiveProvenance = None#
wfexs_backend.common.NoCratableItem = 'CratableItem(...)'#