wfexs_backend.common
#
Module Contents#
Classes#
|
|
The valid roles come from CASRAI CRediT, and can be visited through http://credit.niso.org/contributor-roles/{term}/ |
|
This tuple is used to describe licences |
|
uri: The uri licence: The licence associated to the dataset behind this URI attributions: The attributions associated to the dataset pointed out by this URI secContext: The optional, security context to use when the uri has to be accessed. This is useful for use cases like DRS, where it can provide the authentication metadata |
|
uri: The uri, which can be either a raw or an annotated one metadata: A dictionary with the metadata associated to that URI. preferredName: A pretty way to name this resource. Workflow |
|
A resolved ORCID |
|
|
|
encodingFormat: the kind of content with URIs (currently implemented is text/csv => tabular) setup: The dictionary describing the setup |
|
name: Name of the input values: list of associated values, which can be literal ones or |
|
|
|
|
|
|
|
|
|
name: Name of the output. It should be a public identifier whenever it is possible expectedCardinality: Whether it was expected to be optional, a single value or |
|
dir: The path to the directory where the checkout was applied relPath: Inside the checkout, the relative path to the workflow definition effectiveCheckout: hex hash of the materialized checkout langVersion: workflow language version / revision relPathFiles: files and directories composing the workflow, which can be either local or remote ones (i.e. CWL) |
|
The status of a workflow execution |
|
Types of items which can be exported as such |
|
What can be materialized in the RO-Crate |
Functions#
Data#
API#
- wfexs_backend.common.create_augmented_context(purpose: ssl.Purpose = ssl.Purpose.SERVER_AUTH, *, cafile: str | None = None, capath: str | None = None, cadata: str | bytes | None = None) ssl.SSLContext #
- wfexs_backend.common.DEFAULT_DOCKER_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_SINGULARITY_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_APPTAINER_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_PODMAN_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_JAVA_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_FUSERMOUNT_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_DOT_CMD = 'cast(...)'#
- wfexs_backend.common.DEFAULT_PROGS: wfexs_backend.common.ProgsMapping = None#
- class wfexs_backend.common.EngineMode(*args, **kwds)#
Bases:
enum.Enum
- Local = 'local'#
- Docker = 'docker'#
- wfexs_backend.common.DEFAULT_ENGINE_MODE = None#
- class wfexs_backend.common.ContentKind(*args, **kwds)#
Bases:
enum.Enum
- File = 'file'#
- Directory = 'dir'#
- Value = 'val'#
- ContentWithURIs = 'luris'#
- class wfexs_backend.common.ContainerType(*args, **kwds)#
Bases:
enum.Enum
- Singularity = 'singularity'#
- Apptainer = 'singularity'#
- Docker = 'docker'#
- UDocker = 'udocker'#
- Podman = 'podman'#
- Conda = 'conda'#
- NoContainer = 'none'#
- class wfexs_backend.common.ContainerTaggedName#
- origTaggedName: Symbolic name or identifier of the container
(including tag) which appears in the workflow.
- type: Compatible container type with this symbolic name
Container factories have to decide whether they bear with it.
registries: an optional mapping from container type to registry, to be used by different container materialization solutions.
- type: wfexs_backend.common.ContainerType = None#
- registries: Mapping[ContainerType, str] | None = None#
- class wfexs_backend.common.AttributionRole(*args, **kwds)#
Bases:
enum.Enum
The valid roles come from CASRAI CRediT, and can be visited through http://credit.niso.org/contributor-roles/{term}/
Initialization
- Conceptualization = 'conceptualization'#
- DataCuration = 'data-curation'#
- FormalAnalysis = 'formal-analysis'#
- FundingAcquisition = 'funding-acquisition'#
- Investigation = 'investigation'#
- Methodology = 'methodology'#
- ProjectAdministration = 'project-administration'#
- Resources = 'resources'#
- Software = 'software'#
- Supervision = 'supervision'#
- Validation = 'validation'#
- Visualization = 'visualization'#
- WritingOriginalDraft = 'writing-original-draft'#
- WritingReviewEditing = 'writing-review-editing'#
- class wfexs_backend.common.Attribution#
Bases:
typing.NamedTuple
- pid: wfexs_backend.common.URIType = None#
- roles: Sequence[AttributionRole] = []#
- classmethod ParseRawAttribution(rawAttribution: Mapping[str, Any]) wfexs_backend.common.Attribution #
- classmethod ParseRawAttributions(rawAttributions: Sequence[Mapping[str, Any]] | None) Sequence[Attribution] #
- wfexs_backend.common.NoLicence: Final[URIType] = 'cast(...)'#
- wfexs_backend.common.DefaultNoLicenceTuple: Tuple[URIType, ...] = ()#
- class wfexs_backend.common.LicenceDescription#
Bases:
typing.NamedTuple
This tuple is used to describe licences
- uris: Sequence[URIType] = None#
- get_uri() wfexs_backend.common.URIType #
- wfexs_backend.common.NoLicenceDescription: Final[LicenceDescription] = 'LicenceDescription(...)'#
- wfexs_backend.common.CC_BY_40_LicenceDescription: Final[LicenceDescription] = 'LicenceDescription(...)'#
- class wfexs_backend.common.LicensedURI#
Bases:
typing.NamedTuple
uri: The uri licence: The licence associated to the dataset behind this URI attributions: The attributions associated to the dataset pointed out by this URI secContext: The optional, security context to use when the uri has to be accessed. This is useful for use cases like DRS, where it can provide the authentication metadata
- uri: wfexs_backend.common.URIType = None#
- licences: Tuple[URIType | LicenceDescription, ...] = None#
- attributions: Sequence[Attribution] = []#
- class wfexs_backend.common.URIWithMetadata#
Bases:
typing.NamedTuple
uri: The uri, which can be either a raw or an annotated one metadata: A dictionary with the metadata associated to that URI. preferredName: A pretty way to name this resource. Workflow
execution can decide whether to honour it or not
- uri: wfexs_backend.common.URIType = None#
- class wfexs_backend.common.ResolvedORCID#
Bases:
typing.NamedTuple
A resolved ORCID
orcid: The resolved ORCID id url: The URL of the ORCID profile record: The fetched, public ORCID record record_fetch_metadata: Metadata about the resolution process
- url: wfexs_backend.common.URIType = None#
- record: wfexs_backend.common.ORCIDPublicRecord = None#
- record_fetch_metadata: Sequence[URIWithMetadata] = None#
- class wfexs_backend.common.MaterializedContent#
Bases:
typing.NamedTuple
- local: Local absolute path of the content which was materialized. It
can be either a path in the cached inputs directory, or an absolute path in the inputs directory of the execution
- licensed_uri: Either an URL or a CURIE of the content which was materialized,
needed for the provenance
- prettyFilename: The preferred filename to use in the inputs directory
of the execution environment
- fingerprint: If it is available, propagate the computed fingerprint
from the cache.
- clonable: If it is true, copies of this materialized content can be
performed. Otherwise, content should remain in the original place represented by “local”.
- local: wfexs_backend.common.PathlibLike = None#
- licensed_uri: wfexs_backend.common.LicensedURI = None#
- prettyFilename: wfexs_backend.common.RelPath = None#
- kind: wfexs_backend.common.ContentKind = None#
- metadata_array: Sequence[URIWithMetadata] | None = None#
- classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any] #
- class wfexs_backend.common.ContentWithURIsDesc#
Bases:
typing.NamedTuple
encodingFormat: the kind of content with URIs (currently implemented is text/csv => tabular) setup: The dictionary describing the setup
- setup: wfexs_backend.common.ContentWithURIsSetup = None#
- class wfexs_backend.common.MaterializedInput#
Bases:
typing.NamedTuple
name: Name of the input values: list of associated values, which can be literal ones or
instances from MaterializedContent
- name: wfexs_backend.common.SymbolicParamName = None#
- values: wfexs_backend.common.MaterializedInputValues = None#
- secondaryInputs: Sequence[MaterializedContent] | None = None#
- contentWithURIs: ContentWithURIsDesc | None = None#
- class wfexs_backend.common.ExpectedOutput#
Bases:
typing.NamedTuple
- name: Name of the output. If the workflow engine allows using
symbolic names attached to the outputs, this name must match that. Otherwise, a matching pattern must be defined.
kind: The kind of output. Either an atomic value. preferredFilename: Relative “pretty” name which is going to be used
to export the file to external storage.
- cardinality: Whether it is expected to be optional, a single value or
multiple ones.
- glob: When the workflow engine does not use symbolic
names to label the outputs, this is the filename pattern to capture the local path, based on the output / working directory.
syntheticOutput: It is true for outputs which do not really exist either as parameter or explicit outputs.
- name: wfexs_backend.common.SymbolicOutputName = None#
- kind: wfexs_backend.common.ContentKind = None#
- classmethod _unmarshall(**obj: Any) wfexs_backend.common.ExpectedOutput #
- class wfexs_backend.common.AbstractGeneratedContent#
Bases:
abc.ABC
- local: Local absolute path of the content which was generated. It
is an absolute path in the outputs directory of the execution.
- uri: A putative URL or a CURIE of the content which was generated,
needed for the provenance and upload matters.
signature: Computed checksum from the file preferredFilename: The preferred relative filename to use when it is
uploaded from the computational environment
- local: wfexs_backend.common.PathlibLike = None#
- uri: LicensedURI | None = None#
- classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any] #
- class wfexs_backend.common.GeneratedContent#
Bases:
wfexs_backend.common.AbstractGeneratedContent
- local: Local absolute path of the content which was generated. It
is an absolute path in the outputs directory of the execution.
- uri: A putative URL or a CURIE of the content which was generated,
needed for the provenance and upload matters.
signature: Computed checksum from the file preferredFilename: The preferred relative filename to use when it is
uploaded from the computational environment
- secondaryFiles: Sequence[AbstractGeneratedContent] | None = None#
- class wfexs_backend.common.GeneratedDirectoryContent#
Bases:
wfexs_backend.common.AbstractGeneratedContent
- local: Local absolute path of the content which was generated. It
is an absolute path in the outputs directory of the execution.
- uri: A putative URL or a CURIE of the content which was generated,
needed for the provenance and upload matters.
- values: The list of contents of the directory, which are either
GeneratedContent or GeneratedDirectoryContent
signature: Optional computed checksum from the directory preferredFilename: The preferred relative filename to use when it is
uploaded from the computational environment
- values: Sequence[AbstractGeneratedContent] | None = None#
- secondaryFiles: Sequence[AbstractGeneratedContent] | None = None#
- class wfexs_backend.common.MaterializedOutput#
Bases:
typing.NamedTuple
name: Name of the output. It should be a public identifier whenever it is possible expectedCardinality: Whether it was expected to be optional, a single value or
multiple ones.
local: Local absolute path of the output prettyFilename: Relative “pretty” name to be used in provenance
- name: wfexs_backend.common.SymbolicOutputName = None#
- kind: wfexs_backend.common.ContentKind = None#
- class wfexs_backend.common.LocalWorkflow#
Bases:
typing.NamedTuple
dir: The path to the directory where the checkout was applied relPath: Inside the checkout, the relative path to the workflow definition effectiveCheckout: hex hash of the materialized checkout langVersion: workflow language version / revision relPathFiles: files and directories composing the workflow, which can be either local or remote ones (i.e. CWL)
- dir: wfexs_backend.common.PathlibLike = None#
- classmethod _mapping_fixes(orig: Mapping[str, Any], workdir: pathlib.Path | None) Mapping[str, Any] #
- class wfexs_backend.common.StagedSetup#
Bases:
typing.NamedTuple
- instance_id: wfexs_backend.common.WfExSInstanceId = None#
- container_type: wfexs_backend.common.ContainerType = None#
- creation: datetime.datetime = None#
- engine_tweaks_dir: pathlib.Path | None = None#
- raw_work_dir: pathlib.Path = None#
- work_dir: pathlib.Path | None = None#
- workflow_dir: pathlib.Path | None = None#
- consolidated_workflow_dir: pathlib.Path | None = None#
- inputs_dir: pathlib.Path | None = None#
- extrapolated_inputs_dir: pathlib.Path | None = None#
- outputs_dir: pathlib.Path | None = None#
- intermediate_dir: pathlib.Path | None = None#
- containers_dir: pathlib.Path | None = None#
- meta_dir: pathlib.Path | None = None#
- temp_dir: pathlib.Path = None#
- class wfexs_backend.common.ExecutionStatus(*args, **kwds)#
Bases:
enum.Enum
The status of a workflow execution
Initialization
- Queued = 'queued'#
- Running = 'running'#
- Finished = 'finished'#
- Died = 'died'#
- class wfexs_backend.common.MarshallingStatus#
Bases:
typing.NamedTuple
- container_type: ContainerType | None = None#
- config: bool | datetime.datetime | None = None#
- stage: bool | datetime.datetime | None = None#
- execution: bool | datetime.datetime | None = None#
- export: bool | datetime.datetime | None = None#
- execution_stats: Sequence[Tuple[str, ExecutionStatus, datetime.datetime | None, datetime.datetime, datetime.datetime, ExitVal]] | None = None#
- export_stamps: Sequence[datetime.datetime] | None = None#
- wfexs_backend.common.DEFAULT_CONTAINER_TYPE = None#
- class wfexs_backend.common.StrDocEnum#
Bases:
str
,wfexs_backend.common.ArgTypeMixin
- __new__(value: Any, description: str = '') wfexs_backend.common.StrDocEnum #
- class wfexs_backend.common.ArgsDefaultWithRawHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)#
Bases:
argparse.ArgumentDefaultsHelpFormatter
- _split_lines(text: str, width: int) List[str] #
Formats the given text by splitting the lines at ‘
- ‘.
Overrides argparse.HelpFormatter._split_lines function.
- param text:
help text passed by ArgumentParser.HelpFormatter
- param width:
console width passed by argparse.HelpFormatter
- return:
argparse.HelpFormatter._split_lines function
with new split text argument.
- class wfexs_backend.common.CacheType#
Bases:
wfexs_backend.common.StrDocEnum
- Input = ('input', 'Cached or injected inputs')#
- ROCrate = ('ro-crate', 'Cached RO-Crates (usually from WorkflowHub)')#
- TRS = ('ga4gh-trs', 'Cached files from tools described at GA4GH TRS repositories')#
- Workflow = ('workflow', 'Cached workflows, which come from a git repository')#
- class wfexs_backend.common.ExportItemType(*args, **kwds)#
Bases:
enum.Enum
Types of items which can be exported as such
Initialization
- Param = 'param'#
- Environment = 'envvar'#
- Output = 'output'#
- WorkingDirectory = 'working-directory'#
- StageCrate = 'stage-rocrate'#
- ProvenanceCrate = 'provenance-rocrate'#
- class wfexs_backend.common.CratableItem#
Bases:
enum.IntFlag
What can be materialized in the RO-Crate
Initialization
Initialize self. See help(type(self)) for accurate signature.
- Workflow = 'auto(...)'#
- Containers = 'auto(...)'#
- Inputs = 'auto(...)'#
- Outputs = 'auto(...)'#
- ProspectiveProvenance = None#
- RetrospectiveProvenance = None#
- wfexs_backend.common.NoCratableItem = 'CratableItem(...)'#