Provenance

class snappl.provenance.Provenance(process, major, minor, params={}, environment=None, env_major=None, env_minor=None, omitkeys=['system'], keepkeys=None, upstreams=[])

Bases: object

Instantiate a Provenance

Once instantiated, will have a property id that holds the UUID for this provenance. This UUID is defined from a md5 hash of all the arguments, so will be the same every time you pass the same arguments. (It’s convenient that md5sums and UUIDs are both 128-bit numbers.)

Parmaeters

processstr

The name of the process, e.g. “phrosty”, “campari”, “import_rapid_alert”, etc.

majorint

Semantic major version of the code described by process.

minorint

Semantic minor version of the code described by process.

paramsConfig or dict, default {}

Parameters that uniquely define process. This should include all parameters that would be the same for all runs on one set of data. So, for instance, for difference imaging transient detection software, you would not include the name of the science image or the name of the transient. However, you would include things like configuration parameters to SFFT, detection thresholds, and the name and parameters of however you decided to figure out which template image to use.

You can also pass a snappl.config.Config object, in which case the parameters will be extracted from that. This assumes that the “system” top level key of that Config has all, but only, the system-specific stuff.

environmentint, default None

Which SNPIT environment did the process use? TODO: this still need to be defined.

env_majorint, default None

Semantic major version of environment.

env_minorint, default None

Semantic minor version of environment.

upstreamslist of Provenance

Upstream provenances to this provenance. Only include immediate upstreams; no need for upstreams of upstreams, as those will be tracked by the immedaite upstreams. Can also send a single Provenance.

omitkeyslist of str, default [‘system’]

Ignored unless params is a Config object. In this case, these are the keys from the Config to omit and not include in the parameters dictionary. Only one of omitkeys or keepkeys can be non-None.

keepkeyslist of str, default None

Ignored unless params is a Config object. In this case, only include the specified keys from the Config.

Methods Summary

get(process, major, minor[, params, ...])	Get a Provenance based on properties.
get_by_id(provid[, dbclient, ...])	Return a Provenance pulled from the database.
get_provenance_id(provenance, ...[, dbclient])	Return a Provenance ID from either a provenance, or a provenance_tag and process.
get_provs_for_tag(tag[, process, dbclient])	Get the Provenances for a given provenance tag.
parse_provenance(provdict)
recursive_dict([dbclient])
save_to_db([tag, replace_tag, exists, dbclient])	Save this provenance to the database.
spec_dict()
update_id()	Update self.id based on stored properties.

Methods Documentation

classmethod get(process, major, minor, params={}, environment=None, env_major=None, env_minor=None, upstreams=[], exists=None, savetodb=False, dbclient=None)

Get a Provenance based on properties.

Arguments are the same as are passed to the Provenance constructor, plus:

Parameters:

• process (varied) – These are the same as what’s passed to the Provenance constructor

• major (varied) – These are the same as what’s passed to the Provenance constructor

• minor (varied) – These are the same as what’s passed to the Provenance constructor

• params (varied) – These are the same as what’s passed to the Provenance constructor

• environment (varied) – These are the same as what’s passed to the Provenance constructor

• env_major (varied) – These are the same as what’s passed to the Provenance constructor

• env_minor (varied) – These are the same as what’s passed to the Provenance constructor

• exists (bool, default None) – Normally, you get back the provenance you ask for. (This is, depending on savetodb, just the same as instantiating a Provenance object.) However, if exists is True, then it will raise an exception if the provenance isn’t already saved to the database. If exists is False, then it will raise an exception if the provenance is already saved to the database. (Setting exists=False mostly only makes sense when setting savetodb to True.)

• savetodb (bool, default False) – By default, you get the Provenance you ask for, but thedatabse is not changed. Set this to True to save the provenance to the database. If savetodb is True and exists is True, then nothing is saved, because either the provenance already exists, or an exception was raised. If savetodb is True and exists is None, then the provenance will be saved to the database if it doesn’t already exist. If savetodb is True and exists is False, an exception will be raised if the provenance is already in the database, otherwise the new provenance will be saved.

• dbclient (snappl.dbclient.SNPITDBClient) – This is needed to talk to the Roman SNPIT database web server. If not given, will construct one based on config.

classmethod get_by_id(provid, dbclient=None, return_none_if_not_exists=False)

Return a Provenance pulled from the database.

Raises an exception if it does not exist.

Parameters:

• provid (UUID) – The ID to fetch

• dbclient (snappl.dbclient.SNPITDBClient) – This is needed to talk to the Roman SNPIT database web server. If not given, will construct one based on config.

• return_none_if_not_exists (bool, default False)

Returns:

• Provenance

• If return_none_if_not_exists is True, then None will be

• returned if the provenance does not exist in the database. If

• return_none_if_not_exists is False (the default), then an

• exception will be raised if the provenance does not exist in

• the database.

classmethod get_provenance_id(provenance, provenance_tag, process, dbclient=None)

Return a Provenance ID from either a provenance, or a provenance_tag and process.

Parameters:

• provenance (Provenance, str, or None) – If a Provenance, then this provenance’s ID is returned. Otherwise, if this is not None, return this. If None, then fall back to looking at provenance_tag and process.

• provenance_tag (str or None)

• process (str or None)

Return type:

UUID

classmethod get_provs_for_tag(tag, process=None, dbclient=None)

Get the Provenances for a given provenance tag.

Parameters:

• tag (str) – The provenance tag to search

• process (str, default None) – The process to get provenances for. If None, will get all provenances associated with the tag.

• dbclient (snappl.dbclient.SNPITDBClient) – This is needed to talk to the Roman SNPIT database web server. If not given, will construct one based on config.

Returns:

• Provenance or list of Provenance

• If process is not None, you get back a Provenance (or an

• exception is raised if the provenance wasn’t found). If

• process is None, you get back a list of Provenance.

classmethod parse_provenance(provdict)

recursive_dict(dbclient=None)

save_to_db(tag=None, replace_tag=False, exists=None, dbclient=None)

Save this provenance to the database.

Will call self.update_id() as a side effect, just to make sure the right ID is saved to the database.

If you save a provenance with upstreams, those upstreams must have previously been saved themselves. (So, you can’t create a whole provenance tree and have the whole thing saved in one call; it doesn’t recurse.)

Parmaeters

tagstr, default None

Add this provenance to this provenance tag for this process.

replace_tagbool, default False

Ignored if tag is None. If tag is set, but a provenance already exists for this process and tag, then normally that’s an error. If replace_tag is True, delete the old provenance associated with the tag and set the new provenance.

existsbool, default None

If None, and the provenance already exists in the database, do nothing. If False, and the provenance already exists in the database, raise an exception. If True, and the provenance doesn’t already exist in the database, raise an exception. It doesn’t make a lot of sense, usually, to call this method with exists=True.

dbclient: snappl.dbclient.SNPITDBClient

This is needed to talk to the Roman SNPIT database web server. If not given, will construct one based on config.

spec_dict()

update_id()

Update self.id based on stored properties.

If you change any of the properties of the object that define the Provenance, you must call this to make the id property correct. Probably this is a bad idea; you should view Provenance objects as immutable and not change them after you make them.