crandas.script¶
Functionalities to record scripts.
The user can package a crandas script by first executing
crandas.script.record(path=filename)
. Then, every query that is executed will be
recorded in a script. This does not modify the output: the user still needs to
be able to execute the queries, e.g. by being connected to a test environment,
or by setting session.dry_run = True
.
After executing all their queries, the user can run
crandas.script.end()
, which will package the script into a
json-formatted file. The session.analyst_key is included into the file. This file can be sent to the approvers to be signed off.
After receiving the signed query file from the approvers, it can be loaded
using crandas.script.load(filename)
. Now the user, possibly being connected
to a different engine server that requires authorization, can execute the same
script as they executed when recording the script. Note that exactly the same
queries have to be executed as before, in the same order.
- class crandas.script.Script(mode, name=None, auth_tag=None, session=None, map_dummy_handles=False, stepless=False, linear=True, path=None, analyst_key=None, script_extractor=None, include_outputs=False)¶
Bases:
object
A script represents a sequence of queries, that can be automatically recorded, then approved, and later executed on a different engine instance. Within the context of a script, each query is called a “step”, and it is assigned a number 0, 1, 2, …, called its
script_step
.The objective of a script is to internally link the inputs/outputs of operations, so that for example the output of step 0 is the input to step 1.
An active (not-closed) script is either recording or executing. When recording, each crandas command that is executed and sent to the server is locally appended to a list, with a script_step that is one higher than the last script_step. When all of the desired commands have been run, the script can be saved to a JSON-formatted file using
Script.end()
. This script can subsequently be authorized by approvers.An authorized script can be loaded using
Script.load()
. The resulting script is put in executing mode. For each crandas command that is executed, it is expected to exactly match the next command in the script. The matching authorization is then used, and sent along to the server.By default, scripts are recorded and executed in “linear mode”. This means that all steps need to be executed in order and that if a step results in an error during a script execution, the script execution is not allowed to continue. To allow the script to continue after certain errors, a
allow_errors()
block can be used. It is also possible to disable linear mode (this needs to be done both when recording a script and when loading it); in this case, it is in principle allowed to perform the steps of the script in any order and also to continue executing the script in case an error occurred.Besides the above approach of having an increasing script_step for each executed crandas command, scripts may additionally contain
stepless
queries, that do not have any associatedscript_step
. These commands may be recorded once and executed arbitrarily many times, and are useful for commands that need to be executed a variable number of times. To put a recording or executing script in stepless mode, use:>>> script.stepless = True
To get back to regular “stepped” mode, use:
>>> script.stepless = False
When recording, any crandas command that is executed is then added to the script without a script step. When executing, any crandas command is then matched to the stepless queries in the script. In linear mode, a script can still continue in case of an error in a stepless query.
Limitations: since the step numbers are used to link the outputs and inputs of different steps, the outputs of stepless queries cannot be used as the inputs of other (stepless or stepped) queries. Stepless queries are especially useful together with placeholders, e.g.
crandas.placeholders.Any
.- close()¶
Deprecated. Use
Script.end()
instead.
- end()¶
Denotes the end of a script. Either saves the current script (if it is being recorded) or closes it (if it is being executed).
To close a recording without writing to the .recording file, use
crandas.script.reset()
.
- get_inputs() dict ¶
Get the input tables used in the current script.
This returns a dictionary of Reference -> List[str], where the reference maps to a list of descriptions (currently, source code lines) of the queries.
- print_inputs()¶
Print the input tables used in the current script.
- property run_tag: str¶
For each script execution, we generate a random 32-byte run_tag, to distinguish different executions of the same script to the server.
- save(target=None, analyst_key=None)¶
Deprecated. Use
Script.end()
instead. Save the recorded script to a file.- Parameters:
target (str | Path | file-like object, optional) – The path to save the script to, or a file-like object (such as the result of a call to
open
). This should be specified here, or aspath
atScript.record()
.analyst_key (VerifyKey | SigningKey | str | Path | False | True (default)) –
A script will generally include the analyst_key of the user that created the script. The user may specify a signing or verification key to include, either as a key object, or a path to load the key from disk. This should be specified here, or at
Script.record()
.Alternatively, if True (the default), the current
crandas.base.session
analyst_key will be used. If False, no analyst key will be included in the script.
- skip(steps=1)¶
Skip the specified number of steps in the recorded script
- exception crandas.script.ScriptIsActiveWarning¶
Bases:
UserWarning
- enum crandas.script.ScriptMode(value)¶
Bases:
Enum
An enumeration.
Valid values are as follows:
- RECORDING = <ScriptMode.RECORDING: 0>¶
- EXECUTING = <ScriptMode.EXECUTING: 1>¶
- CLOSED = <ScriptMode.CLOSED: 2>¶
- crandas.script.allow_errors(session=None)¶
Context manager for allowing a script to continue in case of errors
Normally, if a script is run in linear mode (see
Script
), and an error occurs any point during the script, then the script cannot be executed further. By running crandas commands in anallow_errors
block, this behaviour is overridden for the commands in this block. In order to do this, the function must be in anallow_errors
block both in during recording and execution.Example usage:
cdf = cd.get_table("people") # Analysis on people > 65 with cd.script.allow_errors(): print(cdf.filter(cdf["age"]>65, threshold=10).describe()) # Analysis on people <= 65, allowed regardless of whether or not previous analysis succeeded with cd.script.allow_errors(): print(cdf.filter(cdf["age"]<=65, threshold=10).describe())
- crandas.script.authorize_query(json_query, handler, session: Session, *, dry_run, _ignore_mode=False)¶
Authorize query if needed (script-aware)
Perform authorization to submit query to server, if needed. Returns query to be sent to server, which can be the original query or, if authorization is needed, a signed query representing it:
If a script is being recorded: no authorization is performed;
If a script is being executed: query is authorized according to script;
If no script is being recorded, and authorizations are loaded into the session: use loaded authorizations
If no script is being recorded, and no authorizations are loaded into the session: no authorization is performed.
Should be called right before sending the query to the server. In the context of a script recording, internally, performs the following steps: - registers objects in query as belonging to the script’s known_objects - add script step to the query (added in-place to argument); - perform actual authorization by calling crandas.auth.authorize_query.
- Parameters:
json_query – Query. May be edited in-place to add script step
handler – Query handler
session – Session in which to execute query
- Raises:
NoMatchingAuthorizationError – No matching authorization found
- Returns:
Query to be sent to engine
- crandas.script.dummy_name_for(handle) str ¶
Returns the name used in testing environments for the table with handle
handle
.
- crandas.script.end(*, session=None)¶
Denote the end of a script.
See
Script.end()
.
- crandas.script.get_frame(session)¶
Get the calling frame using the current ScriptExtractor, when present. Otherwise, returns None.
- crandas.script.is_executing(session=None) bool ¶
Check if a script is being executed
- Parameters:
session (
crandas.base.Session
(optional)) – Session in which to check; if not given,crandas.base.session
is used- Returns:
True
if the session has an active script and this script is being executed- Return type:
bool
- crandas.script.is_recording(session=None) bool ¶
Check if a script is being recorded
- Parameters:
session (
crandas.base.Session
(optional)) – Session in which to check; if not given,crandas.base.session
is used- Returns:
True
if the session has an active script and this script is being recorded- Return type:
bool
- crandas.script.load(path, *, session=None) Script ¶
Load a script from a file for execution.
- Parameters:
path (str) – File name of script to load
- crandas.script.record(*, path=None, name=None, map_dummy_handles=True, analyst_key=None, session=None, include_python_script=None, include_full_script=None, include_outputs=None, linear=True) Script ¶
Start recording a new script.
Use this command before running a sequence of crandas commands. Afterwards, use
crandas.script.end()
to save the script to a file (orcrandas.script.reset()
to reset the script without saving). The script can subsequently be authorized by approvers. When the approved script is received, usecrandas.script.load()
to load it, and then you may execute the exact same sequence of commands, in the same order, as you did when recording the script.- Parameters:
path (str | Path | file-like object) – The path to save the script to, or a file-like object (such as the result of a call to
open
).name (str (optional)) – A name to attach to the script
map_dummy_handles (bool) – Unless the user sets this to False, this causes all hexadecimal table handles that are used during recording to map to the table name
dummy_name(handle)
. To upload tables with this name, save an object with thedummy_for
argument, e.g.,crandas.upload_pandas_dataframe(df).save(dummy_for="<handle>")
.analyst_key (VerifyKey | SigningKey | str | Path | False | True (default)) –
A script will generally include the analyst_key of the user that created the script. The user may specify a signing or verification key to include, either as a key object, or a path to load the key from disk.
Alternatively, if True (the default), the current
crandas.base.session
analyst_key will be used. If False, no analyst key will be included in the script.include_python_script (bool, default True, the default is configurable by
config.settings.include_python_script
, seecrandas.config
) – Automatically include the recorded Python script used for the analysis in the .recording file.include_full_script (bool, default False, the default is configurable by
config.settings.include_full_script
, seecrandas.config
) – Wheninclude_full_script
is True, include the full python script. When False, only include the analysis part of the recorded script, consisting ofscript.record()
up toscript.end()
.include_outputs (bool, default False, the default is configurable by
config.settings.include_outputs
, seecrandas.config
) – Include the outputs revealed by the recorded script on the design data. Similar outputs are revealed when the approved script is executed on production data. Wheninclude_python_script
is False, no positional information about the outputs can be provided.linear (bool, default True) – Record the script in linear mode. Scripts recorded in this mode need to be executed in the same order as they were recorded, without skipping steps and without raising errors. The latter condition can be relaxed by using
allow_errors()
.
- crandas.script.reset(session=None)¶
Reset the current script execution or recording (without saving the recording to file)
- crandas.script.save(target=None, analyst_key=None, *, session=None)¶
Deprecated. Use
crandas.script.end()
instead. Save the recorded script to a file.See
Script.save()
.
- crandas.script.stepless()¶
Context manager to facilitate stepless recording/execution. See
Script
for details onstepless
queries.Example usage:
# Outside the script handles = [crandas.DataFrame({"a": [i]}).handle for i in range(5)] # Inside the script tables = [] with stepless(): for handle in handles: table = cd.get_table(handle, map_dummy_handles=False) tables.append(table) all_rows = crandas.concat(placeholders.Any(tables))