crandas.script

Functionalities to record scripts.

The user can package a crandas script by first executing crandas.script.record(). Then, every query that is executed will be recorded in a script. This does not modify the output: the user still needs to be able to execute the queries, e.g. by being connected to a test environment, or by setting session.dry_run = True.

After executing all their queries, the user can run crandas.script.save(filename), which will package the script into a json-formatted file. The session.analyst_key is included into the file. This file can be sent to the approvers to be signed off.

After receiving the signed query file from the approvers, it can be loaded using crandas.script.load(filename). Now the user, possibly being connected to a different VDL server that requires authorization, can execute the same script as they executed when recording the script. Note that exactly the same queries have to be executed as before, in the same order.

class crandas.script.Script(mode, name=None, auth_tag=None, session=None, map_dummy_handles=False, stepless=False)

Bases: object

A script represents a sequence of VDL commands, that can be automatically recorded, then approved, and later executed on a different VDL instance.

A script consists of a number of “steps”, each referred to by its script_step.

The function of a script is to internally link the inputs/outputs of operations, so that for example the output of step 0 is the input to step 1.

An active (not-closed) script is either recording or executing. When recording, each crandas command that is executed and sent to the server is locally appended to a list, with a script_step that is one higher than the last script_step. When all of the desired commands have been run, the script can be saved to a JSON-formatted file using Script.save(). This script can subsequently be authorized by approvers.

An authorized script can be loaded using Script.load(). The resulting script is put in executing mode. For each crandas command that is executed, it is expected to exactly match the next command in the script. The matching authorization is then used, and sent along to the server.

Besides the above approach of having an increasing script_step for each executed crandas command, scripts may additionally contain stepless queries, that do not have any associated script_step. These commands may be recorded once and executed arbitrarily many times, and are useful for commands that need to be executed a variable number of times. To put a recording or executing script in stepless mode, use:

>>> script.stepless = True

To get back to regular “stepped” mode, use:

>>> script.stepless = False

When recording, any crandas command that is executed is then added to the script without a script step. When executing, any crandas command is then matched to the stepless queries in the script.

Limitations: since the step numbers are used to link the outputs and inputs of different steps, the outputs of stepless queries cannot be used as the inputs of other (stepless or stepped) queries. Stepless queries are especially useful together with placeholders, e.g. crandas.placeholders.Any.

get_inputs()

Get the input tables used in the current script.

This returns a dictionary of Reference -> List[str], where the reference maps to a list of descriptions (currently, source code lines) of the queries.

print_inputs()

Print the input tables used in the current script.

resolve_references(obj)

Recursively modify obj (atom, list, or dict) in-place, replacing each value of type Reference by the corresponding step placeholder.

property run_tag

For each script execution, we generate a random 32-byte run_tag, to distinguish different executions of the same script to the server.

save(target, analyst_key=True)

Save the recorded script to a file.

Parameters:

  • target (str | Path | file-like object) – The path to save the script to, or a file-like object (such as the result of a call to open).

  • analyst_key (VerifyKey | SigningKey | str | Path | False | True (default)) –

    A script will generally include the analyst_key of the user that created the script. The user may specify a signing or verification key to include, either as a key object, or a path to load the key from disk.

    Alternatively, if True (the default), the current crandas.base.session analyst_key will be used. If False, no analyst key will be included in the script.

skip(steps=1)

Skip the specified number of steps in the recorded script

enum crandas.script.ScriptMode(value)

Bases: Enum

An enumeration.

Valid values are as follows:

RECORDING = <ScriptMode.RECORDING: 0>
EXECUTING = <ScriptMode.EXECUTING: 1>
CLOSED = <ScriptMode.CLOSED: 2>
crandas.script.authorize_query(json_query, handler, session, *, _ignore_mode=False)

Authorize query if needed (script-aware)

Perform authorization to submit query to server, if needed. Returns query to be sent to server, which can be the original query or, if authorization is needed, a signed query representing it:

  • If a script is being recorded: no authorization is performed;

  • If a script is being executed: query is authorized according to script;

  • If no script is being recorded, and authorizations are loaded into the session: use loaded authorizations

  • If no script is being recorded, and no authorizations are loaded into the session: no authorization is performed.

Should be called right before sending the query to the server. In the context of a script recording, internally, performs the following steps: - registers a handle hook to let response handler report handle to script; - add script step to the query (added in-place to argument); - perform actual authorization by calling crandas.auth.authorize_query.

Parameters:

  • json_query – Query. May be edited in-place to add script step

  • handler – Query handler

  • session – Session in which to execute query

Raises:

NoMatchingAuthorizationError – No matching authorization found

Returns:

Query to be sent to engine

crandas.script.current_script(session=None)

Returns the currently recording/loaded script.

crandas.script.dummy_name_for(handle)

Returns the name used in testing environments for the table with handle handle.

crandas.script.load(path, *, session=None)

Load a script from a file for execution.

crandas.script.record(name=None, map_dummy_handles=True, *, session=None)

Start recording a new script.

Use this command before running a sequence of crandas commands. Afterwards, use script.save(filename) to save the script to a file. The script can subsequently be authorized by approvers. When the approved script is received, use script.load() to load it, and then you may execute the exact same sequence of commands, in the same order, as you did when recording the script.

Parameters:

  • name (str (optional)) – A name to attach to the script

  • map_dummy_handles (bool) – Unless the user sets this to False, this causes all hexadecimal table handles that are used during recording to map to the table name dummy_name(handle). To upload tables with this name, use the dummy_for argument to a VDL query command, e.g. crandas.upload_pandas_dataframe(df, dummy_for="<handle>").

crandas.script.save(target, analyst_key=True, *, session=None)

Save the recorded script to a file.

See Script.save().

crandas.script.stepless()

Context manager to facilitate stepless recording/execution. See Script for details on stepless queries.

Example usage:

# Outside the script
handles = [crandas.DataFrame({"a": [i]}).handle for i in range(5)]

# Inside the script
tables = []
with stepless():
    for handle in handles:
        table = cd.get_table(handle, map_dummy_handles=False)
        tables.append(table)

all_rows = crandas.concat(placeholders.Any(tables))