.. _datatypes:

Data types
##########

This section provides an overview of which data types used by the engine and how we can convert between data types using crandas.

The supported data types include:

- ``int8``, ``int16``, ``int24``, ``int32``, ``int40``, ``int48``, ``int56``, ``int64``, ``int72``, ``int80``, ``int88``, ``int96``
- ``fp8``, ``fp16``, ``fp24``, ``fp32``, ``fp40``, ``fp48``, ``fp56``, ``fp64``, ``fp72``, ``fp80``, ``fp88``, ``fp96``
- ``uint8``, ``uint16``, ``uint24``, ``uint32``, ``uint40``, ``uint48``, ``uint56``, ``uint64``, ``uint72``, ``uint80``, ``uint88``, ``uint96``
- ``vec_int`` (vector of integers)
- ``varchar`` (text)
- ``date``
- ``bytes`` (binary data)
- ``bool`` (as integer)

.. versionadded:: 1.10

    Fixed-point numbers from ``fp40`` and bigger

.. versionadded:: 1.8

    Integers from ``int40``and bigger and dates.

.. versionadded:: 1.7

    Fixed-point numbers

For specific details about numeric values, see the :ref:`next section<numeric>`.

For each data type, also a "nullable" variant is supported that can hold missing values.
Nullable data data types are denoted with a question mark, e.g., ``int8?``, ``bytes??``.
Nullable data types have some caveats; :ref:`read on<null_values>` or go to :ref:`their own section<nullable>` for details.

.. warning::

    When boolean values are uploaded into the engine, they are transformed to integers and therefore take the values ``0`` and ``1`` instead of ``False`` and ``True`` respectively.


:class:`.CDataFrame` allow for missing values, but not by default, so they must be specified.

Data Type Conversion
=====================

Crandas provides a method for converting data types using the :meth:`.CSeries.astype` method. In the following example, we will show you how to convert a column of strings to a column of integers.

.. note::

    This is the only type conversion currently supported.

.. code:: python

    import crandas as cd

    # Create a crandas DataFrame with a string column
    uploaded = cd.DataFrame({"vals": ["1","2","3","4"]})

    # Convert the string column to a int column
    uploaded = uploaded.assign(vals2=uploaded["vals"].astype(int))


The above example converts the string column ``vals`` to a new integer column called ``vals2``.

For a more in depth look at specifying integer types, go to the :ref:`next section<numeric>`.

.. note::

    It is also possible to specify the desired type while uploading the data, using ``ctype={"val": "varchar[9]"}``.


.. _ctypes_section:

ctypes
======

Because the engine uses highly specialized algorithms to compute on secret data, it uses a specialized typing system that is more fine-grained than what pandas uses.
Crandas implements this type system, which we call :ref:`ctypes<Ctypes>` (similarly to pandas' ``dtypes``).
In certain situations, it is important to specify the specific type of our data

.. code:: python

    import pandas as pd
    from crandas import ctypes

    # Specify data types for the DataFrame
    table = cd.DataFrame(
        {"ints": [1, 2, 3], "strings": ["a", "bb", "ccc"]},
        ctype={"ints": "int8", "strings": "varchar[5]"},
    )

In the above example, we define the ``ints`` column with a ``NonNullableInteger`` data type (:meth:`crandas.ctypes.NonNullableInteger`), and the ``strings`` column is defined with a ``varchar[5]`` data type (string with at most 5 characters).

.. hint::

    If there are missing/null values in the column that can be specified by adding ``?`` after the ctype (e.g. ``int8?``, ``varchar?[5]``)

Crandas also supports other data types, such as byte arrays:

.. code:: python

    from uuid import uuid4

    # Create a DataFrame with UUIDs stored as bytes
    df = cd.DataFrame({"uuids": [uuid4().bytes for _ in range(5)]}, ctype="bytes")

You are also able to specify types through pandas' typing, known as dtypes. Note that not all dtypes have an equivalent ctypes.

.. code:: python

    # Create a DataFrame with multiple data types
    df = cd.DataFrame(
        {
            "strings": pd.Series(["test", "hoi", "ok"], dtype="string"),
            "int": pd.Series([1, 2, 3], dtype="int"),
            "int64": pd.Series([23, 11, 91238], dtype="int64"),
            "int32": pd.Series([12831, 1231, -1231], dtype="int32"),
        }
    )

.. _type_detection:

Type detection from data
========================

For some types of column, crandas can derive the exact column type from the data being uploaded.
This may lead to ``ColumnBoundDerivedWarning`` warnings, e.g.:

.. code:: text

    ColumnBoundDerivedWarning: Column type for column a (uint8) was automatically
    derived from the input data, see User Guide->Data types->Automatic type detection
    for more information.

This warning is given if no exact ctype (e.g., ``uint8``, ``varchar[ascii]``) is given for the column.

For example, for an integer column, if no size is given, the smallest integer type is selected
that is valid for the column, e.g.:

.. code:: python

    cdf = cd.DataFrame({"vals": [1,2,3]})

In this example, the column ``vals`` will be derived to be of type ``uint8`` (unsigned 8-bit integer)
because all values lie in the range from 0 to 255 (inclusive). See :ref:`int_type_detection`.

Similarly, for varchar (text) columns, it will be derived from the data whether the column is of
ASCII or unicode type. See :ref:`string_ascii`.

Note that, because the column type is derived from the data, this potentially leaks information.
For example, if one party uploads a list of salaries that are all smaller than ``65536`` and another
party uploads a list of salaries that contains higher salaries, then the first column will have
column type ``uint16`` and the second column will have column ``uint24``. The first party can prevent
this by explicitly assigning the ctype ``uint24`` to the uploaded column.

When type information is detected from data, the user gets a warning about this, e.g.,
``ColumnBoundDerivedWarning: Column "vals" was automatically derived to be of type uint8``.
These warnings can be suppressed with the standard Python mechanism, e.g.,

.. code:: python

    import warnings

    # Suppress warnings about automatically derived column types
    warnings.simplefilter("ignore", category=cd.ctypes.ColumnBoundDerivedWarning)

    cdf = cd.DataFrame({"vals": [1,2,3]})

It is also possible to provide `auto_bounds=True` as argument to data uploading functions, or to set
`auto_bounds=True`` globally for a crandas session:

.. code:: python

    import crandas as cd

    # Suppress ColumnBoundDerivedWarning globally...
    cd.base.session.auto_bounds = True

    # or for a single upload
    cdf = cd.DataFrame({"vals": [1,2,3]}, auto_bounds = True)


.. _null_values:

Working with missing values
=============================

Crandas can work with null values, although this requires extra care. Columns do not allow null values by default but this can be achieved in multiple ways. Whenever a column with missing values is added, the engine will determine that such column can have null values. Additionally, it is possible to specify that a column will allow null values when uploading it, even if the column currently does not contain any such values.

.. warning::

    When using a column with missing values in combination with script signing, it is advisable to explicitly specify that the column allows null values, by defining the ``ctype`` as discussed below.
    This way, there will not be a mismatch between the approved analysis and the performed analysis, even if the dummy or actual data does not contain nullable values.
    For example, the following code designates the column ``ints`` as allowing missing values, even if none of the uploaded values are missing.

.. code:: python

    from crandas import ctypes


    table = cd.DataFrame(
        {"ints": [1, 2, 3], "strings": ["a", "bb", None]},
        ctype={"ints": "int32?"},
    )

.. hint::

    When specifying ctypes for columns with missing values you can use ``int32?`` or any other supported data type with ``?`` (this indicates missing values).

Both columns created in this example allow for null values. The first one because it was explictly specified and the latter because it contains a null value.
Crandas considers the same values to be null as pandas; in particular, this includes ``None``, ``pandas.NA``, and ``numpy.nan``.

To turn a nullable column into a non-nullable one, the ``fillna`` function can be used. For example, the following code example replaces all missing values of a string column by the string ``empty``:

.. code:: python

    import crandas as cd

    cdf = cd.DataFrame({"a": ["test", None]})
    cdf = cdf.assign(a=lambda x: x.a.fillna("empty"))

Numeric types have additional particularities that are important to know, both in the typing system and because we can do arithmetic operations over them. The next section deals with these types.