Data types

This section provides an overview of which data types the Virtual Data Lake (VDL) supports and how we can convert between data types using crandas.

The supported data types include:

  • int8, int16, int24, int32

  • uint8, uint16, uint24, uint32

  • vector of integers

  • string

  • bytes

  • bool (as integer)

  • fixed-point numbers [1]

Added in version 1.7: Fixed-point numbers

For specific details about numeric values, see the next section.

Warning

When boolean values are uploaded into the VDL, they are transformed to integers and therefore take the values 0 and 1 instead of False and True respectively.

Integer and string values allow for missing values, yet this must be specified

Data Type Conversion

Crandas provides a method for converting data types using the CSeries.astype() method. In the following example, we will show you how to convert a column of strings to a column of integers.

Note

This is the only type conversion currently supported.

import crandas as cd

# Create a crandas DataFrame with a string column
uploaded = cd.DataFrame({"vals": ["1","2","3","4"]})

# Convert the string column to a int column
uploaded = uploaded.assign(vals2=uploaded["vals"].astype(int))

The above example converts the string column vals to a new integer column called vals2.

For a more in depth look at specifying integer types, go to the next section.

Note

It is also possible to specify the desired type while uploading the data, using ctype={"val": "varchar[9]"}.

ctypes

Because the VDL uses highly specialized algorithms to compute on secret data, it uses a specialized typing system that is more fine-grained than what pandas uses. Crandas implements this type system, which we call ctypes (similarly to pandas’ dtypes). In certain situations, it is important to specify the specific type of our data

import pandas as pd
from crandas import ctypes

# Specify data types for the DataFrame
table = cd.DataFrame(
    {"ints": [1, 2, 3], "strings": ["a", "bb", "ccc"]},
    ctype={"ints": "int8", "strings": "varchar[5]"},
)

In the above example, we define the ints column with a NonNullableInteger data type (crandas.ctypes.NonNullableInteger()), and the strings column is defined with a varchar[5] data type (variable but less than 5 characters).

Note

If there are missing/null values in the column that can be specified by adding ? after the ctype (i.e. int8?)

Crandas also supports other data types, such as byte arrays:

from uuid import uuid4

# Create a DataFrame with UUIDs stored as bytes
df = cd.DataFrame({"uuids": [uuid4().bytes for _ in range(5)]}, ctype="bytes")

You are also able to specify types through pandas’ typing, known as dtypes. Note that not all dtypes have an equivalent ctypes.

# Create a DataFrame with multiple data types
df = cd.DataFrame(
    {
        "strings": pd.Series(["test", "hoi", "ok"], dtype="string"),
        "int": pd.Series([1, 2, 3], dtype="int"),
        "int64": pd.Series([23, 11, 91238], dtype="int64"),
        "int32": pd.Series([12831, 1231, -1231], dtype="int32"),
    }
)

Working with missing values

Crandas can work with null values, although this requires extra care. Columns do not allow null values by default but this can be achieved in multiple ways. Whenever a column with missing values is added, the VDL will determine that such column can have null values. Additionally, it is possible to specify that a column will allow null values when uploading it, even if the column currently does not contain any such values.

Warning

When uploading an integer column with missing values, it is necessary to add the ctype of the column. If there are already missing values it is not necessary to specify ctypes.NullableInteger(), specifying the size of the integer will be enough e.g ctype={"int_column": int8}.

from crandas import ctypes


table = cd.DataFrame(
    {"ints": [1, None, 3], "strings": ["a", "bb", None]},
    ctype={"ints": "int32?, "strings": "varchar[5]"},
)

Note

When specifying ctypes for columns with missing values you can use “int32?” or any other supported data type with ? (this indicates missing values).

Both columns created in this example allow for null values. The first one because it was explictly specified and the latter because it contains a null value. Crandas considers the same values to be null as pandas; in particular, this includes None, pandas.NA, and numpy.nan.

To turn a nullable column into a non-nullable one, the fillna function can be used. For example, the following code example replaces all missing values of a string column by the string empty:

import crandas as cd

cdf = cd.DataFrame({"a": ["test", None]})
cdf = cdf.assign(a=lambda x: x.a.fillna("empty"))

Numeric types have additional particularities that are important to know, both in the typing system and because we can do arithmetic operations over them. The next section deals with these types.