Direct constructors (fastest)#
If you’re willing to think about your data in a columnar way, directly constructing layouts and wrapping them in ak.Array interfaces is the fastest way to make them. (All other methods do this at some level.)
“Thinking about data in a columnar way” is the crucial difference between this method and using ArrayBuilder. The builder method lets you think about a data structure the way you would think about Python objects, in which all fields of a given record or elements of a list are “together” and one record or list is “separate” from another record or list. For example,
import awkward as ak
import numpy as np
builder = ak.ArrayBuilder()
with builder.list():
with builder.record():
builder.field("x").real(1.1)
with builder.field("y").list():
builder.integer(1)
with builder.record():
builder.field("x").real(2.2)
with builder.field("y").list():
builder.integer(1)
builder.integer(2)
with builder.record():
builder.field("x").real(3.3)
with builder.field("y").list():
builder.integer(1)
builder.integer(2)
builder.integer(3)
with builder.list():
pass
with builder.list():
with builder.record():
builder.field("x").real(4.4)
with builder.field("y").list():
builder.integer(3)
builder.integer(2)
with builder.record():
builder.field("x").real(5.5)
with builder.field("y").list():
builder.integer(3)
array = builder.snapshot()
array
[[{x: 1.1, y: [1]}, {x: 2.2, y: [...]}, {x: 3.3, y: [1, 2, 3]}],
[],
[{x: 4.4, y: [3, 2]}, {x: 5.5, y: [3]}]]
----------------------------------------------------------------
type: 3 * var * {
x: float64,
y: var * int64
}array.to_list()
[[{'x': 1.1, 'y': [1]}, {'x': 2.2, 'y': [1, 2]}, {'x': 3.3, 'y': [1, 2, 3]}],
[],
[{'x': 4.4, 'y': [3, 2]}, {'x': 5.5, 'y': [3]}]]
gets all of the items in the first list separately from the items in the second or third lists, and both fields of each record (x and y) are expressed near each other in the flow of the code and in the times when they’re appended.
By contrast, the physical data are laid out in columns, with all x values next to each other, regardless of which records or lists they’re in, and all the y values next to each other in another buffer.
array.layout
<ListOffsetArray len='3'>
<offsets><Index dtype='int64' len='4'>[0 3 3 5]</Index></offsets>
<content><RecordArray is_tuple='false' len='5'>
<content index='0' field='x'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1' field='y'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray></content>
</ListOffsetArray>
To build arrays using the layout constructors, you need to be able to write them in a form similar to the above, with the data already arranged as columns, in a tree representing the type structure of items in the array, not separate trees for each array element.
The Awkward Array library has a closed set of node types. Building the array structure you want will require you to understand the node types that you use.
The node types (with validity rules for each) are documented under ak.contents.Content, but this tutorial will walk through them explaining the situations in which you’d want to use each.
Content classes#
ak.contents.Content is the abstract superclass of all node types. All Content nodes that you would create are concrete subclasses of this class. The superclass is useful for checking isinstance(some_object, ak.contents.Content), since there are some attributes that are only allowed to be Content nodes.
Sections in this document about a subclass of Content are named “Content >: XYZ” (using the “is superclass of” operator).
Parameters#
Each layout node can have arbitrary metadata[1], called “parameters.” Some parameters have built-in meanings, which are described below, and others can be given meanings by defining functions in ak.behavior.
Index classes#
ak.index.Index instances are buffers of integers that are used to give structure to an array. For instance, the offsets in the ListOffsetArrays, above, are Indexes, but the NumpyArray of y list contents are not. ak.contents.NumpyArray is a subclass of ak.contents.Content, and ak.index.Index is not. Indexes are more restricted than general NumPy arrays (must be one-dimensional, C-contiguous integers; dtypes are also prescribed) because they are frequently manipulated by Awkward Array operations, such as slicing.
There are five Index specializations, and each ak.contents.Content subclass has limitations on which ones it can use.
Index8: an Index of signed 8-bit integers
IndexU8: an Index of unsigned 8-bit integers
Index32: an Index of signed 32-bit integers
IndexU32: an Index of unsigned 32-bit integers
Index64: an Index of signed 64-bit integers
Content >: EmptyArray#
ak.contents.EmptyArray is one of the two possible leaf types of a layout tree; the other is ak.contents.NumpyArray (A third, corner-case “leaf type” is a ak.contents.RecordArray with zero fields).
EmptyArray is a trivial node type: it can only represent empty arrays with unknown type. It is an identity — when merging an array against an empty array, the empty array has no effect upon the result type. As such, this node cannot have user-defined parameters; ak.contents.EmptyArray.parameters is always empty.
ak.contents.EmptyArray()
<EmptyArray len='0'/>
ak.Array(ak.contents.EmptyArray())
----------------- type: 0 * unknown
Content >: NumpyArray#
ak.contents.NumpyArray is one of the two possible leaf types of a layout tree; the other is ak.contents.EmptyArray. (A third, corner-case “leaf type” is a ak.contents.RecordArray with zero fields)
NumpyArray represents data the same way as a NumPy np.ndarray. That is, it can be multidimensional, but only rectilinear arrays.
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
ak.contents.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]], np.int16))
<NumpyArray dtype='int16' shape='(2, 3)'>
[[1 2 3]
[4 5 6]]
</NumpyArray>
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])[::2])
<NumpyArray dtype='float64' len='3'>[1.1 3.3 5.5]</NumpyArray>
ak.contents.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]], np.int16)[:, 1:])
<NumpyArray dtype='int16' shape='(2, 2)'>
[[2 3]
[5 6]]
</NumpyArray>
In most array structures, the NumpyArrays only need to be 1-dimensional, since regular-length dimensions can be represented by ak.contents.RegularArray and variable-length dimensions can be represented by ak.contents.ListArray or ak.contents.ListOffsetArray.
The ak.from_numpy function has a regulararray argument to choose between putting multiple dimensions into the NumpyArray node or nesting a 1-dimensional NumpyArray in RegularArray nodes.
ak.from_numpy(
np.array([[1, 2, 3], [4, 5, 6]], np.int16), regulararray=False, highlevel=False
)
<NumpyArray dtype='int16' shape='(2, 3)'>
[[1 2 3]
[4 5 6]]
</NumpyArray>
ak.from_numpy(
np.array([[1, 2, 3], [4, 5, 6]], np.int16), regulararray=True, highlevel=False
)
<RegularArray size='3' len='2'>
<content><NumpyArray dtype='int16' len='6'>[1 2 3 4 5 6]</NumpyArray></content>
</RegularArray>
All of these representations look the same in an ak.Array (high-level view).
ak.Array(ak.contents.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]])))
[[1, 2, 3], [4, 5, 6]] ------------------- type: 2 * 3 * int64
ak.Array(
ak.contents.RegularArray(ak.contents.NumpyArray(np.array([1, 2, 3, 4, 5, 6])), 3)
)
[[1, 2, 3], [4, 5, 6]] ------------------- type: 2 * 3 * int64
If you are producing arrays, you can pick any representation that is convenient. If you are consuming arrays, you need to be aware of the different representations.
Since this is such a simple node type, let’s use it to show examples of adding parameters.
ak.Array(
ak.contents.NumpyArray(
np.array([[1, 2, 3], [4, 5, 6]]),
parameters={"name1": "value1", "name2": {"more": ["complex", "value"]}}
)
)
[[1, 2, 3],
[4, 5, 6]]
----------------------------------------------------------------------------------------------
type: 2 * [3 * int64, parameters={"name1": "value1", "name2": {"more": ["complex", "value"]}}]Content >: RegularArray#
ak.contents.RegularArray represents regular-length lists (lists with all the same length). This was shown above as being equivalent to dimensions in a ak.contents.NumpyArray, but it can also contain irregular data.
layout = ak.contents.RegularArray(
ak.from_iter([1, 2, 3, 4, 5, 6], highlevel=False),
3,
)
layout
<RegularArray size='3' len='2'>
<content><NumpyArray dtype='int64' len='6'>[1 2 3 4 5 6]</NumpyArray></content>
</RegularArray>
ak.Array(layout)
[[1, 2, 3], [4, 5, 6]] ------------------- type: 2 * 3 * int64
layout = ak.contents.RegularArray(
ak.from_iter(
[[], [1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5]], highlevel=False
),
3,
)
layout
<RegularArray size='3' len='2'>
<content><ListOffsetArray len='6'>
<offsets><Index dtype='int64' len='7'>
[ 0 0 1 3 6 10 15]
</Index></offsets>
<content><NumpyArray dtype='int64' len='15'>
[1 1 2 1 2 3 1 2 3 4 1 2 3 4 5]
</NumpyArray></content>
</ListOffsetArray></content>
</RegularArray>
ak.Array(layout)
[[[], [1], [1, 2]], [[1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5]]] -------------------------------------------- type: 2 * 3 * var * int64
The data type for a RegularArray is ak.types.RegularType, printed above as the “3 *” in the type above. (The “2 *” describes the length of the array itself, which is always “regular” in the sense that there’s only one of them, equal to itself.)
The “var *” is the type of variable-length lists, nested inside of the RegularArray.
RegularArray is the first array type that can have unreachable data: the length of its nested content might not evenly divide the RegularArray’s regular size.
ak.Array(
ak.contents.RegularArray(
ak.contents.NumpyArray(np.array([1, 2, 3, 4, 5, 6, 7])),
3,
)
)
[[1, 2, 3], [4, 5, 6]] ------------------- type: 2 * 3 * int64
In the high-level array, we only see [[1, 2, 3], [4, 5, 6]] and not 7. Since the 7 items in the nested NumpyArray can’t be subdivided into lists of length 3. This 7 exists in the underlying physical data, but in the high-level view, it is as though it did not.
Content >: ListArray#
ak.contents.ListArray and ak.contents.ListOffsetArray are the two node types that describe variable-length lists (ak.types.ListType, represented in type strings as “var *”). ak.contents.ListArray is the most general. It takes two Indexes, starts and stops, which indicate where each nested list starts and stops.
layout = ak.contents.ListArray(
ak.index.Index64(np.array([0, 3, 3])),
ak.index.Index64(np.array([3, 3, 5])),
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
)
layout
<ListArray len='3'>
<starts><Index dtype='int64' len='3'>[0 3 3]</Index></starts>
<stops><Index dtype='int64' len='3'>[3 3 5]</Index></stops>
<content><NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray></content>
</ListArray>
ak.Array(layout)
[[1.1, 2.2, 3.3], [], [4.4, 5.5]] ----------------------- type: 3 * var * float64
The nested content, [1.1, 2.2, 3.3, 4.4, 5.5] is divided into three lists, [1.1, 2.2, 3.3], [], [4.4, 5.5] by starts=[0, 3, 3] and stops=[3, 3, 5]. That is to say, the first list is drawn from indexes 0 through 3 of the content, the second is empty (from 3 to 3), and the third is drawn from indexes 3 through 5.
Content >: ListOffsetArray#
ak.contents.ListOffsetArray and ak.contents.ListArray are the two node types that describe variable-length lists (ak.types.ListType, represented in type strings as “var *”). ak.contents.ListOffsetArray is an important special case, in which
starts = offsets[:-1]
stops = offsets[1:]
for a single array offsets. If we were only representing arrays and not doing computations on them, we would always use ListOffsetArrays, because they are the most compact. Knowing that a node is a ListOffsetArray can also simplify the implementation of some operations. In a sense, operations that produce ListArrays (previous section) can be thought of as delaying the part of the operation that would propagate down into the content, as an optimization.
layout = ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 3, 3, 5])),
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
)
layout
<ListOffsetArray len='3'>
<offsets><Index dtype='int64' len='4'>[0 3 3 5]</Index></offsets>
<content><NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray></content>
</ListOffsetArray>
ak.Array(layout)
[[1.1, 2.2, 3.3], [], [4.4, 5.5]] ----------------------- type: 3 * var * float64
The length of the offsets array is one larger than the length of the array itself; an empty array has an offsets of length 1.
However, the offsets does not need to start at 0 or stop at len(content).
ak.Array(
ak.contents.ListOffsetArray(
ak.index.Index64(np.array([1, 3, 3, 4])),
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
)
)
[[2.2, 3.3], [], [4.4]] ----------------------- type: 3 * var * float64
In the above, offsets[0] == 1 means that the 1.1 in the content is unreachable, and offsets[-1] == 4 means that the 5.5 is unreachable. Just as in ListArrays, unreachable data in ListOffsetArrays usually comes about because we don’t want computations to always propagate all the way down large trees.
Nested lists#
As with all non-leaf Content nodes, arbitrarily deep nested lists can be built by nesting RegularArrays, ListArrays, and ListOffsetArrays. In each case, the starts, stops, or offsets only index the next level down in structure.
For example, here is an array of 5 lists, whose length is approximately 20 each.
layout = ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 18, 42, 59, 83, 100])),
ak.contents.NumpyArray(np.arange(100)),
)
array = ak.Array(layout)
array[0], array[1], array[2], array[3], array[4]
(<Array [0, 1, 2, 3, 4, 5, 6, ..., 11, 12, 13, 14, 15, 16, 17] type='18 * int64'>,
<Array [18, 19, 20, 21, 22, 23, ..., 36, 37, 38, 39, 40, 41] type='24 * int64'>,
<Array [42, 43, 44, 45, 46, 47, ..., 53, 54, 55, 56, 57, 58] type='17 * int64'>,
<Array [59, 60, 61, 62, 63, 64, ..., 77, 78, 79, 80, 81, 82] type='24 * int64'>,
<Array [83, 84, 85, 86, 87, 88, ..., 94, 95, 96, 97, 98, 99] type='17 * int64'>)
Making an array of 3 lists that contains these 5 lists requires us to make indexes that go up to 5, not 100.
array = ak.Array(
ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 3, 3, 5])),
layout,
)
)
array[0], array[1], array[2]
(<Array [[0, 1, 2, 3, 4, ..., 13, 14, 15, 16, 17], ...] type='3 * var * int64'>,
<Array [] type='0 * var * int64'>,
<Array [[59, 60, 61, 62, 63, ..., 79, 80, 81, 82], ...] type='2 * var * int64'>)
Strings and bytestrings#
As described above, any Content node can have any parameters, but some have special meanings. Most parameters are intended to associate runtime behaviors with data structures, the way that methods add computational abilities to a class. Unicode strings and raw bytestrings are an example of this.
Awkward Array has no “StringArray” type because such a thing would be stored, sliced, and operated upon in most circumatances as a list-type array (RegularArray, ListArray, ListOffsetArray) of bytes. A parameter named “__array__” adds behaviors, such as the string interpretation, to arrays.
"__array__": "byte"interprets anak.contents.NumpyArrayas raw bytes in a bytestring,"__array__": "char"interprets anak.contents.NumpyArrayas characters in a Unicode-encoded string,"__array__": "bytestring"interprets a list-type array (ak.contents.RegularArray,ak.contents.ListArray,ak.contents.ListOffsetArray) as a raw bytestring,"__array__": "string"interprets a list-type array (ak.contents.RegularArray,ak.contents.ListArray,ak.contents.ListOffsetArray) as a Unicode-encoded string.
The ak.contents.NumpyArray must be directly nested within the list-type array, which can be checked with ak.is_valid() and ak.validity_error().
Here is an example of a raw bytestring:
ak.Array(
ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 3, 8, 11, 15])),
ak.contents.NumpyArray(
np.array(
[
104,
101,
121,
116,
104,
101,
114,
101,
121,
111,
117,
103,
117,
121,
115,
],
np.uint8,
),
parameters={"__array__": "byte"},
),
parameters={"__array__": "bytestring"},
)
)
[b'hey', b'there', b'you', b'guys'] --------------- type: 4 * bytes
And here is an example of a Unicode-encoded string (UTF-8):
ak.Array(
ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 3, 12, 15, 19])),
ak.contents.NumpyArray(
np.array(
[
104,
101,
121,
226,
128,
148,
226,
128,
148,
226,
128,
148,
121,
111,
117,
103,
117,
121,
115,
],
np.uint8,
),
parameters={"__array__": "char"},
),
parameters={"__array__": "string"},
)
)
['hey', '———', 'you', 'guys'] ---------------- type: 4 * string
As with any other lists, strings can be nested within lists. Only the ak.contents.RegularArray/ak.contents.ListArray/ak.contents.ListOffsetArray corresponding to the strings should have the "__array__": "string" or "bytestring" parameter.
ak.Array(
ak.contents.ListOffsetArray(
ak.index.Index64([0, 2, 4]),
ak.contents.ListOffsetArray(
ak.index.Index64(np.array([0, 3, 12, 15, 19])),
ak.contents.NumpyArray(
np.array(
[
104,
101,
121,
226,
128,
148,
226,
128,
148,
226,
128,
148,
121,
111,
117,
103,
117,
121,
115,
],
np.uint8,
),
parameters={"__array__": "char"},
),
parameters={"__array__": "string"},
),
)
)
[['hey', '———'], ['you', 'guys']] ---------------------- type: 2 * var * string
Content >: RecordArray#
ak.contents.RecordArray and ak.contents.UnionArray are the only two node types that have multiple contents, not just a single content (and the property is pluralized to reflect this fact). RecordArrays represent a “product type,” data containing records with fields x, y, and z have x’s type AND y’s type AND z’s type, whereas UnionArrays represent a “sum type,” data that are x’s type OR y’s type OR z’s type.
RecordArrays have no ak.index.Index-valued properties; they may be thought of as metadata-only groupings of Content nodes. Since the RecordArray node holds an array for each field, it is a “struct of arrays,” rather than an “array of structs.”
RecordArray fields are ordered and provided as an ordered list of contents and field names (the recordlookup).
layout = ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
[
"x",
"y",
],
)
layout
<RecordArray is_tuple='false' len='5'>
<content index='0' field='x'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1' field='y'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray>
ak.Array(layout)
[{x: 1.1, y: [1]},
{x: 2.2, y: [1, 2]},
{x: 3.3, y: [1, 2, 3]},
{x: 4.4, y: [3, 2]},
{x: 5.5, y: [3]}]
------------------------
type: 5 * {
x: float64,
y: var * int64
}ak.to_list(layout)
[{'x': 1.1, 'y': [1]},
{'x': 2.2, 'y': [1, 2]},
{'x': 3.3, 'y': [1, 2, 3]},
{'x': 4.4, 'y': [3, 2]},
{'x': 5.5, 'y': [3]}]
RecordArray fields do not need to have names. If the recordlookup is None, the RecordArray is interpreted as an array of tuples. (The word “tuple,” in statically typed environments, usually means a fixed-length type in which each element may be a different type.)
layout = ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
None,
)
layout
<RecordArray is_tuple='true' len='5'>
<content index='0'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray>
ak.Array(layout)
[(1.1, [1]),
(2.2, [1, 2]),
(3.3, [1, 2, 3]),
(4.4, [3, 2]),
(5.5, [3])]
------------------
type: 5 * (
float64,
var * int64
)ak.to_list(layout)
[(1.1, [1]), (2.2, [1, 2]), (3.3, [1, 2, 3]), (4.4, [3, 2]), (5.5, [3])]
Since the RecordArray node holds an array for each of its fields, it is possible for these arrays to have different lengths. In such a case, the length of the RecordArray can be given explicitly or it is taken to be the length of the shortest field-array.
content0 = ak.contents.NumpyArray(np.array([1, 2, 3, 4, 5, 6, 7, 8]))
content1 = ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
content2 = ak.from_iter(
[[1], [1, 2], [1, 2, 3], [3, 2, 1], [3, 2], [3]], highlevel=False
)
print(f"{len(content0) = }, {len(content1) = }, {len(content2) = }")
layout = ak.contents.RecordArray([content0, content1, content2], ["x", "y", "z"])
print(f"{len(layout) = }")
len(content0) = 8, len(content1) = 5, len(content2) = 6
len(layout) = 5
layout = ak.contents.RecordArray(
[content0, content1, content2], ["x", "y", "z"], length=3
)
print(f"{len(layout) = }")
len(layout) = 3
RecordArrays are also allowed to have zero fields. This is an unusual case, but it is one that allows a RecordArray to be a leaf node (like ak.contents.EmptyArray and ak.contents.NumpyArray). If a RecordArray has no fields, a length must be given.
ak.Array(ak.contents.RecordArray([], [], length=5))
[{},
{},
{},
{},
{}]
-----------
type: 5 * {
}ak.Array(ak.contents.RecordArray([], None, length=5))
[(),
(),
(),
(),
()]
-----------
type: 5 * (
)Scalar Records#
An ak.contents.RecordArray is an array of records. Just as you can extract a scalar number from an array of numbers, you can extract a scalar record. Unlike numbers, records may still be sliced in some ways like Awkward Arrays:
array = ak.Array(
ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
[
"x",
"y",
],
)
)
record = array[2]
record
{x: 3.3,
y: [1, 2, 3]}
------------------
type: {
x: float64,
y: var * int64
}record["y", -1]
np.int64(3)
Therefore, we need an ak.record.Record type, but this Record is not an array, so it is not a subclass of ak.contents.Content.
Due to the columnar orientation of Awkward Array, a RecordArray does not contain Records, a Record contains a RecordArray.
record.layout
<Record at='2'>
<array><RecordArray is_tuple='false' len='5'>
<content index='0' field='x'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1' field='y'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray></array>
</Record>
It can be built by passing a RecordArray as its first argument and the item of interest in its second argument.
layout = ak.record.Record(
ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
[
"x",
"y",
],
),
2,
)
record = ak.Record(layout) # note the high-level ak.Record, rather than ak.Array
record
{x: 3.3,
y: [1, 2, 3]}
------------------
type: {
x: float64,
y: var * int64
}Naming record types#
The records discussed so far are generic. Naming a record not only makes it easier to read type strings, it’s also how ak.behavior overloads functions and adds methods to records as though they were classes in object-oriented programming.
A name is given to an ak.contents.RecordArray node through its “__record__” parameter.
layout = ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
[
"x",
"y",
],
parameters={"__record__": "Special"},
)
layout
<RecordArray is_tuple='false' len='5'>
<parameter name='__record__'>'Special'</parameter>
<content index='0' field='x'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1' field='y'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray>
ak.Array(layout)
[{x: 1.1, y: [1]},
{x: 2.2, y: [1, 2]},
{x: 3.3, y: [1, 2, 3]},
{x: 4.4, y: [3, 2]},
{x: 5.5, y: [3]}]
------------------------
type: 5 * Special[
x: float64,
y: var * int64
]ak.type(layout)
ArrayType(RecordType([NumpyType('float64'), ListType(NumpyType('int64'))], ['x', 'y'], parameters={'__record__': 'Special'}), 5, None)
Behavioral overloads are presented in more depth in ak.behavior, but here are three examples:
ak.behavior[np.sqrt, "Special"] = lambda special: np.sqrt(special.x)
np.sqrt(ak.Array(layout))
[1.05, 1.48, 1.82, 2.1, 2.35] ----------------- type: 5 * float64
class SpecialRecord(ak.Record):
def len_y(self):
return len(self.y)
ak.behavior["Special"] = SpecialRecord
ak.Record(layout[2]).len_y()
3
class SpecialArray(ak.Array):
def len_y(self):
return ak.num(self.y)
ak.behavior["*", "Special"] = SpecialArray
ak.Array(layout).len_y()
[1, 2, 3, 2, 1] --------------- type: 5 * int64
Content >: IndexedArray#
ak.contents.IndexedArray is the only Content node that has exactly the same type as its content. An IndexedArray rearranges the elements of its content, as though it were a lazily applied array-slice.
layout = ak.contents.IndexedArray(
ak.index.Index64(np.array([2, 0, 0, 1, 2])),
ak.contents.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3])),
)
layout
<IndexedArray len='5'>
<index><Index dtype='int64' len='5'>[2 0 0 1 2]</Index></index>
<content><NumpyArray dtype='float64' len='4'>[0. 1.1 2.2 3.3]</NumpyArray></content>
</IndexedArray>
ak.Array(layout)
[2.2, 0, 0, 1.1, 2.2] ----------------- type: 5 * float64
As such, IndexedArrays can be used to (lazily) remove items, permute items, and/or duplicate items.
IndexedArrays are used as an optimization when performing some operations on ak.contents.RecordArray, to avoid propagating them down to every field. This is more relevant for RecordArrays than other nodes because RecordArrays can contain multiple contents (and UnionArrays, which also have multiple contents, have an index to merge an array-slice into).
For example, slicing the following recordarray by [3, 2, 4, 4, 1, 0, 3] does not affect any of the RecordArray’s fields.
recordarray = ak.contents.RecordArray(
[
ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
],
None,
)
recordarray
<RecordArray is_tuple='true' len='5'>
<content index='0'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray>
recordarray[[3, 2, 4, 4, 1, 0, 3]]
<IndexedArray len='7'>
<index><Index dtype='int64' len='7'>
[3 2 4 4 1 0 3]
</Index></index>
<content><RecordArray is_tuple='true' len='5'>
<content index='0'>
<NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray>
</content>
<content index='1'>
<ListOffsetArray len='5'>
<offsets><Index dtype='int64' len='6'>
[0 1 3 6 8 9]
</Index></offsets>
<content><NumpyArray dtype='int64' len='9'>[1 1 2 1 2 3 3 2 3]</NumpyArray></content>
</ListOffsetArray>
</content>
</RecordArray></content>
</IndexedArray>
Categorical data#
IndexedArrays, with the "__array__": "categorical" parameter, can represent categorical data (sometimes called dictionary-encoding).
layout = ak.contents.IndexedArray(
ak.index.Index64(np.array([2, 2, 1, 4, 0, 5, 3, 3, 0, 1])),
ak.from_iter(["zero", "one", "two", "three", "four", "five"], highlevel=False),
parameters={"__array__": "categorical"},
)
ak.to_list(layout)
['two', 'two', 'one', 'four', 'zero', 'five', 'three', 'three', 'zero', 'one']
The above has only one copy of strings from "zero" to "five", but they are effectively replicated 10 times in the array.
Any IndexedArray can perform this lazy replication, but labeling it as "__array__": "categorical" is a promise that the content contains only unique values.
Functions like ak.to_arrow() and ak.to_parquet() will project (eagerly evaluate) an IndexedArray that is not labeled as "__array__": "categorical" and dictionary-encode an IndexedArray that is. This distinguishes between the use of IndexedArrays as invisible optimizations and intentional, user-visible ones.
Content >: IndexedOptionArray#
ak.contents.IndexedOptionArray is the most general of the four nodes that allow for missing data (ak.contents.IndexedOptionArray, ak.contents.ByteMaskedArray, ak.contents.BitMaskedArray, and ak.contents.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).
ak.contents.IndexedOptionArray is an ak.contents.IndexedArray in which negative values in the index are interpreted as missing. Since it has an index, the content does not need to have “dummy values” at each index of a missing value. It is therefore more compact for record-type data with many missing values, but ak.contents.ByteMaskedArray and especially ak.contents.BitMaskedArray are more compact for numerical data or data without many missing values.
layout = ak.contents.IndexedOptionArray(
ak.index.Index64(np.array([2, -1, 0, -1, -1, 1, 2])),
ak.contents.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3])),
)
layout
<IndexedOptionArray len='7'>
<index><Index dtype='int64' len='7'>[ 2 -1 0 -1 -1 1 2]</Index></index>
<content><NumpyArray dtype='float64' len='4'>[0. 1.1 2.2 3.3]</NumpyArray></content>
</IndexedOptionArray>
ak.Array(layout)
[2.2, None, 0, None, None, 1.1, 2.2] ------------------ type: 7 * ?float64
Because of its flexibility, most operations that output potentially missing values use an IndexedOptionArray. ak.to_packed() and conversions to/from Arrow or Parquet convert it back into a more compact type.
Content >: ByteMaskedArray#
ak.contents.ByteMaskedArray is the simplest of the four nodes that allow for missing data (ak.contents.IndexedOptionArray, ak.contents.ByteMaskedArray, ak.contents.BitMaskedArray, and ak.contents.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).
ak.contents.ByteMaskedArray is most similar to NumPy’s masked arrays, except that Awkward ByteMaskedArrays can contain any data type and variable-length structures. The valid_when parameter lets you choose whether True means an element is valid (not masked/not None) or False means an element is valid.
layout = ak.contents.ByteMaskedArray(
ak.index.Index8(np.array([False, False, True, True, False, True, False], np.int8)),
ak.contents.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6])),
valid_when=False,
)
layout
<ByteMaskedArray valid_when='false' len='7'>
<mask><Index dtype='int8' len='7'>[0 0 1 1 0 1 0]</Index></mask>
<content><NumpyArray dtype='float64' len='7'>[0. 1.1 2.2 3.3 4.4 5.5 6.6]</NumpyArray></content>
</ByteMaskedArray>
ak.Array(layout)
[0, 1.1, None, None, 4.4, None, 6.6] ------------------ type: 7 * ?float64
Content >: BitMaskedArray#
ak.contents.BitMaskedArray is the most compact of the four nodes that allow for missing data (ak.contents.IndexedOptionArray, ak.contents.ByteMaskedArray, ak.contents.BitMaskedArray, and ak.contents.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).
ak.contents.BitMaskedArray is just like ak.contents.ByteMaskedArray except that the booleans are bit-packed. It is motivated primarily by Apache Arrow, which uses bit-packed masks; supporting bit-packed masks in Awkward Array allows us to represent Arrow data directly. Most operations immediately convert BitMaskedArrays into ByteMaskedArrays.
Since bits always come in groups of at least 8, an explicit length must be supplied to the constructor. Also, lsb_order=True or False determines whether the bytes are interpreted least-significant bit first or most-significant bit first, respectively.
layout = ak.contents.BitMaskedArray(
ak.index.IndexU8(
np.packbits(np.array([False, False, True, True, False, True, False], np.uint8))
),
ak.contents.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6])),
valid_when=False,
length=7,
lsb_order=True,
)
layout
<BitMaskedArray valid_when='false' lsb_order='true' len='7'>
<mask><Index dtype='uint8' len='1'>[52]</Index></mask>
<content><NumpyArray dtype='float64' len='7'>[0. 1.1 2.2 3.3 4.4 5.5 6.6]</NumpyArray></content>
</BitMaskedArray>
ak.Array(layout)
[0, 1.1, None, 3.3, None, None, 6.6] ------------------ type: 7 * ?float64
Changing only the lsb_order changes the interpretation in important ways!
ak.Array(
ak.contents.BitMaskedArray(
layout.mask,
layout.content,
layout.valid_when,
len(layout),
lsb_order=False,
)
)
[0, 1.1, None, None, 4.4, None, 6.6] ------------------ type: 7 * ?float64
Content >: UnmaskedArray#
ak.contents.UnmaskedArray describes formally missing data, but in a case in which no data are actually missing. It is a corner case of the four nodes that allow for missing data (ak.contents.IndexedOptionArray, ak.contents.ByteMaskedArray, ak.contents.BitMaskedArray, and ak.contents.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).
layout = ak.contents.UnmaskedArray(
ak.contents.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
)
layout
<UnmaskedArray len='5'>
<content><NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray></content>
</UnmaskedArray>
ak.Array(layout)
[1.1, 2.2, 3.3, 4.4, 5.5] ------------------ type: 5 * ?float64
The only distinguishing feature of an UnmaskedArray is the question mark ? or option in its type.
ak.type(layout)
ArrayType(OptionType(NumpyType('float64')), 5, None)
Content >: UnionArray#
ak.contents.UnionArray and ak.contents.RecordArray are the only two node types that have multiple contents, not just a single content (and the property is pluralized to reflect this fact). RecordArrays represent a “product type,” data containing records with fields x, y, and z have x’s type AND y’s type AND z’s type, whereas UnionArrays represent a “sum type,” data that are x’s type OR y’s type OR z’s type.
In addition, ak.contents.UnionArray has two ak.index.Index-typed attributes, tags and index; it is the most complex node type. The tags specify which content array to draw each array element from, and the index specifies which element from that content.
The UnionArray element at index i is therefore:
contents[tags[i]][index[i]]
Although the ability to make arrays with mixed data type is very expressive, not all operations support union type (including iteration in Numba). If you intend to make union-type data for an application, be sure to verify that it will work by generating some test data using ak.from_iter().
Awkward Array’s UnionArray is equivalent to Apache Arrow’s dense union. Awkward Array has no counterpart for Apache Arrow’s sparse union (which has no index). ak.from_arrow() generates an index on demand when reading sparse union from Arrow.
layout = ak.contents.UnionArray(
ak.index.Index8(np.array([0, 1, 2, 0, 0, 1, 1, 2, 2, 0], np.int8)),
ak.index.Index64(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])),
[
ak.contents.NumpyArray(
np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9])
),
ak.from_iter(
[
[],
[1],
[1, 2],
[1, 2, 3],
[1, 2, 3, 4],
[1, 2, 3, 4, 5],
[6],
[6, 7],
[6, 7, 8],
[6, 7, 8, 9],
],
highlevel=False,
),
ak.from_iter(
[
"zero",
"one",
"two",
"three",
"four",
"five",
"six",
"seven",
"eight",
"nine",
],
highlevel=False,
),
],
)
layout
<UnionArray len='10'>
<tags><Index dtype='int8' len='10'>[0 1 2 0 0 1 1 2 2 0]</Index></tags>
<index><Index dtype='int64' len='10'>[0 1 2 3 4 5 6 7 8 9]</Index></index>
<content index='0'>
<NumpyArray dtype='float64' len='10'>
[0. 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9]
</NumpyArray>
</content>
<content index='1'>
<ListOffsetArray len='10'>
<offsets><Index dtype='int64' len='11'>
[ 0 0 1 3 6 10 15 16 18 21 25]
</Index></offsets>
<content><NumpyArray dtype='int64' len='25'>
[1 1 2 1 2 3 1 2 3 4 1 2 3 4 5 6 6 7 6 7 8 6 7 8 9]
</NumpyArray></content>
</ListOffsetArray>
</content>
<content index='2'>
<ListOffsetArray len='10'>
<parameter name='__array__'>'string'</parameter>
<offsets><Index dtype='int64' len='11'>
[ 0 4 7 10 15 19 23 26 31 36 40]
</Index></offsets>
<content><NumpyArray dtype='uint8' len='40'>
<parameter name='__array__'>'char'</parameter>
[122 101 114 111 111 110 101 116 119 111 116 104 114 101 101
102 111 117 114 102 105 118 101 115 105 120 115 101 118 101
110 101 105 103 104 116 110 105 110 101]
</NumpyArray></content>
</ListOffsetArray>
</content>
</UnionArray>
ak.to_list(layout)
[0.0, [1], 'two', 3.3, 4.4, [1, 2, 3, 4, 5], [6], 'seven', 'eight', 9.9]
The index can be used to prevent the need to set up “dummy values” for all contents other than the one specified by a given tag. The above example could thus be more compact with the following (no unreachable data):
layout = ak.contents.UnionArray(
ak.index.Index8(np.array([0, 1, 2, 0, 0, 1, 1, 2, 2, 0], np.int8)),
ak.index.Index64(np.array([0, 0, 0, 1, 2, 1, 2, 1, 2, 3])),
[
ak.contents.NumpyArray(np.array([0.0, 3.3, 4.4, 9.9])),
ak.from_iter([[1], [1, 2, 3, 4, 5], [6]], highlevel=False),
ak.from_iter(["two", "seven", "eight"], highlevel=False),
],
)
layout
<UnionArray len='10'>
<tags><Index dtype='int8' len='10'>[0 1 2 0 0 1 1 2 2 0]</Index></tags>
<index><Index dtype='int64' len='10'>[0 0 0 1 2 1 2 1 2 3]</Index></index>
<content index='0'>
<NumpyArray dtype='float64' len='4'>[0. 3.3 4.4 9.9]</NumpyArray>
</content>
<content index='1'>
<ListOffsetArray len='3'>
<offsets><Index dtype='int64' len='4'>
[0 1 6 7]
</Index></offsets>
<content><NumpyArray dtype='int64' len='7'>[1 1 2 3 4 5 6]</NumpyArray></content>
</ListOffsetArray>
</content>
<content index='2'>
<ListOffsetArray len='3'>
<parameter name='__array__'>'string'</parameter>
<offsets><Index dtype='int64' len='4'>
[ 0 3 8 13]
</Index></offsets>
<content><NumpyArray dtype='uint8' len='13'>
<parameter name='__array__'>'char'</parameter>
[116 119 111 115 101 118 101 110 101 105 103 104 116]
</NumpyArray></content>
</ListOffsetArray>
</content>
</UnionArray>
ak.to_list(layout)
[0.0, [1], 'two', 3.3, 4.4, [1, 2, 3, 4, 5], [6], 'seven', 'eight', 9.9]
ak.from_iter() is by far the easiest way to create UnionArrays for small tests.
Relationship to ak.from_buffers#
The generic buffers tutorial describes a function, ak.from_buffers() that builds an array from one-dimensional buffers and an ak.forms.Form. Forms describe the complete tree structure of an array without the array data or lengths, and the array data are in the buffers. The ak.from_buffers() function was designed to operate on data produced by ak.to_buffers(), but you can also prepare its form, length, and buffers manually.
The ak.from_buffers() builds arrays using the above constructors, but the interface allows these structures to be built as data, rather than function calls. (Forms have a JSON representation.) If you are always building the same type of array, directly calling the constructors is likely easier. If you’re generating different data types programmatically, preparing data for ak.from_buffers() may be easier than generating and evaluating Python code that call these constructors.
Every ak.contents.Content subclass has a corresponding ak.forms.Form, and you can see a layout’s Form through its form property.
array = ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
array.layout
<ListOffsetArray len='3'>
<offsets><Index dtype='int64' len='4'>
[0 3 3 5]
</Index></offsets>
<content><NumpyArray dtype='float64' len='5'>[1.1 2.2 3.3 4.4 5.5]</NumpyArray></content>
</ListOffsetArray>
# Abbreviated JSON representation
array.layout.form
ListOffsetForm('i64', NumpyForm('float64'))
# Full JSON representation
print(array.layout.form.to_json())
{"class": "ListOffsetArray", "offsets": "i64", "content": {"class": "NumpyArray", "primitive": "float64", "inner_shape": [], "parameters": {}, "form_key": null}, "parameters": {}, "form_key": null}
In this way, you can figure out how to generate Forms corresponding to the Content nodes you want ak.from_buffers() to make.