---
jupytext:
text_representation:
extension: .md
format_name: myst
format_version: 0.13
jupytext_version: 1.14.0
kernelspec:
display_name: Python 3 (ipykernel)
language: python
name: python3
---
Generic buffers
===============
Most of the conversion functions target a particular library: NumPy, Arrow, Pandas, or Python itself. As a catch-all for other storage formats, Awkward Arrays can be converted to and from sets of named buffers. The buffers are not (usually) intelligible on their own; the length of the array and a JSON document are needed to reconstitute the original structure. This section will demonstrate how an array-set can be used to store an Awkward Array in an HDF5 file, which ordinarily wouldn't be able to represent nested, irregular data structures.
```{code-cell} ipython3
import awkward as ak
import numpy as np
import h5py
import json
```
From Awkward to buffers
-----------------------
Consider the following complex array:
```{code-cell} ipython3
ak_array = ak.Array(
[
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
[],
[{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}],
]
)
ak_array
```
The {func}`ak.to_buffers` function decomposes it into a set of one-dimensional arrays (a zero-copy operation).
```{code-cell} ipython3
form, length, container = ak.to_buffers(ak_array)
```
The pieces needed to reconstitute this array are:
* the {class}`ak.forms.Form`, which defines how structure is built from one-dimensional arrays,
* the length of the original array,
* the one-dimensional arrays in the `container` (a {class}`collections.abc.MutableMapping`).
The {class}`ak.forms.Form` is like an Awkward {class}`ak.types.Type` in that it describes how the data are structured, but with more detail: it includes distinctions such as the difference between {class}`ak.contents.ListArray` and {class}`ak.contents.ListOffsetArray`, as well as the integer types of structural {class}`ak.index.Index`.
It is usually presented as JSON, and has a compact JSON format (when {meth}`ak.forms.Form.tojson` is invoked).
```{code-cell} ipython3
form
```
In this case, the `length` is just an integer. It would be a list of integers if `ak_array` was partitioned.
```{code-cell} ipython3
length
```
This `container` is a new dict, but it could have been a user-specified {class}`collections.abc.MutableMapping` if passed into {func}`ak.to_buffers` as an argument.
```{code-cell} ipython3
container
```
From buffers to Awkward
-----------------------
The function that reverses {func}`ak.to_buffers` is {func}`ak.from_buffers`. Its first three arguments are `form`, `length`, and `container`.
```{code-cell} ipython3
ak.from_buffers(form, length, container)
```
Minimizing the size of the output buffers
-----------------------------------------
The {func}`ak.to_buffers`/{func}`ak.from_buffers` functions exactly preserve an array, warts and all. Often, you'll want to only write {func}`ak.to_packed` arrays. "Packing" replaces an array structure with an equivalent structure that has no unreachable elements—data that you can't see as part of the array, and therefore probably don't want to write.
Here is an example of an array in need of packing:
```{code-cell} ipython3
unpacked = ak.Array(
ak.contents.ListArray(
ak.index.Index64(np.array([4, 10, 1])),
ak.index.Index64(np.array([7, 10, 3])),
ak.contents.NumpyArray(np.array([999, 4.4, 5.5, 999, 1.1, 2.2, 3.3, 999])),
)
)
unpacked
```
This {class}`ak.contents.ListArray` is in a strange order and the `999` values are unreachable. (Also, using `starts[1] == stops[1] == 10` to represent an empty list is a little odd, though allowed by the specification.)
The {func}`ak.to_buffers` function dutifully writes the `999` values into the output, even though they're not visible in the array.
```{code-cell} ipython3
ak.to_buffers(unpacked)
```
If the intended purpose of calling {func}`ak.to_buffers` is to write to a file or send data over a network, this is wasted space. It can be trimmed by calling the {func}`ak.to_packed` function.
```{code-cell} ipython3
packed = ak.to_packed(unpacked)
packed
```
At high-level, the array appears to be the same, but its low-level structure is quite different:
```{code-cell} ipython3
unpacked.layout
```
```{code-cell} ipython3
packed.layout
```
This version of the array is more concise when written with {func}`ak.to_buffers`:
```{code-cell} ipython3
ak.to_buffers(packed)
```
Saving Awkward Arrays to HDF5
-----------------------------
The [h5py](https://www.h5py.org/) library presents each group in an HDF5 file as a {class}`collections.abc.MutableMapping`, which we can use as a container for an array-set. We must also save the `form` and `length` as metadata for the array to be retrievable.
```{code-cell} ipython3
file = h5py.File("/tmp/example.hdf5", "w")
group = file.create_group("awkward")
group
```
We can fill this `group` as a `container` by passing it in to {func}`ak.to_buffers`. (See the previous section for more on {func}`ak.to_packed`.)
```{code-cell} ipython3
form, length, container = ak.to_buffers(ak.to_packed(ak_array), container=group)
```
```{code-cell} ipython3
container
```
Now the HDF5 group has been filled with array pieces.
```{code-cell} ipython3
container.keys()
```
Here's one.
```{code-cell} ipython3
np.asarray(container["node0-offsets"])
```
Now we need to add the other information to the group as metadata. Since HDF5 accepts string-valued metadata, we can put it all in as JSON or numbers.
```{code-cell} ipython3
group.attrs["form"] = form.to_json()
group.attrs["form"]
```
```{code-cell} ipython3
group.attrs["length"] = length
group.attrs["length"]
```
Reading Awkward Arrays from HDF5
--------------------------------
With that, we can reconstitute the array by supplying {func}`ak.from_buffers` the right arguments from the group and metadata.
The group can't be used as a `container` as-is, since subscripting it returns {class}`h5py.Dataset` objects, rather than arrays.
```{code-cell} ipython3
reconstituted = ak.from_buffers(
ak.forms.from_json(group.attrs["form"]),
group.attrs["length"],
{k: np.asarray(v) for k, v in group.items()},
)
reconstituted
```