---
jupytext:
text_representation:
extension: .md
format_name: myst
format_version: 0.13
jupytext_version: 1.14.1
kernelspec:
display_name: Python 3 (ipykernel)
language: python
name: python3
---
How to convert to/from Arrow and Parquet
========================================
The [Apache Arrow](https://arrow.apache.org/) data format is very similar to Awkward Array's, but they're not exactly the same. As such, arrays can _usually_ be shared without copying, but not always.
The [Apache Parquet](https://parquet.apache.org/) file format has strong connections to Arrow with a large overlap in available tools, and while it's also a columnar format like Awkward and Arrow, it is implemented in a different way, which emphasizes compact storage over random access.
```{code-cell} ipython3
import awkward as ak
import pyarrow as pa
import pyarrow.csv
import urllib.request
```
From Arrow to Awkward
---------------------
The function for Arrow → Awkward conversion is {func}`ak.from_arrow`.
The argument to this function can be any of the following types from the pyarrow library:
* {class}`pyarrow.lib.Array`
* {class}`pyarrow.lib.ChunkedArray`
* {class}`pyarrow.lib.RecordBatch`
* {class}`pyarrow.lib.Table`
and they are converted into non-partitioned, non-virtual Awkward Arrays. (Any disjoint chunks in the Arrow array are concatenated.)
```{code-cell} ipython3
pa_array = pa.array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
pa_array
```
```{code-cell} ipython3
ak.from_arrow(pa_array)
```
Here is an example of an Arrow Table, derived from CSV. (Printing a table shows its field types.)
```{code-cell} ipython3
pokemon = urllib.request.urlopen(
"https://gist.githubusercontent.com/armgilles/194bcff35001e7eb53a2a8b441e8b2c6/raw/92200bc0a673d5ce2110aaad4544ed6c4010f687/pokemon.csv"
)
table = pyarrow.csv.read_csv(pokemon)
table
```
Awkward Array doesn't make a deep distinction between "arrays" and "tables" the way Arrow does: the Awkward equivalent of an Arrow table is just an Awkward Array of record type.
```{code-cell} ipython3
array = ak.from_arrow(table)
array
```
The Awkward equivalent of Arrow's schemas is {func}`ak.type`.
```{code-cell} ipython3
ak.type(array)
```
```{code-cell} ipython3
ak.to_list(array[0])
```
This array is ready for data analysis.
```{code-cell} ipython3
array[array.Legendary].Attack - array[array.Legendary].Defense
```
From Awkward to Arrow
---------------------
The function for Awkward → Arrow conversion is {func}`ak.to_arrow`. This function always returns
* {class}`pyarrow.lib.Array`
type.
```{code-cell} ipython3
ak_array = ak.Array(
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}]
)
ak_array
```
```{code-cell} ipython3
pa_array = ak.to_arrow(ak_array)
pa_array
```
```{code-cell} ipython3
type(pa_array)
```
```{code-cell} ipython3
isinstance(pa_array, pa.lib.Array)
```
If you need {class}`pyarrow.lib.RecordBatch`, you can build this using pyarrow:
```{code-cell} ipython3
pa_batch = pa.RecordBatch.from_arrays(
[
ak.to_arrow(ak_array.x),
ak.to_arrow(ak_array.y),
],
["x", "y"],
)
pa_batch
```
If you need {class}`pyarrow.lib.Table`, you can build this using pyarrow:
```{code-cell} ipython3
pa_table = pa.Table.from_batches([pa_batch])
pa_table
```
The columns of this Table are {class}`pa.lib.ChunkedArray` instances:
```{code-cell} ipython3
pa_table[0]
```
```{code-cell} ipython3
pa_table[1]
```
shares memory as much as is possible, which [can be faster than constructing Pandas directly](https://ursalabs.org/blog/fast-pandas-loading/).
+++ {"tags": []}
Reading/writing data streams and random access files
----------------------------------------------------
Arrow has several methods for interfacing to data streams and disk-bound files, [see the official documentation](https://arrow.apache.org/docs/python/ipc.html) for instructions.
When following those instructions, remember that {func}`ak.from_arrow` can accept {class}`pyarrow.lib.Array`, {class}`pyarrow.lib.ChunkedArray`, {class}`pyarrow.lib.RecordBatch`, and {class}`pyarrow.lib.Table`, but {func}`ak.to_arrow` _only returns {class}`pyarrow.lib.Array`_.
For instance, when writing to an IPC stream, Arrow requires {class}`pyarrow.lib.RecordBatch`, so you need to build them:
```{code-cell} ipython3
ak_array = ak.Array(
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}]
)
ak_array
```
```{code-cell} ipython3
first_batch = pa.RecordBatch.from_arrays(
[
ak.to_arrow(ak_array.x),
ak.to_arrow(ak_array.y),
],
["x", "y"],
)
first_batch.schema
```
```{code-cell} ipython3
sink = pa.BufferOutputStream()
writer = pa.ipc.new_stream(sink, first_batch.schema)
writer.write_batch(first_batch)
for i in range(5):
next_batch = pa.RecordBatch.from_arrays(
[
ak.to_arrow(ak_array.x),
ak.to_arrow(ak_array.y),
],
["x", "y"],
)
writer.write_batch(next_batch)
writer.close()
```
```{code-cell} ipython3
bytes(sink.getvalue())
```
But when reading them back, we can just pass the record batches (yielded by the {class}`pyarrow.lib.RecordBatchStreamReader` `reader`) to {func}`ak.from_arrow`:
```{code-cell} ipython3
reader = pa.ipc.open_stream(sink.getvalue())
reader.schema
```
```{code-cell} ipython3
for batch in reader:
print(repr(ak.from_arrow(batch)))
```
Reading/writing the Feather file format
---------------------------------------
Feather is a lightweight file format that puts Arrow Tables in disk-bound files, [see the official documentation](https://arrow.apache.org/docs/python/feather.html) for instructions.
When following those instructions, remember that {func}`ak.from_arrow` can accept {class}`pyarrow.lib.Table`, but {func}`ak.to_arrow` _only returns {class}`pyarrow.lib.Array`_.
For instance, when writing to a Feather file, Arrow requires {class}`pyarrow.lib.Table`, so you need to build them:
```{code-cell} ipython3
ak_array = ak.Array(
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}]
)
ak_array
```
```{code-cell} ipython3
pa_batch = pa.RecordBatch.from_arrays(
[
ak.to_arrow(ak_array.x),
ak.to_arrow(ak_array.y),
],
["x", "y"],
)
pa_table = pa.Table.from_batches([pa_batch])
pa_table
```
```{code-cell} ipython3
import pyarrow.feather
pyarrow.feather.write_feather(pa_table, "/tmp/example.feather")
```
But when reading them back, we can just pass the Arrow Table to {func}`ak.from_arrow`.
```{code-cell} ipython3
from_feather = pyarrow.feather.read_table("/tmp/example.feather")
from_feather
```
```{code-cell} ipython3
type(from_feather)
```
```{code-cell} ipython3
ak.from_arrow(from_feather)
```
Reading/writing the Parquet file format
---------------------------------------
With data converted to and from Arrow, it can then be saved and loaded from Parquet files. [Arrow's official Parquet documentation](https://arrow.apache.org/docs/python/parquet.html) provides instructions for converting Arrow to and from Parquet, but Parquet is a sufficiently important file format that Awkward has specialized functions for it.
The {func}`ak.to_parquet` function writes Awkward Arrays as Parquet files. It has relatively few options.
```{code-cell} ipython3
ak_array = ak.Array(
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}]
)
ak_array
```
```{code-cell} ipython3
ak.to_parquet(ak_array, "/tmp/example.parquet")
```
The {func}`ak.from_parquet` function reads Parquet files as Awkward Arrays, with quite a few more options. Basic usage just gives you the Awkward Array back.
```{code-cell} ipython3
ak.from_parquet("/tmp/example.parquet")
```
Since the data in a Parquet file may be huge, there are `columns` and `row_groups` options to read back only _part_ of the file.
* Parquet's "columns" correspond to Awkward's record "fields," though Parquet columns cannot be nested.
* Parquet's "row groups" are ranges of contiguous elements, such as "`1000-2000`". They correspond to Awkward's "partitioning." Neither Parquet row groups nor Awkward partitions can be nested.
For instance, the expression
```{code-cell} ipython3
ak.from_parquet("/tmp/example.parquet", columns=["x"])
```
Doesn't read column `"y"`.