Parquet files download

Apache Arrow lets you work efficiently with large, multi-file datasets. The R package provides a interface to Arrow Datasets, as well as other tools for interactive exploration of Arrow data.

This vignette introduces Datasets and shows how to use to analyze them. It describes both what is possible to do with Arrow now and what is on the immediate development roadmap.

Example: NYC taxi data

The New York City taxi trip record data is widely used in big data exercises and competitions. For demonstration purposes, we have hosted a Parquet-formatted version of about 10 years of the trip data in a public AWS S3 bucket.

The total file size is around 37 gigabytes, even in the efficient Parquet file format. That's bigger than memory on most people's computers, so we can't just read it all in and stack it into a single data frame.

In Windows and macOS binary packages, S3 support is included. On Linux when installing from source, S3 support is not enabled by default, and it has additional system requirements. See for details. To see if your installation has S3 support, run

Even with S3 support enabled network, speed will be a bottleneck unless your machine is located in the same AWS region as the data. So, for this vignette, we assume that the NYC taxi dataset has been downloaded locally in a “nyc-taxi” directory.

If your build has S3 support, you can sync the data locally with:

If your build doesn't have S3 support, you can download the files with some additional code:

Note that these download steps in the vignette are not executed: if you want to run with live data, you'll have to do it yourself separately. Given the size, if you're running this locally and don't have a fast connection, feel free to grab only a year or two of data.

If you don't have the taxi data downloaded, the vignette will still run and will yield previously cached output for reference. To be explicit about which version is running, let's check whether we're running with live data:

Getting started

Because is not necessary for many Arrow workflows, it is an optional () dependency. So, to work with Datasets, we need to load both and .

The first step is to create our Dataset object, pointing at the directory of data.

The default file format for is Parquet; if we had a directory of Arrow format files, we could include in the call. Other supported formats include: “feather” (an alias for “arrow”, as Feather v2 is the Arrow file format), “csv”, “tsv” (for tab-delimited), and “text” for generic text-delimited files. For text files, you can pass any parsing options (“delim”, “quote”, etc.) to that you would otherwise pass to .

The argument lets us specify how the file paths provide information about how the dataset is chunked into different files. Our files in this example have file paths like

By providing a character vector to , we're saying that the first path segment gives the value for “year” and the second segment is “month”. Every row in has a value of for “year” and 1 for “month”, even though those columns may not actually be present in the file.

Indeed, when we look at the dataset, we see that in addition to the columns present in every file, there are also columns “year” and “month”.

The other form of partitioning currently supported is Hive-style, in which the partition variable names are included in the path segments. If we had saved our files in paths like

we would not have had to provide the names in : we could have just called and the partitions would have been detected automatically.

Querying the dataset

Up to this point, we haven't loaded any data: we have walked directories to find files, we've parsed file paths to identify partitions, and we've read the headers of the Parquet files to inspect their schemas so that we can make sure they all line up.

In the current release, supports methods for selecting a window of data: , , and . Aggregation is not yet supported, nor is deriving or projecting new columns, so before you call or , you'll need to the data first, which pulls your selected window of data into an in-memory R data frame. While we could have made those methods the data they needed automatically and invisibly to the end user, we thought it best to make it explicit when you're pulling data into memory so that you can construct your queries most efficiently and not be surprised when some query consumes way more resources than expected.

Here's an example. Suppose I was curious about tipping behavior among the longest taxi rides. Let's find the median tip percentage for rides with fares greater than $ in , broken down by the number of passengers:

We just selected a window out of a dataset with around 2 billion rows and aggregated on it in under 2 seconds on my laptop. How does this work?

First, /, , and record their actions but don't evaluate on the data until you run .

This returns instantly and shows the window selection you've made, without loading data from the files. Because the evaluation of these queries is deferred, you can build up a query that selects down to a small window without generating intermediate datasets that would potentially be large.

Second, all work is pushed down to the individual data files, and depending on the file format, chunks of data within the files. As a result, we can select a window of data from a much larger dataset by collecting the smaller slices from each file–we don't have to load the whole dataset in memory in order to slice from it.

Third, because of partitioning, we can ignore some files entirely. In this example, by filtering , all files corresponding to other years are immediately excluded: we don't have to load them in order to find that no rows match the filter. Relatedly, since Parquet files contain row groups with statistics on the data within, there may be entire chunks of data we can avoid scanning because they have no rows where .

More dataset options

There are a few ways you can control the Dataset creation to adapt to special use cases. For one, you can specify a argument to declare the columns and their data types. This is useful if you have data files that have different storage schema (for example, a column could be in one and in another) and you want to ensure that the resulting Dataset has a specific type. To be clear, it's not necessary to specify a schema, even in this example of mixed integer types, because the Dataset constructor will reconcile differences like these. The schema specification just lets you declare what you want the result to be.

Similarly, you can provide a Schema in the argument of in order to declare the types of the virtual columns that define the partitions. This would be useful, in our taxi dataset example, if you wanted to keep “month” as a string instead of an integer for some reason.

Another feature of Datasets is that they can be composed of multiple data sources. That is, you may have a directory of partitioned Parquet files in one location, and in another directory, files that haven't been partitioned. Or, you could point to an S3 bucket of Parquet data and a directory of CSVs on the local file system and query them together as a single dataset. To create a multi-source dataset, provide a list of datasets to instead of a file path, or simply concatenate them like .

Writing datasets

As you can see, querying a large dataset can be made quite fast by storage in an efficient binary columnar format like Parquet or Feather and partitioning based on columns commonly used for filtering. However, we don't always get our data delivered to us that way. Sometimes we start with one giant CSV. Our first step in analyzing data is cleaning is up and reshaping it into a more usable form.

The function allows you to take a Dataset or other tabular data object—an Arrow or , or an R —and write it to a different file format, partitioned into multiple files.

Assume we have a version of the NYC Taxi data as CSV:

We can write it to a new location and translate the files to the Feather format by calling on it:

Next, let's imagine that the “payment_type” column is something we often filter on, so we want to partition the data by that variable. By doing so we ensure that a filter like will touch only a subset of files where payment_type is always 3.

One natural way to express the columns you want to partition on is to use the method:

This will write files to a directory tree that looks like this:

Note that the directory names are and similar: this is the Hive-style partitioning described above. This means that when we call on this directory, we don't have to declare what the partitions are because they can be read from the file paths. (To instead write bare values for partition segments, i.e. rather than , call with .)

Perhaps, though, is the only data we ever care about, and we just want to drop the rest and have a smaller working set. For this, we can them out when writing:

The other thing we can do when writing datasets is select a subset of and/or reorder columns. Suppose we never care about , and being a string column, it can take up a lot of space when we read it in, so let's drop it:

Note that while you can select a subset of columns, you cannot currently rename columns when writing.