# Ingesting from common file formats

For ingesting data from common file formats, it's recommend to use the [Tilebox Python SDK](/docs/sdks/python/install), since
it provides out-of-the-box support for reading many common formats through third party libraries for loading data as either
`pandas.DataFrame` or `xarray.Dataset`, which can then be directly ingested into Tilebox.

## Reading and previewing the data

To ingest data from a file, you first need to read it into a `pandas.DataFrame` or an `xarray.Dataset`.
How that can be achieved depends on the file format. The following sections show examples for a couple of common
file formats.

### CSV

Comma-separated values (CSV) is a common file format for tabular data. It's widely used in data science. Tilebox
supports CSV ingestion using the `pandas.read_csv` function.

Assuming you have a CSV file named `data.csv` with the following content. If you want to follow along, you can
download the file [here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.csv).

```csv title="ingestion_data.csv"
time,value,sensor,precise_time,sensor_history,some_unwanted_column
2025-03-28T11:44:23Z,45.16,A,2025-03-28T11:44:23.345761444Z,"[-12.15, 13.45, -8.2, 16.5, 45.16]","Unsupported"
2025-03-28T11:45:19Z,273.15,B,2025-03-28T11:45:19.128742312Z,"[300.16, 280.12, 273.15]","Unsupported"
```

**Python**

```python title="Python"
import pandas as pd

data = pd.read_csv("ingestion_data.csv")
```

### Parquet

[Apache Parquet](https://parquet.apache.org/) is an open source, column-oriented data file format designed for efficient data storage and retrieval.
Tilebox supports Parquet ingestion using the `pandas.read_parquet` function.

The parquet file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.parquet).

**Python**

```python title="Python"
import pandas as pd

data = pd.read_parquet("ingestion_data.parquet")
```

### GeoParquet

[GeoParquet](https://geoparquet.org/) is an extension of the Parquet file format, adding geospatial
features support to Parquet. Tilebox supports GeoParquet ingestion using the `geopandas.read_parquet` function.

The GeoParquet file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/modis_MCD12Q1.geoparquet).

**Python**

```python title="Python"
import geopandas as gpd

data = gpd.read_parquet("modis_MCD12Q1.geoparquet")
```

For a step-by-step guide to ingesting a GeoParquet file, see [Ingest into a spatio-temporal catalog](/docs/guides/datasets/ingest-into-spatiotemporal-catalog).

### Feather

[Feather](https://arrow.apache.org/docs/python/feather.html) is a file format originating from the Apache Arrow project,
designed for storing tabular data in a fast and memory-efficient way. Tilebox supports Feather ingestion using the `pandas.read_feather` function.

The feather file used in this example [is available here](https://storage.googleapis.com/tbx-web-assets-2bad228/docs/data-samples/ingestion_data.feather).

**Python**

```python title="Python"
import pandas as pd

data = pd.read_feather("ingestion_data.feather")
```

## Mapping columns to dataset fields

Once data is read into a `pandas.DataFrame` or an `xarray.Dataset`, it can be ingested into Tilebox directly.
The column names of the `pandas.DataFrame` or the variables and coordinates of the `xarray.Dataset` are mapped to the fields of the
Tilebox dataset to ingest into.

Depending on how closely the column names or variable/coordinate names match the field names in the Tilebox dataset,
you might need to rename some columns/variables/coordinates before ingestion.

### Renaming fields

**Pandas**

```python title="Pandas"
data = data.rename({"precise_time": "measurement_time"})
```

**Xarray**

```python title="Xarray"
data = data.rename({"precise_time": "measurement_time"})
```

## Dropping fields

In case you want to skip certain columns/variables/coordinates entirely, you can drop them before ingestion.

**Pandas**

```python title="Pandas"
data = data.drop(columns=["some_unwanted_column"])
```

**Xarray**

```python title="Xarray"
data = data.drop_vars(["some_unwanted_variable"])
```

## Ingesting the data

Once the data is in the correct format, you can ingest it into Tilebox.

**Python**

```python title="Python"
from tilebox.datasets import Client

client = Client()
# my_custom_dataset has a schema that matches the data we want to ingest
dataset = client.dataset("my_org.my_custom_dataset")

collection = dataset.get_or_create_collection("Measurements")
collection.ingest(data)
```