Datasets

Related Guides

Dataset types

Each dataset is of a specific type. Each dataset type comes with a set of required fields for each data point. The dataset type also determines the query capabilities for a dataset, for example, whether a dataset supports time-based queries or additionally also spatially filtered queries.

To find out which fields are required for each dataset type check out the documentation for the available dataset types below.

Dataset specific fields

Additionally, each dataset has a set of fields that are specific to that dataset. Fields are defined during dataset creation. That way, all data points in a dataset are strongly typed and are validated during ingestion. The required fields of the dataset type, as well as the custom fields specific to each dataset together make up the dataset schema.

Once a dataset schema is defined, existing fields cannot be removed or edited as soon as data has been ingested into it. You can add new non-queryable fields to a non-empty dataset because all custom fields are optional. You cannot add a new queryable field or change whether an existing field is queryable after ingesting data.

Field types

When defining the data schema, you can specify each field’s type. The following field types are supported.

Primitives

Time

Identifier

Geospatial

STAC metadata

Structured STAC types preserve common metadata without flattening it into separate custom fields. In Python, import these types from tilebox.datasets.schema.

See Assets and storage for the asset access model.

Arrays

Every type is also available as an array, allowing to ingest multiple values of the underlying type for each data point. The size of the array is flexible, and can be different for each data point.

Queryable fields

Custom fields can be marked as queryable when you define the dataset schema. Tilebox projects these fields into query storage so you can filter datapoints by their values together with temporal, spatial, and collection filters.

Listing datasets

You can use your client instance to access the datasets available to you. To list all available datasets, use the datasets method of the client.

from tilebox.datasets import Client

client = Client()
datasets = client.datasets()
print(datasets)

package main

import (
  "context"
  "log"

  "github.com/tilebox/tilebox-go/datasets/v1"
)

func main() {
  ctx := context.Background()

  client := datasets.NewClient()

  allDatasets, err := client.Datasets.List(ctx)
  if err != nil {
    log.Fatalf("Failed to list datasets: %v", err)
  }

  for _, dataset := range allDatasets {
    log.Println(dataset)
  }
}

open_data:
    asf:
        ers_sar: European Remote Sensing Satellite (ERS) Synthetic Aperture Radar (SAR) Granules
    copernicus:
        landsat8_oli_tirs: Landsat-8 is part of the long-running Landsat programme ...
        sentinel1_sar: The Sentinel-1 mission is the European Radar Observatory for the ...
        sentinel2_msi: Sentinel-2 is equipped with an optical instrument payload that samples ...
        sentinel3_olci: OLCI (Ocean and Land Colour Instrument) is an optical instrument used to ...
        ...

Once you have your dataset object, you can use it to list the available collections for the dataset.

Creating / Updating a dataset

Create a dataset by defining its kind, custom fields, and field annotations. The Tilebox Console provides a visual schema editor.

from datetime import timedelta
from tilebox.datasets import Client
from tilebox.datasets.data.datasets import DatasetKind

client = Client()

dataset = client.create_or_update_dataset(
    DatasetKind.SPATIOTEMPORAL,
    "my_landsat8_oli_tirs",
    [
        {
            "name": "granule_name",
            "type": str,
            "description": "The name of the Landsat-8 granule.",
            "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1",
            "queryable": True,
        },
        {
            "name": "cloud_cover",
            "type": float,
            "description": "Cloud cover percentage",
            "example_value": "91",
            "queryable": True,
        },
        {
            "name": "proj_shape",
            "type": list[int],
            "description": "Raster shape",
            "example_value": "[6971, 7801]",
        },
    ],
    name="Personal Landsat-8 catalog",
)

dataset, err := client.Datasets.Create(ctx,
  datasets.KindSpatiotemporal,
  "my_landsat8_oli_tirs",
  "Personal Landsat-8 catalog",
  []datasets.Field{
    field.String("granule_name").Description("The name of the Earth View Granule").ExampleValue("20220830_185202_SN18_10N_552149_5297327").Queryable(),
    field.Float64("cloud_cover").Description("Cloud cover percentage").ExampleValue("91").Queryable(),
    field.Int64("proj_shape").Repeated().Description("Raster shape").ExampleValue("[6971, 7801]"),
  }
)

{
  "kind": "spatiotemporal",
  "fields": [
    {
      "name": "granule_name",
      "type": "string",
      "description": "The name of the Landsat-8 granule.",
      "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1",
      "queryable": true
    },
    {
      "name": "cloud_cover",
      "type": "float64",
      "description": "Cloud cover percentage",
      "example_value": "91",
      "queryable": true
    },
    {
      "name": "proj_shape",
      "type": "int64",
      "repeated": true,
      "description": "Raster shape",
      "example_value": "[6971, 7801]"
    }
  ]
}

tilebox dataset create \
  --name "Personal Landsat-8 catalog" \
  --code-name my_landsat8_oli_tirs \
  --schema-file schema.json

This code will create a new catalog with 7 fields in total. 4 of those fields are auto-generated by choosing the spatio-temporal dataset type, and 3 (granule_name, cloud_cover, proj_shape) are custom fields that are defined. granule_name and cloud_cover can be used in custom field filters; the repeated proj_shape field cannot be queryable.

Accessing a dataset

Each dataset has an automatically generated slug that can be used to access it. The slug is the name of the group, followed by a dot, followed by the dataset code name. For example, the slug for the Sentinel-2 MSI dataset, which is part of the open_data.copernicus group, is open_data.copernicus.sentinel2_msi.

To access a dataset, use the dataset method of your client instance and pass the slug of the dataset as an argument.

from tilebox.datasets import Client

client = Client()
s2_msi_dataset = client.dataset("open_data.copernicus.sentinel2_msi")

s2MsiDataset, err := client.Datasets.Get(ctx, "open_data.copernicus.sentinel2_msi")

Once you have your dataset object, you can use it to access available collections for the dataset.

Deleting a dataset

Datasets can be deleted through the Tilebox Console by clicking the Delete button in the dataset page.

Empty datasets will be deleted right away. A dataset is considered empty if none of its collections contain any data points.

A non-empty dataset can also be deleted, but is a destructive operation. Every data point in every collection of the dataset will be deleted.

As a safety measure, Tilebox soft-deletes the dataset for 7 days before permanently deleting it. Please get in touch if you deleted a dataset by accident and want to restore it.