Protobuf

Tilebox uses Protocol Buffers, with a custom generation tool, combined with standard Go data structures.

Protocol Buffers (often referred to as protobuf) is a schema definition language with an efficient binary format and native language support for lots of languages, including Go. Protocol buffers are open source since 2008 and are maintained by Google.

Generate Go types

Protobuf schemas are typically defined in a .proto file, and then converted to a native Go struct using the protobuf compiler. Tilebox datasets already define a protobuf schema as well. Use tilebox dataset generate to generate Go structs for existing datasets.

See Installation for more details on how to install the Tilebox command-line tool.

tilebox dataset generate --slug open_data.copernicus.sentinel1_sar

Copy

The preceding command will generate a ./protogen/tilebox/v1/sentinel1_sar.pb.go file. Use --out, --package, and --name to change the default output directory, protobuf package, or message name.

This file contains everything needed to work with the Sentinel-1 SAR dataset. It’s recommended to check the generated files you use in your version control system.

If you open this file, you will see that it starts with // Code generated by protoc-gen-go. DO NOT EDIT.. It means that the file was generated by the protoc-gen-go tool, which is part of the protobuf compiler. After editing a dataset, you can call the generate command again to ensure that the changes are reflected in the generated file.

The file contains a Sentinel1Sar struct, which is a Go struct that represents a datapoint in the dataset.

Go

type Sentinel1Sar struct {
  xxx_hidden_GranuleName         *string                `protobuf:"bytes,1,opt,name=granule_name,json=granuleName"`
  xxx_hidden_ProcessingLevel     v1.ProcessingLevel     `protobuf:"varint,2,opt,name=processing_level,json=processingLevel,enum=datasets.v1.ProcessingLevel"`
  // more fields
}

Copy

Notice that the fields are private (starting with a lowercase letter), so they are not accessible. Protobuf hides the fields and provides getters and setters to access them.

Protobuf 101

Initializing a message

Here is how to initialize a v1.Sentinel1Sar message.

Go

import (
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/timestamppb"
)

datapoint := v1.Sentinel1Sar_builder{
	Time:        timestamppb.New(time.Now()),
	GranuleName: proto.String("S1A_EW_GRDH_1SSH_20141004T020507_20141004T020611_002673_002FAF_8645_COG.SAFE"),
	ProductType: proto.String("EW_GRDH_1S-COG"),
	FileSize:    proto.Int64(488383473),
}.Build()

Copy

Protobuf fields are private and provides a builder pattern to create a message.

proto.String is a helper function that converts string to *string. This allows protobuf to differentiate between a field that is set to an empty string and a field that is not set (nil). An exhaustive list of those helper functions can be found here.

Only primitives have a proto.XXX helper function. Complex types such as timestamps, durations, UUIDs, and geometries have a constructor function.

Getters and setters

Protobuf provides methods to get, set, clear and check if a field is set.

Go

fmt.Println(datapoint.GetGranuleName())

datapoint.SetGranuleName("my amazing granule")

datapoint.ClearGranuleName()

if datapoint.HasGranuleName() {
  fmt.Println("Granule name is set")
}

Copy

Getters for primitive types will return a Go native type (for example, int64, string, etc.). Getters for complex types such as timestamps, durations, UUIDs, and geometries can also be converted to more standard types using AsXXX methods.

Well known types

Beside Go primitives, Tilebox supports some well known types:

• Duration: A duration of time. See Duration for more information.
• Timestamp: A point in time. See Timestamp for more information.
• UUID: A universally unique identifier (UUID).
• Geometry: Geospatial geometries of type Point, LineString, Polygon or MultiPolygon.

They have a couple of useful methods to work with them.

Constructors

Go

import (
	"github.com/paulmach/orb"
	datasetsv1 "github.com/tilebox/tilebox-go/protogen/datasets/v1"
	"google.golang.org/protobuf/types/known/durationpb"
	"google.golang.org/protobuf/types/known/timestamppb"
)

timestamppb.New(time.Now())
durationpb.New(10 * time.Second)
datasetsv1.NewUUID(uuid.New())
datasetsv1.NewGeometry(orb.Point{1, 2})

Copy

CheckValid method

CheckValid returns an error if the field is invalid.

Go

err := datapoint.GetTime().CheckValid()
if err != nil {
  fmt.Println(err)
}

Copy

IsValid method

IsValid reports whether the field is valid. It’s equivalent to CheckValid == nil.

Go

if datapoint.GetTime().IsValid() {
  fmt.Println("Valid")
}

Copy

AsXXX methods

AsXXX methods convert the field to a more user friendly type.

• AsUUID will convert a datasetsv1.UUID field to a uuid.UUID type
• AsTime will convert a timestamppb.Timestamp field to a time.Time type
• AsDuration will convert a durationpb.Duration field to a time.Duration type
• AsGeometry will convert a datasetsv1.Geometry field to an orb.Geometry interface

Go

datapoint.GetId().AsUUID() // uuid.UUID
datapoint.GetTime().AsTime() // time.Time
datapoint.GetDuration().AsDuration() // time.Duration
datapoint.GetGeometry().AsGeometry() // orb.Geometry

Copy

Those methods performs conversion on a best-effort basis. Type validity must be checked beforehand using IsValid or CheckValid methods.

Common data operations

Datapoints are contained in a standard Go slice so all the usual slice operations and slice functions can be used.

The usual pattern to iterate over data in Go is by using a for loop.

As an example, here is how to extract the copernicus_id fields from the datapoints.

Go

// assuming datapoints has been filled using `client.Datapoints.QueryInto` method
var datapoints []*v1.Sentinel1Sar

copernicusIDs := make([]uuid.UUID, len(datapoints))
for i, dp := range datapoints {
  copernicusIDs[i] = dp.GetCopernicusId().AsUUID()
}

Copy

Here is an example of filtering out datapoints that have been published before January 2000 and are not from the Sentinel-1C platform.

Go

jan2000 := time.Date(2000, time.January, 1, 0, 0, 0, 0, time.UTC)
// slice of length of 0, but preallocate a capacity of len(datapoints)
s1cDatapoints := make([]*v1.Sentinel1Sar, 0, len(datapoints))

for _, dp := range datapoints {
  if dp.GetPublished().AsTime().Before(jan2000) {
    continue
  }
  if dp.GetPlatform() != "S1C" {
    continue
  }

  s1cDatapoints = append(s1cDatapoints, proto.CloneOf(dp)) // Copy the underlying data
}

Copy

Converting to JSON

Protobuf messages can be converted to JSON without loss of information. This is useful for interoperability with other systems that doesn’t use protobuf. A guide on protoJSON can be found format here: https://protobuf.dev/programming-guides/json/

Go

originalDatapoint := datapoints[0]

// Convert proto.Message to JSON as bytes
jsonDatapoint, err := protojson.Marshal(originalDatapoint)
if err != nil {
  log.Fatalf("Failed to marshal datapoint: %v", err)
}
fmt.Println(string(jsonDatapoint))

Copy

Output

{"time":"2001-01-01T00:00:00Z","id":{"uuid":"AOPHpzQAAmV2MZ4+Zv+JGg=="},"ingestionTime":"2025-03-25T10:26:10.577385176Z","granuleName":"MCD12Q1.A2001001.h02v08.061.2022146033342.hdf","geometry":{"wkb":"AQMAAAABAAAABQAAAFIi9vf7TmTAXsX3////I0Bexff///9jwAAAAAAAAAAACUn4//+/YsAAAAAAAAAAAC7AdjgMCmPAXsX3////I0BSIvb3+05kwF7F9////yNA"},"endTime":"2001-12-31T23:59:59Z","horizontalTileNumber":"2","verticalTileNumber":"8","tileId":"51002008","fileSize":"176215","checksum":"771212892","checksumType":"CKSUM","dayNightFlag":"Day","publishedAt":"2022-06-23T10:58:13.895Z"}

Copy

It can also be converted back to a proto.Message.

Go

// Convert JSON bytes to proto.Message
unmarshalledDatapoint := &v1.Sentinel1Sar{}
err = protojson.Unmarshal(jsonDatapoint, unmarshalledDatapoint)
if err != nil {
  log.Fatalf("Failed to unmarshal datapoint: %v", err)
}

fmt.Println("Are both equal?", proto.Equal(unmarshalledDatapoint, originalDatapoint))

Copy

Output

Are both equal? true

Copy