IDEA: Publish raster/tensor data in additional format(s)

[afeld]

Background

Hello! I'm working with the gridded fire risk data, trying to bring it into DuckDB. (OCR, Icechunk, and Zarr are all new to me.) After a lot of trial and error, this was the simplest thing I could come up with, building off the example:

burn_prob = california_subset["bp_2011"]
ddf = burn_prob.to_dask_dataframe()
ddf.to_parquet("/tmp/burn_prob")

Then, from DuckDB:

SELECT
    *,
    -- making the grids
    ST_MakeEnvelope(
        longitude - (990 / (111320.0 * COS(RADIANS(latitude)))) / 2.0,
        latitude  - (990 / 111320.0) / 2.0,
        longitude + (990 / (111320.0 * COS(RADIANS(latitude)))) / 2.0,
        latitude  + (990 / 111320.0) / 2.0
    ) AS geom
FROM read_parquet('/tmp/burn_prob/*.parquet');

The issue

Moving the data through:

1. Icechunk
2. Xarray
3. Dask
4. Parquet
5. DuckDB

is a lot of indirection, and was challenging despite working as a Data Engineer and teaching in a Data Science program 🙃 It would be even more difficult for someone who isn't comfortable with Python.

Proposal

Publish the raster/tensor data in a common geospatial format such as GeoParquet. This would allow the data to be read directly from a much broader swath of tools (like DuckDB) without conversion.

Side note: Using square Polygons for the geometries would make my life easier, but understandable if you want to keep the file sizes smaller by sticking to Points.

---

Thanks!

[afeld]

I just came across https://www.earthmover.io/blog/tensors-vs-tables, which was helpful for understanding the advantages of Icechunk. Still, I think an alternative would be useful. Thanks!

[norlandrhagen]

Hey there @afeld! Thanks for opening the issue. Wondering what is your use-case with the vector data? We published geoparquet datasets for the buildings across CONUS, as well as regional statistics. Are you wanting to sample points from the raster data?

[afeld]

Hey! I'm working on https://github.com/investinopen/ptps-wildfire-demo (the Analysis part is what's relevant here). We had come up with several queries to drive the work, such as:

• Where's fire risk high right now? (Red Flag Warning + active fire detections + burn history)
• Who's downwind and breathing it? (smoke plumes + PM2.5 forecast)
• Who's most vulnerable to that smoke? (chronic disease rates + social vulnerability)
• Whose power might get cut because of it? (infrastructure/outage risk)

I was thinking about this in terms of area rather than points, hence the choice to use the gridded raster/tensor data rather than the building-level risk. Does that make sense?

[afeld]

(I understand the regional statistics would also work for this purpose; was looking for more granularity.)