Skip to content

Data Catalog ​

The Data Catalog holds the datasets your dataflows read: the ones that come with Curio, files you import from your computer, and results your own nodes save. Add a dataset to a dataflow, drag it onto the canvas, and Curio writes the code that loads it.

Find a dataset ​

On the canvas, click Data Catalog in the top bar, or open the Data Catalog dropdown in the Tools panel on the left and click Browse Data Catalog +. The drawer has three tabs: Browse all, In project for the datasets this dataflow uses, and Computed for outputs your nodes saved. In the catalogs, a project is one of your saved dataflows.

Click Add to project on a dataset and confirm, and it appears in the Tools panel's Data Catalog dropdown. Remove from project takes it out again.

Data Catalog, in the top bar of the Projects page, shows the same datasets for your whole account. Click one to open its panel, where Add to all projects adds it to every project you have and to new ones. When the server allows it, the panel also offers Publish, to share a dataset of yours with everyone on that server.

Load it on the canvas ​

Drag a dataset from the Tools panel onto empty canvas, and Curio creates a Data Loading node whose Data box holds it. Its code, on the Code tab, loads what the box holds the right way for its format. Drop a dataset onto a Data Loading node, and its Data box takes it; onto another node, and the loading code is added to that node's code instead. The code names the dataset rather than a file path, so it keeps working when you share or move the dataflow. Functions node code calls lists the calls that read a dataset, with their arguments.

Clicking a dataset in the dropdown highlights the nodes that use it. A node that reads a dataset shows a DATASET pill in its title bar; click the pill to find the dataset in the Tools panel.

Bring your own data ​

Drop a file from your computer on the canvas or on a Data Loading node, or click Choose file in a Data Loading node's Data box: Curio imports it into your Data Catalog, adds it to the dataflow and loads it. Import dataset, at the bottom of the drawer or at the top of the Data Catalog page, imports a file without loading it. Curio accepts:

  • CSV (.csv), JSON (.json) and Parquet (.parquet) files
  • GeoJSON (.geojson) files
  • GeoTIFF rasters (.tif, .tiff)
  • Shapefiles, as a .zip holding the .shp with its .dbf and .shx (and its .prj), which become one dataset
  • OpenStreetMap extracts (.pbf) and GeoPackages (.gpkg), which become one dataset per layer
  • A .zip that Curio exported, with its bundle.json, which becomes one multi-part dataset again

Import dataset does not add the file to the open dataflow, so click Add to project on it next. To get a dataset from an open data portal, storage or a service instead, see Discovery Catalog.

Delete removes one of your uploads or computed datasets from your account and from every dataflow that uses it. Removing an upload from the last dataflow that uses it also deletes it, and the confirmation says so first.

Save node outputs as datasets ​

Among the buttons in a node's header, which show while the pointer is over the node, is a small database toggle, off by default. Turn it on and run the node: its output is saved to your Data Catalog as a computed dataset and appears in the drawer's Computed tab, ready to add to any dataflow. Running it again replaces the saved copy, and the dataflow must be saved for its outputs to be stored.

Tables are saved as Parquet (a GeoDataFrame keeps its geometry and coordinate system), rasters as GeoTIFF, plain Python values as JSON, and a tuple of results as a multi-part dataset. A node whose output was saved shows an OUTPUT pill.

Follow a dataset's lineage ​

View details on a dataset opens its Overview, Schema, Table Preview and Lineage tabs. For a computed dataset, Lineage shows the node it was Generated by, that node's inputs, and the nodes it is Consumed by, then lists the projects that use it. The record stays with the dataset even after you remove it from every dataflow. Export downloads most datasets as a file, and a multi-part dataset or a group of layers as one .zip: a file per part, named after the part, and a bundle.json that lists the parts.