Skip to main content

Sending Data Between Metaframes

Overview

One of the core functions of Metapages is the transfer of data between metaframes. This connectivity is what enables you to build powerful workflows and data processing pipelines. Data flow can be configured through:

  • The metapage's Data Flow tab
  • The metaframe's connect button

Creating Data Connections

A data connection is specified using up to two parameters:

  1. Source name or glob: Defines which outputs to send from the source metaframe to the target metaframe. If you want to send a single file, this would be a string including the filename or file path if the file is located inside a folder. For sending multiple files, you can specify a glob with wildcards to match the files you want to send. If this field is left blank, all files routed to the outputs will be sent.
  2. Target file or folder: Allows you to change the name of the file (in the case only one file is sent) and/or specify the destination path of the sent file(s) in the target metaframe.

What Data Can Be Transferred?

Container Metaframes

For containers, data is copied to cloud storage (by default), and a reference is sent downstream to avoid unnecessary large file uploads/downloads. (and subsequently downloaded by the worker).

JavaScript Metaframes

JavaScript frames can send various data types directly:

These objects arrive fully typed at the destination metaframe.

Output Filtering

The Source name or glob parameter lets you control which files are sent using glob patterns:

PatternDescriptionExample
** or emptyAll filesMatches everything
*.jsonFiles with specific extensionMatches data.json but not data/info.json
**/*.jsonFiles with specific extension in any folderMatches both data.json and data/info.json
data/*Files in a specific folderMatches data/file.txt but not data/subfolder/file.txt
**/report*Files with specific name pattern in any folderMatches report.csv, reports.xlsx, and data/report_final.pdf

The table below illustrates the necessary output filter for passing certain files:

Output nameOutput filterPassed downstream?
any output(empty string)
any output**
foo.bar*.bar
dir1/foo.bar*.bar
dir1/foo.bar**/*.bar
dir1/foo.bar*/foo

Input Mapping

The Target file or folder parameter determines how files appear in the target metaframe:

Mapping TypeDefinitionResult
Same Path (empty)Leave target mapping emptyFiles keep the same name and path structure
Directory Prefix (ends with /)Specify a folder name with trailing slashFiles are placed in that folder in the target
File Rename (doesn't end with /)Specify a filenameSource files are renamed to this single name

The input part of the connection can be:

OptionBehavior
EmptyIf empty, the filtered outputs are passed as inputs without modification to the path or name
Directory name (ends with /)The directory name will be prepended to the full path of the input(s)
File name (does not end with /)Upstream output(s) are renamed as the input file name. If multiple outputs match, then only one will be the input

Examples

ConnectionSource OutputTarget InputDescription
*.json → data//outputs/results.json/inputs/data/results.jsonSend all JSON files to a folder
report.csv → summary.csv/outputs/report.csv/inputs/summary.csvRename a specific file
** → or empty/outputs/data/file.txt/inputs/data/file.txtSend all files without changes
**/*.csv → reports//outputs/june/sales.csv/inputs/reports/june/sales.csvSend nested CSV files to a folder

Working with Inputs and Outputs in Different Metaframes

Each type of metaframe has its own way of handling inputs and outputs. This section provides implementation details for developers creating or working with specific metaframe types.

Container (Docker) Metaframes

In container metaframes, inputs and outputs are accessible as files in specific directories:

  • Inputs are available in the /inputs/ directory
  • Outputs should be written to the /outputs/ directory

For example, if a container metaframe receives a file called data.json, it would be accessible at /inputs/data.json.

JavaScript Metaframes

JavaScript metaframes handle inputs and outputs through JavaScript functions:

// Handling inputs
export function onInputs(inputs) {
// inputs is a plain object with keys and values
console.log(inputs.mydata);
}

// Sending outputs
setOutput("outputname", 42); // Send a single output
setOutputs({ // Send multiple outputs
result: true,
data: [1, 2, 3]
});

Output values can be:

  • Strings
  • JSON
  • Objects
  • Arrays
  • Numbers
  • ArrayBuffers
  • Typed arrays

You can define expected inputs and outputs in the Settings panel, which makes it easier to connect metaframes in the metapage editor.

Python (Pyodide) Metaframes

See pyodide documentation