Sending Data Between Metaframes
Overview
One of the core functions of Metapages is the transfer of data between metaframes. This connectivity is what enables you to build powerful workflows and data processing pipelines. Data flow can be configured through:
- The metapage's Data Flow tab
- The metaframe's connect button
Creating Data Connections
A data connection is specified using up to two parameters:
- Source name or glob: Defines which outputs to send from the source metaframe to the target metaframe. If you want to send a single file, this would be a string including the filename or file path if the file is located inside a folder. For sending multiple files, you can specify a glob with wildcards to match the files you want to send. If this field is left blank, all files routed to the outputs will be sent.
- Target file or folder: Allows you to change the name of the file (in the case only one file is sent) and/or specify the destination path of the sent file(s) in the target metaframe.
What Data Can Be Transferred?
Container Metaframes
For containers, data is copied to cloud storage (by default), and a reference is sent downstream to avoid unnecessary large file uploads/downloads. (and subsequently downloaded by the worker).
JavaScript Metaframes
JavaScript frames can send various data types directly:
These objects arrive fully typed at the destination metaframe.
Output Filtering
The Source name or glob parameter lets you control which files are sent using glob patterns:
Pattern | Description | Example |
---|---|---|
** or empty | All files | Matches everything |
*.json | Files with specific extension | Matches data.json but not data/info.json |
**/*.json | Files with specific extension in any folder | Matches both data.json and data/info.json |
data/* | Files in a specific folder | Matches data/file.txt but not data/subfolder/file.txt |
**/report* | Files with specific name pattern in any folder | Matches report.csv , reports.xlsx , and data/report_final.pdf |
The table below illustrates the necessary output filter for passing certain files:
Output name | Output filter | Passed downstream? |
---|---|---|
any output | (empty string) | ✅ |
any output | ** | ✅ |
foo.bar | *.bar | ✅ |
dir1/foo.bar | *.bar | ❌ |
dir1/foo.bar | **/*.bar | ✅ |
dir1/foo.bar | */foo | ✅ |
Input Mapping
The Target file or folder parameter determines how files appear in the target metaframe:
Mapping Type | Definition | Result |
---|---|---|
Same Path (empty) | Leave target mapping empty | Files keep the same name and path structure |
Directory Prefix (ends with / ) | Specify a folder name with trailing slash | Files are placed in that folder in the target |
File Rename (doesn't end with / ) | Specify a filename | Source files are renamed to this single name |
The input part of the connection can be:
Option | Behavior |
---|---|
Empty | If empty, the filtered outputs are passed as inputs without modification to the path or name |
Directory name (ends with / ) | The directory name will be prepended to the full path of the input(s) |
File name (does not end with / ) | Upstream output(s) are renamed as the input file name. If multiple outputs match, then only one will be the input |
Examples
Connection | Source Output | Target Input | Description |
---|---|---|---|
*.json → data/ | /outputs/results.json | /inputs/data/results.json | Send all JSON files to a folder |
report.csv → summary.csv | /outputs/report.csv | /inputs/summary.csv | Rename a specific file |
** → or empty | /outputs/data/file.txt | /inputs/data/file.txt | Send all files without changes |
**/*.csv → reports/ | /outputs/june/sales.csv | /inputs/reports/june/sales.csv | Send nested CSV files to a folder |
Working with Inputs and Outputs in Different Metaframes
Each type of metaframe has its own way of handling inputs and outputs. This section provides implementation details for developers creating or working with specific metaframe types.
Container (Docker) Metaframes
In container metaframes, inputs and outputs are accessible as files in specific directories:
- Inputs are available in the
/inputs/
directory - Outputs should be written to the
/outputs/
directory
For example, if a container metaframe receives a file called data.json
, it would be accessible at /inputs/data.json
.
JavaScript Metaframes
JavaScript metaframes handle inputs and outputs through JavaScript functions:
// Handling inputs
export function onInputs(inputs) {
// inputs is a plain object with keys and values
console.log(inputs.mydata);
}
// Sending outputs
setOutput("outputname", 42); // Send a single output
setOutputs({ // Send multiple outputs
result: true,
data: [1, 2, 3]
});
Output values can be:
- Strings
- JSON
- Objects
- Arrays
- Numbers
- ArrayBuffers
- Typed arrays
You can define expected inputs and outputs in the Settings panel, which makes it easier to connect metaframes in the metapage editor.