Dynamic Configurable Live Leaderboards of Workflows
Problem: evaluating AI models and similar tools is time consuming, often bespoke, replicates work done by others, and is difficult to update.
Solution: web-first, automated, dynamic leaderboards, where the leaderboard entries are fully functional workflows, and the results are automatically collated and presented. The leaderboard is itself a dynamic, live, workflow.
Example web-first, dynamic live leaderboard
https://metapage.io/m/ab7a2dc814fa46fabb2aafac75061583
This example compares two methods of predicting a protein structure only using a FASTA file as a reference, using custom AI models. The two methods are:
Notable features
Dynamic competition inputs
This leaderboard provides the target input sequence. This input is dynamic, if it changes, then all downstream workflows have their input updated
Dynamic workflow entries
When the leaderboard entry workflows have their input updated, then the workflow can the be re-run with a single click, and the results are then updated automatically.
All source code and running containers are fully available and are self-contained components
Unlike other leaderboards that at most might point to a github repo of code, all components are either self-contained javascript, or self-contained docker containers. Every component can be copied and taken individually, so that everyone benefits from the work of everyone else.
The leaderboard can be copied and run on your own workstations/infrastructure
Due to the public compute grid infrastructure, all workflows can be run by anyone, provided they have sufficient compute resources attached to their own compute queues.
The workflows are all shareable URLs
Not installation needed, no configuration. Provided you have sufficient computation resources, the workflows run without any complex time-intensive steps, directly in the browser. Heavy compute is abstracted away.
In Summary:
- open source, all source code embedded in the workflow
- copyable with a single click (when logged in)
- computation containers just work, no installation needed, just runs in the browser
The future of human collaboration requires automation and open-sharing infrastructure.