CoMETS (the Cosmo Model Experimentation Toolbox) is a Python library aiming at simplifying and automating experimentation with our Cosmo Tech models.
For a non-expert end-user, the standard experimentation procedure is to construct What-If scenarios, by manually changing some lever value in a Cosmo Tech web application. It triggers a new scenario, in which the full run template is executed. The modifications are handled by the Parameter Handler: it modifies the lever to the user’s new input value.
In an alternative procedure, adapted to more advanced experimentation, we can define an ensemble of simulation runs directly in the run template engine. An example is the uncertainty analysis experiment implemented in our Supply Chain pre-packaged solution: a large number of simulations are launched together for one execution of the run template.
CoMETS aims at reproducing and extending this second experimentation procedure. It provides unified, easy-to-use and model agnostic tools to configure advanced experiments in the run template engine, using the Python language.
CoMETS has to be installed on the same machine as the simulator. An interface to communicate with the simulator can be built with a few lines of codes. Then, it provides a single, user-friendly syntax to call algorithms performing Optimization, Uncertainty Analysis and many more types of experimentation on the simulator.
CoMETS targets two usage modes depending on the user profile:
- Jupyter notebooks to perform experiments on a model you have locally on your machine: this mode will be preferred by modelers and data scientists
- Python scripts to be included in the run template engine: this mode will be used by solution integrators to deliver advanced experimentations to the cloud
Using CoMETS for a solution developer
- Interface to load and manipulate Cosmo simulators
- Interface to load and manipulate Python simulators
- Interface to load and manipulate CLI scripts as simulators
- Parallel manager for transparent multi-processing, including nested parallelism for compound experiments
- Interactive experiments (stop/resume an experiment, predefined and custom stopping criteria, etc…)
- Monitoring experiments (printing intermediate results, saving simulator evaluation history, etc)
- Logging mechanism for debugging
- Nevergrad library Algorithms (~ 100 algorithms)
- Optimization monitoring
- Support for floating point, integer and categorical variables
- Support for vector variables
- Uncertainty Analysis:
- Monte Carlo analyses
- Quasi Monte Carlo analyses
- Uncertainty source given by scipy distributions or custom generators
- Large choice of descriptive statistics
- Parameter Sweep
- Compounds of the above
- Documentation available on Cosmo Tech portal:
- Onboarding material and tutorials (available in the documentation)