Workfow Management and Meta Job Scheduler

This tutorial illustrates how computations can be executed on large (or small) compute servers, aka. high-performance-computers (HPC), aka. compute clusters, aka. supercomputers, etc.

This means, the computation is not executed on the local workstation (or laptop) but on some other computer. This approach is particulary handy for large computations, which run for multiple hours or days, since a user can e.g. shutdown or restart his personal computer without killing the compute job.

BoSSS features a set of classes and routines (an API, application programing interface) for communication with compute clusters. This is especially handy for scripting, e.g. for parameter studies, where dozens of computations have to be started and monitored.

First, we initialize the new worksheet; Note:

  1. This tutorial can be found in the source code repository as as MetaJobManager.ipynb. One can directly load this into Jupyter to interactively work with the following code examples.
  2. In the following line, the reference to BoSSSpad.dll is required. You must either set #r "BoSSSpad.dll" to something which is appropirate for your computer (e.g. C:\Program Files (x86)\FDY\BoSSS\bin\Release\net5.0\BoSSSpad.dll if you installed the binary distribution), or, if you are working with the source code, you must compile BoSSSpad and put it side-by-side to this worksheet file (from the original location in the repository, you can use the scripts getbossspad.sh, resp. getbossspad.bat).

Batch Processing

First, we have to select a batch system (aka.execution queue, aka. queue) that we want to use. Batch systems are a common approach to organize workloads (aka. compute jobs) on compute clusters. On such systems, a user typically does not starts a simulation manually/interactively. Instead, he specifies a so-called compute job. The scheduler (i.e. the batch system) collects compute jobs from all users on the compute cluster, sorts them according to some priority and puts the jobs into some queue, also called batch. The jobs in the batch are then executed in order, depending on the available hardware and the scheduling policies of the system.

The BoSSS API provides front-ends (clients) for the following batch system software:

A list of clients for various batch systems, which are loaded at the Init() command can be configured through the
~/.BoSSS/etc/BatchProcessorConfig.json-file. If this file is missing, a default setting, containing a mini batch processor, is initialized.

The list of all execution queues can be accessed through:

In order to run a simulation job, one can either manually select one of these queues -- or, one culd just use the default queue. The default queue for execution can be configured by two options:

Note on the Mini Batch Processor:

The batch processor for local jobs can be started separately (by launching MiniBatchProcessor.exe or dotnet MiniBatchProcessor.dll), which is the prefferred option. Alternatively, it can be started from Jupyter Notebook; it depends on the operating system, whether the MiniBatchProcessor.exe is terminated with the notebook kernel, or not. If no mini-batch-processor is running, it is started (hopefully) upon Job activation.

Initializing the workflow management

In order to use the workflow management, the very first thing we have to do is to initialize it by defineing a project name, here it is MetaJobManager_Tutorial. This is used to generate names for the compute jobs and to identify sessions in the database:

For this project, the default execution queue is set to:

We verify that we have no jobs defined so far ...

The initialization of the Workflow Management environment already creates, resp. opens a BoSSS database with the same name as the project name as the project. The current default database is set as:

Notes on databases:

All currently opened databases can be listed using:

Loading a BoSSS-Solver and Setting up a Simulation

As an example, we use the workflow management tools to simulate incompressible channel flow, therefore we have to import the namespace, and repeat the steps from the IBM example (Tutorial 2) in order to setup the control object:

We create a grid with boundary conditions:

One can save this grid explicitly to a database, but it is not a must; The grid should be saved automatically, when the job is activated.

Next, we create the control object for the incompressible simulation:

The specification of boundary conditions and initial values is a bit more complicated if the job manager is used:

Since the solver is executed in an external program, the control object has to be saved in a file. For lots of complicated objects, especially for delegates, C# does not support serialization (converting the object into a form that can be saved on disk, or transmitted over a network), so a workaround is needed. This is achieved e.g. by the Formula object, where a C#-formula is saved as a string.

Testing the formula:

Finally, we set boundary values for our simulation. The initial values are set to zero per default; for the steady-state simulation initial values are irrelevant anyway:

Initial Values are set to 0

Activation and Monitoring of the the Job

Finally, we are ready to deploy the job at the batch processor; In a usual work flow scenario, we do not want to (re-) submit the job every time we run the worksheet -- usually, one wants to run a job once.

The concept to overcome this problem is job activation. If a job is activated, the meta job manager first checks the databases and the batch system, if a job with the respective name and project name is already submitted. Only if there is no information that the job was ever submitted or started anywhere, the job is submitted to the respective batch system.

First, a `Job* -object is created from the control object:

This job is not activated yet, it can still be configured:

Starting the compute Job

One can change e.g. the number of MPI processes:

Note that these jobs are desigend to be persistent: This means the computation is only started once for a given control object, no matter how often the worksheet is executed.

Such a behaviour is useful for expensive simulations, which run on HPC servers over days or even weeks. The user (you) can close the worksheet and maybe open and execute it a few days later, and he can access the original job which he submitted a few days ago (maybe it is finished now).

Then, the job is activated, resp. submitted, resp. deployed to one batch system. If job persistency is not wanted, traces of the job can be removed on request during activation, causing a fresh job deployment at the batch system:

All jobs can be listed using the workflow management:

Check the present job status:

Evaluation of Job

Here, we block until both of our jobs have finished:

We examine the output and error stream of the job: This directly accesses the \tt stdout-redirection of the respective job manager, which may contain a bit more information than the Stdout-copy in the session directory.

Additionally we display the error stream and hope that it is empty:

We can also obtain the session which was stored during the execution of the job:

We can also list all attempts to run the job at the assigend processor:

Finally, we check the status of our jobs:

If anything failed, hints on the reason why are provides by the GetStatus method:

Exporting Plots

Each run of the solver corresponds to one session in the database. A session is basically a collection of information on the entire solver run, i.e. the simulation result, input and solver settings as well as meta-data such as computer and daten and time.

Since in this tutorial only one solver run was executed, there is only one session in the Workflow Management (wmg is just an alias for BoSSSshell.WorkflowMgm.):

We select the first (and only) session and create an export instruction object. The supersampling setting increases the output resolution. This is required to vizualize high-order DG ploynomials with the low-order Tecplot-format. Tecplot can only vizualize a linear interpolation within a cell. With a second-degree supersampling, each cell is subdivided twice (in 2D, one subdivision is 4 cells, i.e. 2 subdivisions are $4^2 = 16$ cells). In this way, the curve of e.g. a secondd order polynomial can be represented with the linear interpolation over 16 cells.

On the respective directory (see output above) one should finaaly find plot-files which than can be used for further post processing in third-party software such as Paraview, LLNL Visit or Tecplot.

The Do() command returns the location of the output files:

To finalize this tutorial, we list all files in the plot output directory: