Jets Homepage

JetsMachine handbook

1. Introduction
2. System requirements
3. Installation
4. Project - Task - Job
5. Main scripts
6. Parallelism
7. Example

Introduction

Lets start with an example. Given a PDE with the right hand side depending on some unknown function f(u), e.g. u_t = f u_x + u_xx, solve the following problem: how does the symmetry algebra depend on f?

Above problem may be solved using Jets (see Chapter 10 Classification problems of JetGuide), but there are several tedious steps, which must be done by the hand repetitively.

JetsMachine aka Distributed Jets is a tool which (partially) eliminates the manual interventions when solving such a classification tasks (at least, when the resolving failures are linear). Moreover, it may be run in a parallel way on a multi-core machine or within several computers.

System requirements

Basic user knowledge of Jets.
Unix-like command line (e.g. bash).
Maple licence (of Maple version supported by Jets).
Command-line version of Maple properly installed.
Optionally: For the setup distributed within several machines, shared network directory (e. g. samba or nfs share) available to all the working machines.

Installation

When single computer is used, prepare empty writable directory. In the multiple machines scenario, setup a network shared writable directory on all the working computers.
Installation consists of cloning the git repository
> git clone https://gitlab.slu.cz/bar0001/JetMachine
Alternatively, if git is not available, download the zipped source tree, unzip it and rename the result to JetMachine.

Make sure Maple is installed, licence is available and command line is ok. maple terminal command must invoke a Maple command line. Test whether 1+1 is 2 and Jets.s is available:
> maple

			
    |\^/|     Maple 2015 (APPLE UNIVERSAL OSX)
._|\|   |/|_. Copyright (c) Maplesoft, a division of Waterloo Maple Inc. 2015
 \  MAPLE  /  All rights reserved. Maple is a trademark of
 <____ ____>  Waterloo Maple Inc.
      |       Type ? for help. 
> 1+1;
       2
> read ("mc/Jets.s");
Jets 5.95  as of May 11, 2022 (2023-03-14 09:58:42)
>

Place your task init file to mc/ directory and use bin/Prepare and bin/Run scripts to prepare the root job and run the computations, see bellow.

Optional gadget:
You do not have to use the installation directory as your working directory. Instead, you can make a new working directory anywhere else (within a shared drive if multiple machines used!), cd to it and softlink the local bin/ scripts directory and mc/* source files to the global repository (i.e. to ..../JetMachine/bin and ..../JetMachine/mc/*, resp). There is a script for such a purpose, try (in the new empty working directory)
> cd somewhere > path/to/JetMachine/bin/InstallLinks
You may check whether your links are ok:
> ls bin


            CreateDirs OpenErrorsLogs Statistics DeleteAll OpenSuccess ...

> ls mc


            CommonProc.mc SingleStep.mc Jets.s ...

Project - Task - Job

In the project directory, several independent tasks may be placed. Each task has its unique TaskID (consisting of a few letters). The appropriate mc/TaskID.init.mc file is the place where the task is set up. The task is a tree whose nodes are jobs.

The task can be initialized by creating its root job TaskID.1 by invoking the script bin/Prepare TaskID. At the beginning, every task has its root job only; this root job branches out to the tree of consequent child jobs.

The job is the node in a given task branching tree; no matter it is already processed (and what the result was), under execution or still waiting for processing.

Every job has its unique JobID. The JobID is a concatenation of the TaskID and a position of the job in the task tree.

The primary approach used in implementing the main script bin/Run (the "worker") is based on moving job files, which represent different pieces of work (jobs), between pre-defined subdirectories. This involves moving the job files from the Ready/ directory to the InProgress/ directory, and finally to the Done/ directory. While the job is running, its maple session is logged and finally some status/report/success files are created. When the job forks into several new child branches, new child jobs (defined by the newly created job files) are generated in the Ready/ directory.

Job states and .runme job file

The job is defined in its JobID.runme job file. In this file, there are (among other technical stuff) encoded an assumptions additional to its parent job(s).

Every job is at every moment in a one of well-defined states. The state of the job is determined by the location of corresponding job file JobID.runme:

Job file JobID.runme location Job state Job state description

Ready/ Ready Ready to be run, waiting for processing

InProgress/Machine/JobID/ In progress Currently under execution.

Done/ Done Done (no matter what the job result is but without error)

Errors/Machine/JobID/ Crashed Error or exception raised during the processing

Be careful when you moving .runme file manually and do not move job files which are under execution!

Job results

As a result of job processing, several files at multiple places may be created (and the job file is moved as described above). The result of the job is determined by Jets's run() command final message.

run() result	Job status	Destination directory where `JobID.runme` has moved	File(s) created	Notes
`Success!`	Done	`Done/`	`Result/JobID.success`	Result found. Report file created. You can setup this report in `PrintSuccessReport()` function. Better than editing this function is placing its user version to the `mc/TaskID.init.mc`.
`Linear failure`	Done	`Done/`	Several `Ready/newJobID.runme` child jobs files	Forked, child jobs created.
`Nonlinear failure`	Done	`Done/`	`Ready/JobID.nonlinfail`	Waiting for manual intervention. Resolving (and child forks) must be handled manually.
`1=0`	Done	`Done/`	`Abandoned/JobID.ce`	Contradictory equations, job is finished.
`Nonimportant state`	Done	`Abandoned/`	`Abandoned/JobID.nonimportant`	Job abandoned, detected as nonimportant. No child jobs created. Setup `TestOfNonImportantness()` function of your `mc/TaskID.init.mc` file to define the nonimportantness.
`Error`	Crashed	`Errors/`	See `Logs/JobID.log`	Maple exception raised. The whole job directory has moved to `Errors/Machine/JobID`

How is the job run

We can process jobs waiting in Ready/ directory by invoking the bin/Run script. It is an infinity loop, whose iterations works in the following way:

Some job file Ready/JobID.runme is moved away to InProgress/Machine/JobID/
Job is executed in the new Maple session:
1. task init file mc/TaskID.init.mc is loaded,
2. parent job state is recovered from States/ParentJobID.state,
3. a new assumption (given in the job file) is set and
4. Maple (Jets) run(S) command is invoked (where the argument S is the global variable initialised in the task init file). Hurrah, computation is running.
5. Once run(S) terminates, computation state data are stored in States/JobID.state.
6. Depending on the run(S) result:
  - New child jobs (defined by creating its job Ready/JobIDn.runme files) are created or
  - success report is created or
  - other report files created (nonlinear failure, nonimportant state etc.) or
  - everything is moved to Errors/Machine/JobID/ directory.
The job file is moved to Done/

During the job execution, the log of Maple job processing is stored in the Logs/JobID.log file. This log is being written during the processing so you can look to the log for progress of the job, reason of failure or details about the result.

When the job is done (not matter what the result is), there is created States/JobID.state file where the result of Jets store() command is stored.

Stopping jobs

The running job cannot be stopped (except 'killing' its process, see bellow), but you can avoid starting of new computations.

To stop grabbing of any new jobs, create the file InProgress/stop (no matter what the content is). There will be no new computation started until you delete (or rename) this file.

To stop grabbing new jobs on some particular machine you can create the file InProgress/Machine/stop which triggers off grabbing of new jobs on the machine Machine (where the bin/Run command is running).

When you kill the job process on the working machine, you have to clean up its working directory InProgress/Machine/JobID, i. e.

move the file InProgress/Machine/JobID/JobID.runme to the Ready directory (or somewhere else, see bellow for Ready/Sleeping) and

delete the whole InProgress/Machine/JobID directory.

Beware of not deleting .runme file.

Putting jobs away from the processing

When you want a particular job ready for processing to be put away (because you know it is too exhausting or you are interested in other forks), you can move its .runme job file away from Ready directory. For example, you can use Ready/Sleeping directory for such a purpose.

Main scripts overview

All the commands are scripts stored in the bin/ directory. Usually, we use a softlink bin in the project home directory which points to the shared bin/ directory common for several projects.

Bellow, by {x|y} we mean optional x or y (exclusive). The star is well-known unix * operator: by x* we mean that x and all considerable symbols beginning with x are taken (e.g. abc* matches abc, abcd, abcde, abc1 etc. but does not abz).

In general, low-case commands are advanced and should not be used by the common user.

Management scripts

bin/Prepare TaskID
prepares the task for the processing. Already existing files (or results) are deleted. Root job file Ready/TaskID.1.runme is prepared.

bin/DeleteAll {TaskID | JobID}
deletes all TaskID* tasks and/or JobID* jobs (including results, states, reports and auxiliary files).
Without the parameter, deletes the whole project.
If TaskID is specified, all matching tasks are deleted.
If JobID is specified, deletes all the files related to the jobs specified by JobID and all its consequences so the matching branch is cut off (the parameter is extended by star and used as an unix mask).

Getting information

bin/Statistics
counts number of jobs in all the possible states (to be run, done, in progress, ready for processing, errors etc).

bin/ListErrors {TaskID | JobID} bin/CheckErrors {TaskID | JobID}
gives a short list of errors, resp. makes a deep check (looking for the word error in all the logfiles).

Running the computations

On the computer(s), where computations are to be executed (Maple must be available!), we can run the worker by the script
bin/Run {TaskID | JobID}
which runs continuous processing of JobID* jobs, i.e. it will grab the job JobID and/or all its childs. Without the parameter, it will process any job from Ready/.

The unix & operator or nohup command for background processing may be used in conjunction with nice. For example, try
ssh user@machine cd projectdir nohup nice bin/Run TaskID &

Parallelism

Single job is not parallelised nor multi-threaded due to Maple limitations (this may be changed in the future Maple versions).

Multiple instances of bin/Run worker script may be run in several sessions within multiple machines, which allows computing of the task tree in the parallel way.

This parallelism rely on the requirement that all the processes on all machines uses a single shared working directory. In the case of single multi-core machine, all the sessions simply uses the same working directory and no network file sharing is required. If multiple machines involved, network share must be set up (e. g. samba or nfs). Beware of bin/ and mc/ softlinks in such a case. When many workers are run, the file server may be overloaded soon.

Example

Lets go back to the introductory example, looking for the u_t = f u_x + u_xx symmetry algebra, where f(u) is a function.

The manual approach is described in Chapter 10 Classification problems of JetGuide (the classification was done by A.M. Vinogradov, I.S. Krasil’shchik and V.V. Lychagin, Geometry of Jet Spaces and Non-linear Partial Differential Equations, 1986).

To treat this example without manual interventions, run your terminal app or ssh to your machine and:

Go to your home directory (or any writable directory) and install as described above by invoking
> git clone https://gitlab.slu.cz/bar0001/JetMachine
Our example is already prepared:
> cd JetMachine/Examples/JetGuideExamples/JetGuideClassificationExample
This will be your working directory.
Study the file mc/JetGuideClassEx.init.mc where our example task is defined.
Invoke the script
> ../../../bin/InstallLinks
and press enter to continue (due to existing mc/JetGuideClassEx.init.mc file).
Links to ../../../JetMachine/mc/* (source files) and ../../../JetMachine/bin (scripts directory) are installed into the current working dir.
Try
> ls -l
lrwxr-xr-x bin -> ../../../bin drwxr-xr-x mc
> ls bin
CreateDirs OpenErrorsLogs Statistics DeleteAll OpenSuccess ...
> ls mc
CommonProc.mc SingleStep.mc Jets.s ...
If get in troubles, use the main JetMachine/ directory as a working directory and simply copy JetMachine/Examples/JetGuideExamples/JetGuideClassificationExample/mc/JetGuideClassEx.init.mc to the JetMachine/mc/ working directory.
Initialize the task configured in mc/JetGuideClassEx.init.mc by command
> bin/Prepare JetGuideClassEx
The root job of the classification task tree is now ready:
> ls Ready JetGuideClassEx.1.runme
Run the worker and wait for the results
> bin/Run
Run started at 10:25:52 14/03/23 on host myhost.slu.cz, pid is 64768 JetGuideClassEx.1 (10:25:52 14/03/23) done (10:25:54 14/03/23). JetGuideClassEx.11 (10:25:55 14/03/23) done (10:25:59 14/03/23). JetGuideClassEx.12 (10:26:00 14/03/23) done (10:26:02 14/03/23). JetGuideClassEx.112 (10:26:03 14/03/23) done (10:26:04 14/03/23). JetGuideClassEx.111 (10:26:05 14/03/23) done (10:26:07 14/03/23). JetGuideClassEx.1112 (10:26:08 14/03/23) done (10:26:10 14/03/23). JetGuideClassEx.1111 (10:26:11 14/03/23) done (10:26:13 14/03/23). ls: Ready/*.runme: No such file or directory ls: Ready/*.runme: No such file or directory
Once
ls: Ready/*.runme: No such file or directory
message appears, we are done, since there are no more waiting jobs in Ready/. We can stop the worker now by pressing ^C.
The results overview command
> bin/Statistics
shows we have 4 success results.
> ls Results
Open the Results/*.success files to see the results.

Jets Homepage Distributed Jets project

Job file `JobID.runme` location	Job state	Job state description
`Ready/`	Ready	Ready to be run, waiting for processing
`InProgress/Machine/JobID/`	In progress	Currently under execution.
`Done/`	Done	Done (no matter what the job result is but without error)
`Errors/Machine/JobID/`	Crashed	Error or exception raised during the processing