MagicTree Documentation: Commands Execution

< Back Reference Variables in XPath Queries | Table of Contents | Scripts and Samples >

MagicTree allows user to invoke commands, pass input data and arguments to the commands and gather output data from the commands. Commands can be invoked locally or remotely over SSH. Commands are executed in the background, using "screen' tool. User can attach to the console of the running command and interact with it.

Commands execution assumes UNIX-like environment. Minimal Cygwin installation is bundled with MagicTree.

Command

The text of the command is executed by BASH shell as is. MagicTree does not interpret it in any way (internally it writes the command into a file and tries to execute it). As any shell script, it can contain multiple commands separated with semicolons, pipes, input redirection etc.

Input

Input data, a result of a table query can be fed to a command via environment variables or tab-separated on stdin.

When data are passed in environment variables, the application executes one command per row of data. Value from each column in the table is stored in an environment variable with name equal to the name of the column. Consider the following table query results:

host | port
192.168.1.1 | 22
192.168.1.2 | 23

and the following command:

amap -h $host -p $port

The application will run two commands

amap -h 192.168.1.1 -p 22
amap -h 192.168.1.2 -p 23

In tab-separated mode, a tab-separated version of the table view results will be generated and stored in a temporary file. The command can refer to that file using $in environment variable.

Output Files

The command can generate output files and have the picked up by the application afterwards. For this to work, output file names must be created using $out environment variable.

nmap -sS 192.168.1.1 -oX $out

Environment variable $out is set by the application to a temporary path. Any file with a name starting with $out will be loaded by the application after the command exits. For example, the following command:

uname > $out ; hostname > $out.abc

will produce two files (not counting screenlog)

Console Log

While the command is running its console output gets logged. This output is produced by "screen' tool which automatically interprets some of the escape characters to ensure nice appearance of interactive sessions. If you require a verbatim copy of the output, you can redirect stdout/stderr to a file.

Interactive Commands

Thanks to the use of "screen' tool, user can attach to a running command at any time. (Unfortunately it does not work perfectly under Windows, see Known Bugs.) This is useful for commands requiring user interaction, such as sudo.

Detached Execution

All commands gets executed by "screen' tool in the background, independently from MagicTree process. The user can save the project, re-start MagicTree, open saved project, and access the commands launched earlier. This works for both local and remote commands, transparently to the user.

Remote Execution

A command can be executed on a remote host. The only requirement is OpenSSH client installed on local host and "screen' tool has to be installed on the remote host. MagicTree was tested against remote hosts running Linux, but should works with other UNIX-like OSes and Cygwin.

For remote execution to work, it is necessary to arrange for MagicTree to be able to log into remote host without a password, for example by generating SSH public/private key pair and placing the public key into .ssh/authorized keys file on the remote host.

Press "Push SSH Key' button to run a configuration script facilitating this task. When prompted, enter "username@host'. When SSH client asks for a password, enter it. MagicTree will append newly generated key to ~/.ssh/authorized keys. The keys have a specific comment "Autogenerated by MagicTree', so you can easily identify and remove them later if necessary. Pay attention to error messages generated by the script. If everything went fine it will say "Succeeded!". After this SSH key is pushed to the remote host, you can execute commands there by specifying the same "username@host' as host when launching command. You should specify the same "username@host' to avoid problems.

If you have to configure this manually for whatever reason, place id rsa, id rsa.pub, and known hosts files under .ssh directory under mtdir. Also see bin/config file under mtdir to get more clue how ssh client gets invoked.

Known Bugs

Couple bugs on Cygwin platform:

"Screen' tool does not work well in Cygwin. In particular, an attempt to connect to the console of a running command will fail for the first command. It will work fine for subsequent invokation though. The problem was reported to Cygwin team, but they say they don't have resources to troubleshoot the problem.
Asyncronous I/O between parent process and a subprocess is a bit crippled in Java on Windows (at least when it deals with Cygwin subprocesses). If a subprocess sends something to stderr deadlock likely to occur. To troubleshoot: kill Cygwin processes (from Windows Task Manager) and check the content of stderr in the magictree.log file. Do the necessary to prevent subprocesses from displaying anything to stderr.

There are also the following design flaws, which we hopefully will resolve in the next versions:

MagicTree limits the number of concurrently running processes per host (local or remote). The limit is introduced to prevent spawning thousands of processes by executing a command in env input mode with a large input table. We are considering introducing a configurable per-method concurrency limit instead.
Detached execution is implemented as described above, but it is limited to already started commands. Commands which are scheduled for execution, but not started yet (for example because of the concurrency limit) will not be started if MagicTree is not running.