guile-ssh/doc/api-dist.texi

@c -*-texinfo-*-
@c This file is part of Guile-SSH Reference Manual.
@c Copyright (C) 2015, 2016 Artyom V. Poptsov
@c See the file guile-ssh.texi for copying conditions.

@node Distributed Forms
@section Distributed Forms

@cindex secure distributed computing

The @code{(ssh dist)} module provides the spirit of distributed computing for
Guile.

To make use of the procedures listed in this section you will need an SSH
daemon and a GNU Guile REPL server both running on the remote host.

Also note that currently there may be cases in which distributed procedures
may fail to (de)serialise data; namely @code{make-vector} is one of such
procedures which output may be troublesome for @code{with-ssh}.  To overcome
this specific case one could pass the @code{fill} argument to
@code{make-vector} to fill the newly created vector with the specified filling
instead of @code{#<unspecified>}.

Node management procedures:

@deffn {Scheme Procedure} make-node session [repl-port=37146] [#:start-repl-server?=#t] [#:stop-repl-server?=#f]
Make a new node that uses an SSH @var{session} to connect to a @var{repl-port}
number on the remote side.  Return a new node.

If @var{start-repl-server?} is set to @code{#t} (which is by default) then
start a REPL server on a remote host automatically in case when it is not
started yet.

If @var{stop-repl-server?} is set to #t then a REPL server will be stopped as
soon as an evaluation is done.  Alternatively you could use
@code{node-stop-server} procedure from @code{(ssh dist node)} to stop the
server when it is not needed anymore.

Note that when either @var{start-repl-server?} or @var{stop-repl-server?} are
specified, Guile-SSH tries to use @command{pgrep}/@command{pkill} from
@url{https://gitlab.com/procps-ng/procps, procps} package first to start/stop
a remote Guile process, but if the package is missing then the Guile-SSH falls
back to its own implementation of the commands (for more details see
@pxref{Shell, Guile-SSH shell API})

@end deffn

@deffn {Scheme Procedure} node? x
Return @code{#t} if @var{x} is a node object, @code{#f} otherwise.
@end deffn

@deffn {Scheme Procedure} node-session node
Get underlying SSH session from @var{node}.
@end deffn

@deffn {Scheme Procedure} node-repl-port node
Get REPL port number from a @var{node}.
@end deffn

@deffn {Scheme Procedure} node-loadavg node
Get average load of a @var{node}.  Return multiple values.  The 1st value is
an alist of five elements as described in proc(5) man page.  The rest of
values are as described in documentation for @code{node-eval} procedure.

For example:

@lisp
(use-modules (ssh auth)
             (ssh session)
             (ssh dist node))

(let ((s (make-session #:host "example.org")))
  (connect! s)
  (userauth-agent! s)
  (let ((n (make-node s)))
    (node-loadavg n)))
@result{} ((one . 0.15) (five . 0.14) (fifteen . 0.16) (scheduling-entities 1 189) (last-pid . 15629))
@result{} 1
@result{} "(guile-user)"
@result{} "scheme"
@end lisp

@end deffn

Interaction with remote REPLs:

@deffn {Scheme Procedure} distribute nodes expr ...
Evaluate each @var{expr} in parallel, using distributed computation.  Split
the job to nearly equal parts and hand out each of resulting sub-jobs to
@var{nodes} list.  Return the results of N expressions as a set of N multiple
values (@pxref{Multiple Values,,, guile, The GNU Guile Reference Manual}).
@end deffn

@deffn {Scheme Procedure} dist-map nodes proc lst
Do list mapping using distributed computation.  Split the work into nearly
equal parts and hand out the resulting jobs to @var{nodes} list.  Return the
result of computation.

If for some reason a job could not be executed on a node (for example, if
connection to a remote REPL fails), @code{dist-map} transfers the job to
another node from the @var{nodes} list.  When job execution failed on all
nodes, an error is reported.

In a case when an error that occured during job execution is considered
non-recoverable (eg. when evaluation of @var{proc} on a node failed due to an
unbound variable) then execution of a job stops immediately.
@end deffn

@deffn {Scheme Procedure} with-ssh node exp ...
Evaluate expressions on a remote REPL using a @var{node}, return four values:
an evaluation result, a number of the evaluation, a module name and a language
name.  Throw @code{node-error} or @code{node-repl-error} on an error.

Example:
@lisp
(use-modules (ssh session)
             (ssh auth)
             (ssh dist))

(let ((session (make-session #:user "alice" #:host "www.example.org")))
  (connect! session)
  (userauth-agent! session)
  (display (with-ssh (make-node session)
             (gethostname)))
  (newline))
@end lisp

If an expression is evaluated to multiple values then the 1st value returned
by @code{with-ssh} will be a vector of the evaluated values and the 2nd value
will be a vector of evaluation numbers.  In this case the 2nd value can be
used to check whether @code{with-ssh} body evaluated to multiple values or
not.  For example:

@lisp
(use-modules (ssh session)
             (ssh auth)
             (ssh dist))

(let ((session (make-session #:user "alice" #:host "www.example.org")))
  (connect! session)
  (userauth-agent! session)
  (with-ssh (make-node session)
    (values 1 2)))
=> #(1 2)
=> #(39 40)
=> "(guile-user)"
=> "scheme"
@end lisp

@end deffn

@deffn {Scheme Procedure} rrepl node
Start an interactive remote REPL (RREPL) session using @var{node}.
@end deffn

@subsection Low-level API

@subsubsection Nodes
@tindex node

The module @code{(ssh dist node)} provides low-level API for node management.
Here's the description of the format of node type printed representation:

@example
#<node avp@@127.0.0.1:22/37146 a0dbdc0>
       A   A         A  A     A
       |   |         |  |     |
   ,---'   |       ,-'  '---. '-----------.
   |       |       |        |             |
 user     host   port   REPL port    object address
@end example

There are two types of node errors: recoverable and non-recoverable.  The
first group is represented by @code{node-error} exceptions.  If an exception
of this kind is occured then there is a chance that a job can be executed on
another node.  That's because such an exception occures in cases when a node
is unreachable, for example.  The second group is represented by
@code{node-repl-error} exceptions.  Such exceptions mean that an error is
occured during execution of a job on a node's REPL -- eg. due to the malformed
job.  Those errors are non-recoverable because if the job is broken it will
likely fail on another nodes as well.

In addition to @code{make-node}, @code{node?}, @code{node-session} and
@code{node-repl-port} the module provides:

@deffn {Scheme Procedure} node-eval node quoted-exp
Evaluate a @var{quoted-exp} on a @var{node} and return four values: an
evaluation result, a number of the evaluation, a module name and a language
name.  Throw @code{node-repl-error} if a non-recoverable error occured, or
@code{node-error} if the evaluation potentially could be succesfully evaluated
on another node.

@strong{Note} that @url{https://gitlab.com/procps-ng/procps, procps} version
3.3.12 or later is needed on the server side in case of either
@code{start-repl-server?} or @code{stop-repl-server?} was set to @code{#t} for
a @var{NODE} (see the documentation for @code{make-node}.)

@end deffn

@deffn {Scheme Procedure} node-open-rrepl node
Open a remote REPL (RREPL).  Return a new RREPL channel.
@end deffn

@deffn {Scheme Procedure} node-run-server node
Run a REPL server on a @var{node}.
@end deffn

@deffn {Scheme Procedure} node-stop-server node
Stop a RREPL server on a @var{node}.
@end deffn

@deffn {Scheme Procedure} node-guile-version node
Get Guile version installed on a @var{node}, return the version string.
Return @code{#f} if Guile is not installed.
@end deffn

@deffn {Scheme Procedure} node-server-running? node
Check if a REPL server is running on a @var{node}, return @code{#t} if it is
running and listens on an expected port, return @code{#f} otherwise.
@end deffn

@deffn {Scheme Procedure} rrepl-eval rrepl-channel expr
Evaluate expression @var{expr} using @var{rrepl-channel}, return four values:
an evaluation result, a number of the evaluation, a module name and a language
name.  Throw @code{node-repl-error} on an error.
@end deffn

@deffn {Scheme Procedure} rrepl-skip-to-prompt rrepl-channel
Read from @var{rrepl-channel} until REPL is observed.  Throw @code{node-error}
on an error.
@end deffn

@subsubsection Jobs
@tindex job

The module @code{(ssh dist job)} provides low-level API for job management.

Here's the description of the format of node type printed representation:

@example
#<job map #<node avp@@127.0.0.1:22/37147 a0dbdc0> a1345a0>
      A   A                                      A
      |   |                                      |
      |   '----------.                           |
      |              |                           |
   job type   node (see above)           job object address
@end example

@deffn {Scheme Procedure} split lst count
Split a list @var{lst} into @var{count} chunks.  Return a list of chunks.

Example:
@lisp
(split '(a b c d) 2)
@result{} '((a b) (c d))
@end lisp
@end deffn

@deffn {Scheme Procedure} make-job type node data proc
Make a new job of @var{type} using @var{node}.
@end deffn

@deffn {Scheme Procedure} job? x
Return @code{#t} if @var{x} is a job object, @code{#f} otherwise.
@end deffn

@deffn {Scheme Procedure} job-type job
Get type of a @var{job}.
@end deffn

@deffn {Scheme Procedure} job-node job
Get a @var{job} node.
@end deffn

@deffn {Scheme Procedure} set-job-node job node
Transfer @var{job} to a new @var{node}.  Return a new job object.
@end deffn

@deffn {Scheme Procedure} job-data job
Get a @var{job} data.
@end deffn

@deffn {Scheme Procedure} job-proc job
Get a @var{job} procedure.
@end deffn

@deffn {Scheme Procedure} assign-eval nodes expressions
Split an @var{expressions} list to nearly equal parts according to the length
of a @var{nodes} list and assign each evaluation job to a node.  Return a list
of assigned jobs.
@end deffn

@deffn {Scheme Procedure} assign-map nodes lst proc
Split the work to nearly equal parts according to length of @var{nodes} list
and assign each part of work to a node.  Return list of assigned jobs.
@end deffn

@deffn {Scheme Procedure} hand-out-job job
Hand out @var{job} to the assigned node and return the result of computation.
@end deffn

@deffn {Scheme Procedure} job->sexp job
Convert a @var{job} to an equivalent symbolic expression.
@end deffn

@c Local Variables:
@c TeX-master: "guile-ssh.texi"
@c End: