The goal of the project is to develop framework, that will distribute large job between workers and process it in parallel. Framework should be able to adapt to different conditions in order to analyse data as effectively as possible.
Run via docker/podman
Note: One can subsitute command podman
by docker
Simple cluster using UDP discovery
Cluster
podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa
Start monitoring (optional)
podman run -it --rm --net=host --entrypoint salsa-mon registry.openbrain.sk/salsa/salsa -u "tcp://*:5555"
Submit job
podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"
Simple cluster using GOSSIP discovery
Discovery
podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n disovery
Redirector
podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr
Worker
podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n wk
Submit job
podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"
Redirector and worker
podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr+wk
Only redirector with debug message and publisher set
podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr --debug
Extra option as cmd parameter
podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr{ip:192.168.1.20,port:35000}
Installation via yum/dnf
Install OpenBrain
repository
yum install yum-plugin-copr -y
yum copr enable mvala/ob -y
Install salsa package
Develop
Install all dependencies
yum install redhat-lsb cmake gcc-c++ zyre-devel spdlog-devel protobuf-devel jsoncpp-devel yaml-cpp-devel doxygen
Compile project
cd <project-dir>
scripts/make.sh
scripts/make.sh options
Option | Description |
install | Install after compilation. |
clean | Destroy build folder before building. |
clang | Use Clang compiler instead of system default. |
strict | Enable all warnings (requires clang ). Not -Werror ! |
ninja | Use Ninja build system instead of system default. |
doc | Build documentation (via Doxygen) |
release | Build as release. (This is used in pipeline.) Infers doc and strict . Warnings as errors. |
strict
option was created for development purposes, so you can compile with all warnings, while not failing build (no -Werror
).
If you are contributing, make sure that you pass compilation with at least release
option. Even better if you pass with release clang
(thanks to Clang's -Weverything
).
Example running master-worker arch from developemnt
$SALSA_ROOT/build/src/salsa -c $SALSA_ROOT/etc/config.json -n rdr,wk --debug # Runs redirector and 1 worker. Beware! Every worker gets all cores of machine!
$SALSA_ROOT/build/src/salsa-feeder -u <submiter-url> -f <path/to/list> # Command list is one command per line. (<submiter-url> : tcp://localhost:41000)
$SALSA_ROOT/build/src/salsa-feeder -u <submiter-url> -t "cmd arg1 arg2:N" # Using template switch executable is "cmd arg1 arg2" and it is done "N" times
Publishing info about jobs
Run before (on rdr)
export SALSA_PUB_URL=">tcp://localhost:5555"