In its most typical use, Salt is a software application in which clients, called “minions” can be commanded and controlled from a central command server called a “master”.
Commands are normally issued to the minions (via the master) by calling a client script simply called, ‘salt’.
Salt features a pluggable transport system to issue commands from a master to minions. The default transport is ZeroMQ.
The salt client is run on the same machine as the Salt Master and communicates with the salt-master to issue commands and to receive the results and display them to the user.
The primary abstraction for the salt client is called ‘LocalClient’.
When LocalClient wants to publish a command to minions, it connects to the master by issuing a request to the master’s ReqServer (TCP: 4506)
The LocalClient system listens to responses for its requests by listening to the master event bus publisher (master_event_pub.ipc).
The salt-master daemon runs on the designated Salt master and performs functions such as authenticating minions, sending, and receiving requests from connected minions and sending and receiving requests and replies to the ‘salt’ CLI.
When a Salt master starts up, a number of processes are started, all of which are called ‘salt-master’ in a process-list but have various role categories.
Among those categories are:
- Publisher
- EventPublisher
- MWorker
The Publisher process is responsible for sending commands over the designated transport to connected minions. The Publisher is bound to the following:
- TCP: port 4505
- IPC: publish_pull.ipc
Each salt minion establishes a connection to the master Publisher.
The EventPublisher publishes master events out to any event listeners. It is bound to the following:
- IPC: master_event_pull.ipc
- IPC: master_event_pub.ipc
Worker processes manage the back-end operations for the Salt Master.
The number of workers is equivalent to the number of ‘worker_threads’ specified in the master configuration and is always at least one.
Workers are bound to the following:
- IPC: workers.ipc
The Salt request server takes requests and distributes them to available MWorker processes for processing. It also receives replies back from minions.
Each salt minion establishes a connection to the master ReqServer.
The Salt master works by always publishing commands to all connected minions and the minions decide if the command is meant for them by checking themselves against the command target.
The typical lifecycle of a salt job from the perspective of the master might be as follows:
The salt-minion is a single process that sits on machines to be managed by Salt. It can either operate as a stand-alone daemon which accepts commands locally via ‘salt-call’ or it can connect back to a master and receive commands remotely.
When starting up, salt minions connect back to a master defined in the minion config file. They connect to two ports on the master:
- TCP: 4505
- This is the connection to the master Publisher. It is on this port that the minion receives jobs from the master.
- TCP: 4506
- This is the connection to the master ReqServer. It is on this port that the minion sends job results back to the master.
Similar to the master, a salt-minion has its own event system that operates over IPC by default. The minion event system operates on a push/pull system with IPC files at minion_event_<unique_id>_pub.ipc and minion_event_<unique_id>_pull.ipc.
The astute reader might ask why have an event bus at all with a single-process daemon. The answer is that the salt-minion may fork other processes as required to do the work without blocking the main salt-minion process and this necessitates a mechanism by which those processes can communicate with each other. Secondarily, this provides a bus by which any user with sufficient permissions can read or write to the bus as a common interface with the salt minion.
When a salt minion starts up, it attempts to connect to the Publisher and the ReqServer on the salt master. It then attempts to authenticate and once the minion has successfully authenticated, it simply listens for jobs.
Jobs normally come either come from the ‘salt-call’ script run by a local user on the salt minion or they can come directly from a master.
The job flow on a minion, coming from the master via a ‘salt’ command is as follows:
1) A master publishes a job that is received by a minion as outlined by the master’s job flow above. 2) The minion is polling its receive socket that’s connected to the master Publisher (TCP 4505 on master). When it detects an incoming message, it picks it up from the socket and decrypts it. 3) A new minion process or thread is created and provided with the contents of the decrypted message. The _thread_return() method is provided with the contents of the received message. 4) The new minion thread is created. The _thread_return() function starts up and actually calls out to the requested function contained in the job. 5) The requested function runs and returns a result. [Still in thread.] 6) The result of the function that’s run is encrypted and returned to the master’s ReqServer (TCP 4506 on master). [Still in thread.] 7) Thread exits. Because the main thread was only blocked for the time that it took to initialize the worker thread, many other requests could have been received and processed during this time.
A common source of confusion is determining when messages are passed in the clear and when they are passed using encryption. There are two rules governing this behaviour:
1) ClearFuncs is used for intra-master communication and during the initial authentication handshake between a minion and master during the key exchange. 2) AESFuncs is used everywhere else.