Monitoring Medulla services
When monitoring a Medulla On-Premise infrastructure, it is recommended to monitor not only the status of the main Medulla services, but also system resources, network ports, XMPP components, MariaDB databases, and—depending on the architecture—MaxScale and relay servers.
This monitoring can be integrated into your usual monitoring tool to quickly detect the outage, degradation, or malfunction of a Medulla component.
Note:
The elements to be monitored depend on each server’s role. A main server, a relay server, a MariaDB server, or a MaxScale server do not necessarily run the same services.
1. General monitoring principles
For a Medulla infrastructure, we recommend monitoring at least the following elements:
- server availability;
- CPU load;
- memory usage;
- file system usage;
- Web interface availability;
- Medulla service status;
- connectivity on the main ports in use;
- status of the ejabberd XMPP service;
- status and performance of MariaDB when the database is hosted on the server;
- status of MaxScale when it is in use;
- storage space usage for packages, data, and backups.
Recommended monitoring frequency
For critical services, a check every 5 minutes is a reasonable baseline.
To avoid triggering alerts for very brief periods of unavailability, you can configure the system to trigger an alert only after several consecutive failures.
For example:
- Check: every 5 minutes
- Alert: after 3 consecutive failed checks
Recommendation:
Frequencies and thresholds should be tailored to your organization’s availability requirements and architecture.
2. System monitoring
Regardless of the Medulla server’s role, system resources must be monitored.
| Element | Recommended check |
|---|---|
| CPU | Processor load and utilization |
| Memory | RAM usage and available memory |
| File systems | Used and free space |
| Operating system | Availability and system information |
| Web interface | HTTP/HTTPS check of the Medulla URL |
Important:
Special attention must be paid to partitions containing databases, deployment packages, backups, and logs. A full disk can cause multiple simultaneous malfunctions in Medulla.
3. Medulla services to monitor
On a main Medulla server, we recommend monitoring the following services when they are installed:
ejabberd
mmc-agent
pulse-xmpp-agent-relay
pulse2-register-pxe
pulse2-package-server
pulse-xmpp-master-substitute-master
pulse-xmpp-master-substitute-logger
pulse-xmpp-master-substitute-updates
pulse-xmpp-master-substitute-assessor
pulse-xmpp-master-substitute-inventory
pulse-xmpp-master-substitute-deployment
pulse-xmpp-master-substitute-monitoring
pulse-xmpp-master-substitute-registration
pulse-xmpp-master-substitute-subscription
Depending on your version and the installed features, other services may also be present, including:
pulse-xmpp-master-substitute-reconfigurator
Note:
Not all of these services are necessarily present in every installation. The monitoring list should be based on the services that are actually installed and enabled on each server.
4. Check a service’s status
The systemctl command allows you to manually check the status of a service.
For example, to check mmc-agent:
systemctl status mmc-agent
To check only its status:
systemctl is-active mmc-agent
When the service is running normally, the command returns:
active
The following statuses, in particular, should be considered abnormal for an expected service:
inactive
failed
5. Check all Medulla services
To quickly check the main services, you can use the following script:
#!/bin/bash
SERVICES="
ejabberd
mmc-agent
pulse-xmpp-agent-relay
pulse2-register-pxe
pulse2-package-server
pulse-xmpp-master-substitute-master
pulse-xmpp-master-substitute-logger
pulse-xmpp-master-substitute-updates
pulse-xmpp-master-substitute-assessor
pulse-xmpp-master-substitute-inventory
pulse-xmpp-master-substitute-deployment
pulse-xmpp-master-substitute-monitoring
pulse-xmpp-master-substitute-registration
pulse-xmpp-master-substitute-subscription
"
for SERVICE in $SERVICES
do
if systemctl is-active --quiet "$SERVICE"; then
echo "[OK] $SERVICE"
else
echo "[ERROR] $SERVICE"
fi
done
Example output:
[OK] ejabberd
[OK] mmc-agent
[OK] pulse-xmpp-agent-relay
[ERROR] pulse2-package-server
[OK] pulse-xmpp-master-substitute-master
Caution:
This script uses a generic list. Remove any services that are not installed on the monitored server to avoid generating false alerts.
6. Monitoring ejabberd
ejabberd is an important component of the Medulla communication architecture. It is recommended to monitor both its system service and several XMPP metrics.
Service status
systemctl status ejabberd
Or:
systemctl is-active ejabberd
Number of connected users
/usr/bin/ejabberdctl stats onlineusers
This value can be used as an operational indicator and tracked over time.
Number of registered users
/usr/bin/ejabberdctl stats registeredusers
A significant or unexpected change in this value can be used as a monitoring signal.
Offline messages
The number of pending messages can also be monitored for the various Medulla technical accounts.
Example:
/usr/bin/ejabberdctl get_offline_count ACCOUNT_NAME pulse
A significant and persistent increase in the number of offline messages may indicate that a Medulla component is no longer processing its messages correctly.
Recommendation:
The values for onlineusers, registeredusers, and offline message queues are particularly useful to log in order to detect abnormal trends, even when no services are down.
7. Monitoring Medulla server ports
In addition to monitoring services, it is recommended to verify that the necessary ports are listening and accessible from the relevant devices.
Depending on the architecture and features in use, the following ports, among others, can be monitored:
| Port | Usage |
|---|---|
5222 |
XMPP communication |
7080 |
MMC service |
8443 |
Medulla HTTPS service (depending on configuration) |
2222 |
SSH service depending on architecture |
22000 |
Medulla synchronization / dependency, depending on configuration |
23000 |
Medulla synchronization / dependency, depending on configuration |
57040 |
Master agent based on architecture |
Important:
This list is generic. The ports actually used depend on your Medulla architecture, the enabled features, and the server’s role. Do not trigger an alert for a port that is not expected on the server in question.
8. Monitoring the web server and Guacamole
The availability of the web interface must be monitored using an HTTP or HTTPS check.
It is recommended to verify:
- that the web server is running;
- that the Medulla URL responds;
- that the returned HTTP status code matches the expected behavior.
Depending on the installation, the status of the web service can, for example, be checked using:
systemctl status httpd
or with the name of the web service used by your installation.
Guacamole
The availability of the Guacamole component can also be checked via its local URL.
Example of a checkpoint:
http://localhost:8081/guacamole/
In a configuration where direct access to this URL is not permitted, an HTTP 403 response may be the expected behavior and confirm that the service is responding.
9. Monitoring Syncthing
When Syncthing is used in the Medulla architecture, monitor its service:
systemctl status syncthing@syncthing
The associated ports can also be monitored depending on the configuration:
22000
23000
8384
Note:
The ports to be monitored must match the Syncthing configuration actually used by your infrastructure.
10. Monitoring a Medulla relay server
A Medulla relay server generally requires less monitoring than the main server.
Depending on the enabled features, we specifically recommend monitoring:
pulse-xmpp-agent-relay;ejabberd, if it is present on the relay;- Syncthing, if it is in use;
- the web server when the relay distributes packages;
- XMPP connectivity;
- the disk space used by the deployment repository.
Relay service
systemctl status pulse-xmpp-agent-relay
XMPP port
5222
Syncthing
systemctl status syncthing@syncthing
Disk space
The file system containing the deployment packages must be monitored to ensure that sufficient space remains available.
Important:
If a relay’s repository becomes full, it may prevent the synchronization or distribution of new packages without necessarily causing a service outage.
11. Monitoring MariaDB
When MariaDB is hosted on dedicated infrastructure or on the Medulla server, it is recommended to monitor its service, connections, errors, and resource usage.
Service status
systemctl status mariadb
Or:
systemctl is-active mariadb
Recommended items
| Metric | Target |
|---|---|
| MariaDB service | Check that the service is running |
| Connection time | Detect a performance degradation |
| Aborted connections | Detect errors or network issues |
| Connected threads | Monitor usage of available connections |
| Open files | Detect when system or MariaDB limits are being approached |
| Slow queries | Detect performance degradation |
| MariaDB error log | Detect server errors |
| Datadir usage | Prevent disk space exhaustion |
| Backup storage usage | Prevent backup storage from filling up |
Number of connections
In particular, it is recommended to monitor the number of active connections relative to the maximum number of allowed connections.
An alert can be triggered when usage approaches a significant percentage of the maximum capacity.
Example:
An alert threshold can be set before reaching 100% of max_connections to provide a margin for taking action before saturation occurs.
Slow queries
The number of slow queries can be logged to detect a gradual decline in database performance.
MariaDB errors
It is recommended to monitor the MariaDB error log and trigger an alert when a new significant error appears.
12. Monitoring MaxScale
When the Medulla architecture uses MariaDB MaxScale, its status must also be monitored.
Service status
systemctl status maxscale
Or:
systemctl is-active maxscale
Ports
Depending on the MaxScale configuration used, the following ports, among others, can be monitored:
| Port | Usage |
|---|---|
4006 |
Read/write connection depending on configuration |
4008 |
Read-only connection, depending on configuration |
Note:
MaxScale ports must be adapted to the actual configuration defined in your infrastructure.
13. Storage monitoring
Service availability alone is not sufficient to ensure Medulla’s proper operation. File systems must also be monitored.
Special attention must be paid to:
- the root partition
/; - the temporary directory
/tmp; - the MariaDB datadir;
- backup storage;
- the deployment repository;
- the volumes containing Medulla logs and data.
Important:
Set alert thresholds early enough to allow for intervention before the file system becomes completely full.
14. Integration with a monitoring tool
The various checks described in this documentation can be integrated into most monitoring solutions.
The checks can be grouped into several categories:
| Category | Examples |
|---|---|
| System | CPU, memory, disk space, OS availability |
| Services | systemctl is-active |
| Network | TCP ports, HTTP/HTTPS availability |
| XMPP | ejabberd, connected users, message queues |
| Database | MariaDB, connections, errors, slow queries |
| DB proxy | MaxScale and associated ports |
| Storage | Datadir, backups, deployment |
For critical services, a policy such as the following can be used:
- Interval: 5 minutes
- Alert: after 3 consecutive failures
Important:
These values are a general recommendation and should be adapted to your service level and monitoring policy.
15. View a service’s logs
When a service is stopped or experiencing an error, start by checking its status:
systemctl status SERVICE_NAME
Then view its logs:
journalctl -u SERVICE_NAME
To view the 100 most recent events:
journalctl -u SERVICE_NAME -n 100
For example:
journalctl -u mmc-agent -n 100
To monitor events in real time:
journalctl -u SERVICE_NAME -f
16. Restart a service
After analyzing the problem, a service can be restarted if necessary using:
systemctl restart SERVICE_NAME
For example:
systemctl restart mmc-agent
Then check its status:
systemctl status mmc-agent
Caution:
Before restarting a service that has failed, check its status and logs to identify the source of the problem. Restarting the service automatically may mask a recurring issue without addressing its root cause.
17. Summary by server role
| Role | Key checks |
|---|---|
| Main Medulla server | CPU, RAM, disks, HTTP/HTTPS, mmc-agent, XMPP, substitute services, packages, PXE (depending on configuration) |
| Relay server | CPU, RAM, disks, pulse-xmpp-agent-relay, XMPP, Syncthing, repository, HTTP depending on configuration |
| MariaDB server | CPU, RAM, datadir, MariaDB service, connections, errors, slow queries, backups |
| MaxScale server | CPU, RAM, disks, MaxScale service, connection ports |
18. Best practices
- tailor monitoring to each server’s actual role;
- monitor services as well as system resources and ports;
- track key metrics over time to detect gradual deviations;
- avoid triggering an alert for a single isolated failure;
- pay particular attention to available disk space;
- monitor ejabberd message queues;
- monitor MariaDB beyond just the service status;
- review the logs before restarting any service;
- regularly test that alerts are functioning properly.
Recommendation:
Effective Medulla monitoring should not be limited to verifying that processes are running. It is recommended to combine monitoring of services, system resources, network ports, XMPP metrics, MariaDB performance, and storage spaces. Then tailor the checks to the role and the features actually installed on each server in your infrastructure.