Skip to main content

Monitoring Medulla services

When monitoring a Medulla On-Premise infrastructure, it is recommended to monitor not only the status of the main Medulla services, but also system resources, network ports, XMPP components, MariaDB databases, and—depending on the architecture—MaxScale and relay servers.

This monitoring can be integrated into your usual monitoring tool to quickly detect the outage, degradation, or malfunction of a Medulla component.

Note:
The elements to be monitored depend on each server’s role. A main server, a relay server, a MariaDB server, or a MaxScale server do not necessarily run the same services.


1. General monitoring principles

For a Medulla infrastructure, we recommend monitoring at least the following elements:

  • server availability;
  • CPU load;
  • memory usage;
  • file system usage;
  • Web interface availability;
  • Medulla service status;
  • connectivity on the main ports in use;
  • status of the ejabberd XMPP service;
  • status and performance of MariaDB when the database is hosted on the server;
  • status of MaxScale when it is in use;
  • storage space usage for packages, data, and backups.

Recommended monitoring frequency

For critical services, a check every 5 minutes is a reasonable baseline.

To avoid triggering alerts for very brief periods of unavailability, you can configure the system to trigger an alert only after several consecutive failures.

For example:

  1. Check: every 5 minutes
  2. Alert: after 3 consecutive failed checks

Recommendation:
Frequencies and thresholds should be tailored to your organization’s availability requirements and architecture.


2. System monitoring

Regardless of the Medulla server’s role, system resources must be monitored.

Element Recommended check
CPU Processor load and utilization
Memory RAM usage and available memory
File systems Used and free space
Operating system Availability and system information
Web interface HTTP/HTTPS check of the Medulla URL

Important:
Special attention must be paid to partitions containing databases, deployment packages, backups, and logs. A full disk can cause multiple simultaneous malfunctions in Medulla.


3. Medulla services to monitor

On a main Medulla server, we recommend monitoring the following services when they are installed:

ejabberd
mmc-agent
pulse-xmpp-agent-relay
pulse2-register-pxe
pulse2-package-server
pulse-xmpp-master-substitute-master
pulse-xmpp-master-substitute-logger
pulse-xmpp-master-substitute-updates
pulse-xmpp-master-substitute-assessor
pulse-xmpp-master-substitute-inventory
pulse-xmpp-master-substitute-deployment
pulse-xmpp-master-substitute-monitoring
pulse-xmpp-master-substitute-registration
pulse-xmpp-master-substitute-subscription

Depending on your version and the installed features, other services may also be present, including:

pulse-xmpp-master-substitute-reconfigurator

Note:
Not all of these services are necessarily present in every installation. The monitoring list should be based on the services that are actually installed and enabled on each server.


4. Check a service’s status

The systemctl command allows you to manually check the status of a service.

For example, to check mmc-agent:

systemctl status mmc-agent

To check only its status:

systemctl is-active mmc-agent

When the service is running normally, the command returns:

active

The following statuses, in particular, should be considered abnormal for an expected service:

inactive
failed

5. Check all Medulla services

To quickly check the main services, you can use the following script:

#!/bin/bash

SERVICES="
ejabberd
mmc-agent
pulse-xmpp-agent-relay
pulse2-register-pxe
pulse2-package-server
pulse-xmpp-master-substitute-master
pulse-xmpp-master-substitute-logger
pulse-xmpp-master-substitute-updates
pulse-xmpp-master-substitute-assessor
pulse-xmpp-master-substitute-inventory
pulse-xmpp-master-substitute-deployment
pulse-xmpp-master-substitute-monitoring
pulse-xmpp-master-substitute-registration
pulse-xmpp-master-substitute-subscription
"

for SERVICE in $SERVICES
do
    if systemctl is-active --quiet "$SERVICE"; then
        echo "[OK] $SERVICE"
    else
        echo "[ERROR] $SERVICE"
    fi
done

Example output:

[OK] ejabberd
[OK] mmc-agent
[OK] pulse-xmpp-agent-relay
[ERROR] pulse2-package-server
[OK] pulse-xmpp-master-substitute-master

Caution:
This script uses a generic list. Remove any services that are not installed on the monitored server to avoid generating false alerts.


6. Monitoring ejabberd

ejabberd is an important component of the Medulla communication architecture. It is recommended to monitor both its system service and several XMPP metrics.

Service status

systemctl status ejabberd

Or:

systemctl is-active ejabberd

Number of connected users

/usr/bin/ejabberdctl stats onlineusers

This value can be used as an operational indicator and tracked over time.

Number of registered users

/usr/bin/ejabberdctl stats registeredusers

A significant or unexpected change in this value can be used as a monitoring signal.

Offline messages

The number of pending messages can also be monitored for the various Medulla technical accounts.

Example:

/usr/bin/ejabberdctl get_offline_count ACCOUNT_NAME pulse

A significant and persistent increase in the number of offline messages may indicate that a Medulla component is no longer processing its messages correctly.

Recommendation:
The values for onlineusers, registeredusers, and offline message queues are particularly useful to log in order to detect abnormal trends, even when no services are down.


7. Monitoring Medulla server ports

In addition to monitoring services, it is recommended to verify that the necessary ports are listening and accessible from the relevant devices.

Depending on the architecture and features in use, the following ports, among others, can be monitored:

Port Usage
5222 XMPP communication
7080 MMC service
8443 Medulla HTTPS service (depending on configuration)
2222 SSH service depending on architecture
22000 Medulla synchronization / dependency, depending on configuration
23000 Medulla synchronization / dependency, depending on configuration
57040 Master agent based on architecture

Important:
This list is generic. The ports actually used depend on your Medulla architecture, the enabled features, and the server’s role. Do not trigger an alert for a port that is not expected on the server in question.


8. Monitoring the web server and Guacamole

The availability of the web interface must be monitored using an HTTP or HTTPS check.

It is recommended to verify:

  • that the web server is running;
  • that the Medulla URL responds;
  • that the returned HTTP status code matches the expected behavior.

Depending on the installation, the status of the web service can, for example, be checked using:

systemctl status httpd

or with the name of the web service used by your installation.

Guacamole

The availability of the Guacamole component can also be checked via its local URL.

Example of a checkpoint:

http://localhost:8081/guacamole/

In a configuration where direct access to this URL is not permitted, an HTTP 403 response may be the expected behavior and confirm that the service is responding.


9. Monitoring Syncthing

When Syncthing is used in the Medulla architecture, monitor its service:

systemctl status syncthing@syncthing

The associated ports can also be monitored depending on the configuration:

22000
23000
8384

Note:
The ports to be monitored must match the Syncthing configuration actually used by your infrastructure.


10. Monitoring a Medulla relay server

A Medulla relay server generally requires less monitoring than the main server.

Depending on the enabled features, we specifically recommend monitoring:

  • pulse-xmpp-agent-relay ;
  • ejabberd, if it is present on the relay;
  • Syncthing, if it is in use;
  • the web server when the relay distributes packages;
  • XMPP connectivity;
  • the disk space used by the deployment repository.

Relay service

systemctl status pulse-xmpp-agent-relay

XMPP port

5222

Syncthing

systemctl status syncthing@syncthing

Disk space

The file system containing the deployment packages must be monitored to ensure that sufficient space remains available.

Important:
If a relay’s repository becomes full, it may prevent the synchronization or distribution of new packages without necessarily causing a service outage.


11. Monitoring MariaDB

When MariaDB is hosted on dedicated infrastructure or on the Medulla server, it is recommended to monitor its service, connections, errors, and resource usage.

Service status

systemctl status mariadb

Or:

systemctl is-active mariadb

Recommended items

Metric Target
MariaDB service Check that the service is running
Connection time Detect a performance degradation
Aborted connections Detect errors or network issues
Connected threads Monitor usage of available connections
Open files Detect when system or MariaDB limits are being approached
Slow queries Detect performance degradation
MariaDB error log Detect server errors
Datadir usage Prevent disk space exhaustion
Backup storage usage Prevent backup storage from filling up

Number of connections

In particular, it is recommended to monitor the number of active connections relative to the maximum number of allowed connections.

An alert can be triggered when usage approaches a significant percentage of the maximum capacity.

Example:
An alert threshold can be set before reaching 100% of max_connections to provide a margin for taking action before saturation occurs.

Slow queries

The number of slow queries can be logged to detect a gradual decline in database performance.

MariaDB errors

It is recommended to monitor the MariaDB error log and trigger an alert when a new significant error appears.


12. Monitoring MaxScale

When the Medulla architecture uses MariaDB MaxScale, its status must also be monitored.

Service status

systemctl status maxscale

Or:

systemctl is-active maxscale

Ports

Depending on the MaxScale configuration used, the following ports, among others, can be monitored:

Port Usage
4006 Read/write connection depending on configuration
4008 Read-only connection, depending on configuration

Note:
MaxScale ports must be adapted to the actual configuration defined in your infrastructure.


13. Storage monitoring

Service availability alone is not sufficient to ensure Medulla’s proper operation. File systems must also be monitored.

Special attention must be paid to:

  • the root partition /;
  • the temporary directory /tmp;
  • the MariaDB datadir;
  • backup storage;
  • the deployment repository;
  • the volumes containing Medulla logs and data.

Important:
Set alert thresholds early enough to allow for intervention before the file system becomes completely full.


14. Integration with a monitoring tool

The various checks described in this documentation can be integrated into most monitoring solutions.

The checks can be grouped into several categories:

Category Examples
System CPU, memory, disk space, OS availability
Services systemctl is-active
Network TCP ports, HTTP/HTTPS availability
XMPP ejabberd, connected users, message queues
Database MariaDB, connections, errors, slow queries
DB proxy MaxScale and associated ports
Storage Datadir, backups, deployment

For critical services, a policy such as the following can be used:

  1. Interval: 5 minutes
  2. Alert: after 3 consecutive failures

Important:
These values are a general recommendation and should be adapted to your service level and monitoring policy.


15. View a service’s logs

When a service is stopped or experiencing an error, start by checking its status:

systemctl status SERVICE_NAME

Then view its logs:

journalctl -u SERVICE_NAME

To view the 100 most recent events:

journalctl -u SERVICE_NAME -n 100

For example:

journalctl -u mmc-agent -n 100

To monitor events in real time:

journalctl -u SERVICE_NAME -f

16. Restart a service

After analyzing the problem, a service can be restarted if necessary using:

systemctl restart SERVICE_NAME

For example:

systemctl restart mmc-agent

Then check its status:

systemctl status mmc-agent

Caution:
Before restarting a service that has failed, check its status and logs to identify the source of the problem. Restarting the service automatically may mask a recurring issue without addressing its root cause.


17. Summary by server role

Role Key checks
Main Medulla server CPU, RAM, disks, HTTP/HTTPS, mmc-agent, XMPP, substitute services, packages, PXE (depending on configuration)
Relay server CPU, RAM, disks, pulse-xmpp-agent-relay, XMPP, Syncthing, repository, HTTP depending on configuration
MariaDB server CPU, RAM, datadir, MariaDB service, connections, errors, slow queries, backups
MaxScale server CPU, RAM, disks, MaxScale service, connection ports

18. Best practices

  • tailor monitoring to each server’s actual role;
  • monitor services as well as system resources and ports;
  • track key metrics over time to detect gradual deviations;
  • avoid triggering an alert for a single isolated failure;
  • pay particular attention to available disk space;
  • monitor ejabberd message queues;
  • monitor MariaDB beyond just the service status;
  • review the logs before restarting any service;
  • regularly test that alerts are functioning properly.

Recommendation:
Effective Medulla monitoring should not be limited to verifying that processes are running. It is recommended to combine monitoring of services, system resources, network ports, XMPP metrics, MariaDB performance, and storage spaces. Then tailor the checks to the role and the features actually installed on each server in your infrastructure.