Skip to main content

Diagnose a Medulla relay server

Medulla relay servers act as intermediaries between the main Medulla server and client computers.

In particular, they are involved in distributing commands and packages to the computers associated with them.

When a relay encounters a problem, symptoms may include:

  • computers that can no longer receive deployments;
  • packages that are unavailable on certain computers;
  • ABORT RELAY DOWN errors;
  • ABORT ALTERNATIVE RELAYS DOWN errors;
  • ABORT INFO RELAY MISSING errors;
  • computers associated with a relay that no longer exists;
  • a package synchronization issue.

Note:
A Medulla relay server includes, among other components, an XMPP server, a relay agent, and a Guacamole server. The diagnostic process should therefore distinguish between issues related to server accessibility, relay registration, computer assignment, or package distribution.


1. Identify the symptom

Start by determining whether the problem involves:

Symptom Priority diagnosis
ABORT RELAY DOWN Verify that the relay exists and is accessible
ABORT ALTERNATIVE RELAYS DOWN Check for available alternative relays
ABORT INFO RELAY MISSING Check the relay’s assignment to the computer
Only one computer is not working Check its groupdeploy field and its assignment rule
All computers on a relay are affected Check the relay itself first
A package is missing from a relay Check the package distribution and synchronization

2. Verify that the relay server is accessible

In the event of an error:

ABORT RELAY DOWN

the operations guide indicates that the relay may be turned off or no longer exist.

Start by testing its network accessibility:

ping RELAY_IP

Then, if SSH access is permitted:

ssh root@RELAY_IP

Note:
A lack of response to a ping alone does not necessarily mean the server is down, as some environments may filter ICMP. The SSH test helps complete the diagnosis.

If the relay is inaccessible, check:

  • that it is powered on;
  • that its IP address is correct;
  • that routing between the main server and the relay is working;
  • that the necessary flows are not being blocked.

3. Check the relay’s last known status

If the relay is not responding, check its last known status recorded in Medulla:

SELECT *
FROM uptime_machine
WHERE hostname = "RELAY_NAME"
ORDER BY id DESC
LIMIT 1\G;

The status field indicates its last known status:

  • 1: considered online;
  • 0: considered offline.

The updowntime field shows how long the relay has been in that state.


4. Verify that the relay is still registered and enabled

To view the relays recognized by Medulla:

SELECT id, nameserver, enabled
FROM relayserver;

Check in particular:

  • that the relay in question appears in the list;
  • that its name matches the expected server;
  • the value in its enabled field.

When:

enabled = 1

the operations guide indicates that this relay can be provided to computers by the assessor substitute.

Caution:
A relay that no longer exists but remains enabled in the configuration may continue to be offered to computers and cause deployment errors.


5. Case of a relay that no longer exists

If you have confirmed that a relay has been permanently deleted or is no longer in use, the operations guide provides the following command to disable it:

UPDATE relayserver
SET enabled = 0
WHERE id = <relay_id>;

It also instructs you to check the corresponding entry in the machines table:

SELECT id, enabled
FROM machines
WHERE hostname = 'RELAY_NAME';

Then, if the server has indeed been removed:

UPDATE machines
SET enabled = 0
WHERE id = <id>;

Caution:
These commands directly modify the Medulla database. They should only be used once it has been confirmed that the relay has been permanently removed, after a backup of the database.


6. Diagnose ABORT RELAY DOWN

The status:

ABORT RELAY DOWN

indicates that Medulla cannot use the relay designated for the deployment.

Perform the following checks:

  1. ping the relay;
  2. test SSH access to it;
  3. check its latest status in uptime_machine;
  4. check if it is listed in relayserver;
  5. check the value of enabled;
  6. if the relay has been removed, verify that it is no longer offered to computers.

7. Diagnose ABORT ALTERNATIVE RELAYS DOWN

The status:

ABORT ALTERNATIVE RELAYS DOWN

means that none of the usable alternative relays are available.

In this case, the operations guide recommends checking the relays.

For each relay that could be used:

  • check its network accessibility;
  • check its last known status;
  • verify that it is still enabled in relayserver.

Note:
If multiple relays become unavailable simultaneously, also investigate a common network or infrastructure issue rather than an individual failure of each relay.


8. Diagnose ABORT INFO RELAY MISSING

The status:

ABORT INFO RELAY MISSING

indicates that information regarding the relay associated with the computer is missing.

The operations guide lists the following as a possible cause:

  • a relay that was deleted after being assigned to the computer;
  • a computer whose groupdeploy field is empty.

The field:

groupdeploy

in the machines table contains the JID of the relay associated with the computer.

Therefore, verify that the computer does indeed have a value for the groupdeploy field.

For example:

SELECT id, hostname, groupdeploy
FROM machines
WHERE hostname = 'MACHINE_NAME';

If groupdeploy is empty, the computer is not associated with any relay.


9. Verify that the relay specified in groupdeploy exists

A value in groupdeploy is not sufficient on its own: it must correspond to a relay that still exists.

Display the available relays:

SELECT id, nameserver, jid, enabled
FROM relayserver;

Then compare the relay’s JID with the computer’s groupdeploy value.

Caution:
If groupdeploy references an old relay that has been deleted, Medulla may return ABORT INFO RELAY MISSING.


10. Fix a computer with no associated relay

After identifying the correct relay, the operations guide provides the following example:

UPDATE machines
SET groupdeploy = 'agent@medulla-interne/mainrelay'
WHERE id = <machine_id>;

The value used must match the JID of the appropriate relay in the relayserver table.

Caution:
This query directly modifies a computer’s assignment. It must not be executed without first identifying why the automatic assignment failed. Otherwise, the problem may recur during a future reconfiguration.


11. Verify the relay assignment rule

Medulla can assign computers to a relay based on network rules.

The operations guide provides an example for testing whether a subnet matches a regular expression defined in the assignment rules:

SELECT '10.10.19.0/24'
REGEXP '^10\.62\.19\.0\/24$';

In this example:

  • 10.10.19.0/24 corresponds to the computer’s subnet;
  • ^10\.62\.19\.0\/24$ matches the rule to be tested.

The result determines whether or not the subnet matches the rule.

Tip:
If multiple computers on the same network are assigned to the wrong relay, first check for an error in the assignment rule rather than looking for individual issues on each computer.


12. Check the computer’s network information

To understand why a computer is or is not assigned a particular relay, you can view its network information as recorded by Medulla:

SELECT
    m.hostname,
    m.ippublic,
    m.ip_xmpp,
    m.macaddress AS machine_mac,
    n.ipaddress AS network_ip,
    n.broadcast,
    n.gateway,
    n.mask,
    n.mac AS network_mac
FROM xmppmaster.machines AS m
INNER JOIN xmppmaster.network AS n
    ON n.machines_id = m.id
WHERE m.hostname = 'MACHINE_NAME';

In particular, verify that the network reported by the computer matches the network expected by the relay assignment rule.


13. Identify relays with no associated computers

To list relays that have no associated clients, the operations guide provides the following query:

SELECT nameserver
FROM relayserver
WHERE relayserver.jid NOT IN (
    SELECT groupdeploy
    FROM machines
    WHERE agenttype = 'machine'
);

This query identifies relays that are currently registered but are not being used by any computer.

Note:
A relay without a client is not necessarily experiencing an error. It may have been installed in anticipation of a new site or be used as an alternative relay. This query should therefore be interpreted within the context of the architecture.


14. Check the availability of packages on the relay

When a relay is accessible but a package cannot be deployed, verify that the package is available on the server used for its distribution.

An error such as:

Transfer error: Package Server does not have this package

indicates that the package server being used does not have the requested package.

In this situation, the problem does not necessarily mean that the relay itself is offline.

It is important to distinguish between:

Error Interpretation
ABORT RELAY DOWN The relay is unavailable
ERROR TRANSFER FAILED The server being used does not have the package
ABORT TRANSFER FAILED The package transfer failed

15. Check Syncthing if you encounter synchronization issues

The Medulla operations guide describes a case in which Syncthing displays an error related to the UDP buffer size:

failed to sufficiently increase receive buffer size
(was: 208 kiB, wanted: 2048 kiB, got: 416 kiB)

In this specific case, it recommends increasing the maximum read and write buffer sizes:

sysctl -w net.core.rmem_max=2500000
sysctl -w net.core.wmem_max=2500000

Caution:
These commands address a specific Syncthing issue, identifiable by the error message above. They should not be applied systematically to every synchronization problem.


16. Determine whether the problem is caused by the relay or the computer

Before taking action on the relay, determine the scope of the problem.

Findings Diagnostic guidance
Only one computer is affected Check the computer and its groupdeploy
Multiple computers on the same relay are affected Check the relay
All computers on a subnet are incorrectly assigned Check the assignment rule
The computers are communicating, but some packages are missing Check the availability and synchronization of the packages
The relay is inaccessible Check the server and network flows

17. Quick troubleshooting procedure

When a Medulla relay server appears to be experiencing a problem, perform the following checks in this order:

  1. identify the affected computers and functions;
  2. note the exact deployment status, if there is one;
  3. test the relay’s accessibility using ping and SSH;
  4. check its latest status in uptime_machine;
  5. check its presence and whether it is enabled in relayserver;
  6. check the groupdeploy field for the affected computers;
  7. verify that the JID in groupdeploy corresponds to an existing relay;
  8. if the assignment appears incorrect, check the assignment rule;
  9. if only the packages are affected, check their availability and synchronization;
  10. check the Syncthing logs if a synchronization error is suspected.

18. Quick diagnostic table

Symptom Possible cause First check
ABORT RELAY DOWN Relay is off or has been removed ping, SSH, and uptime_machine
ABORT ALTERNATIVE RELAYS DOWN No alternative relay available Check the various relays
ABORT INFO RELAY MISSING Relay missing or empty groupdeploy Check groupdeploy
Relay known but no longer in use Old entry still enabled Check relayserver.enabled
Computer assigned to the wrong relay Incorrect assignment rule Test the subnet with the REGEXP rule
Relay with no computers Unused or backup relay Query on groupdeploy
Package missing on the relay Distribution or synchronization issue Check package availability
Syncthing buffer error Insufficient UDP buffer Search the logs for the exact message

19. Information to collect if the problem persists

Before continuing with troubleshooting or opening a support ticket, gather the following:

  • the relay’s hostname;
  • its IP address;
  • the result of the reachability test;
  • its latest status in uptime_machine;
  • its entry in relayserver;
  • the value of enabled;
  • the hostname of an affected computer;
  • the groupdeploy value for that computer;
  • the exact deployment status;
  • the full audit message;
  • if the issue involves packages, the affected package and relay;
  • any Syncthing errors observed.

Recommendation:
To diagnose a Medulla relay, start by distinguishing between three levels: is the relay accessible?, is the computer correctly associated with this relay? and does the relay have the resources needed for deployment? This approach allows you to quickly distinguish between a relay server outage and an assignment problem or a package synchronization problem.