Diagnose a Medulla relay server
Medulla relay servers act as intermediaries between the main Medulla server and client computers.
In particular, they are involved in distributing commands and packages to the computers associated with them.
When a relay encounters a problem, symptoms may include:
- computers that can no longer receive deployments;
- packages that are unavailable on certain computers;
ABORT RELAY DOWNerrors;ABORT ALTERNATIVE RELAYS DOWNerrors;ABORT INFO RELAY MISSINGerrors;- computers associated with a relay that no longer exists;
- a package synchronization issue.
Note:
A Medulla relay server includes, among other components, an XMPP server, a relay agent, and a Guacamole server. The diagnostic process should therefore distinguish between issues related to server accessibility, relay registration, computer assignment, or package distribution.
1. Identify the symptom
Start by determining whether the problem involves:
| Symptom | Priority diagnosis |
|---|---|
ABORT RELAY DOWN |
Verify that the relay exists and is accessible |
ABORT ALTERNATIVE RELAYS DOWN |
Check for available alternative relays |
ABORT INFO RELAY MISSING |
Check the relay’s assignment to the computer |
| Only one computer is not working | Check its groupdeploy field and its assignment rule |
| All computers on a relay are affected | Check the relay itself first |
| A package is missing from a relay | Check the package distribution and synchronization |
2. Verify that the relay server is accessible
In the event of an error:
ABORT RELAY DOWN
the operations guide indicates that the relay may be turned off or no longer exist.
Start by testing its network accessibility:
ping RELAY_IP
Then, if SSH access is permitted:
ssh root@RELAY_IP
Note:
A lack of response to a ping alone does not necessarily mean the server is down, as some environments may filter ICMP. The SSH test helps complete the diagnosis.
If the relay is inaccessible, check:
- that it is powered on;
- that its IP address is correct;
- that routing between the main server and the relay is working;
- that the necessary flows are not being blocked.
3. Check the relay’s last known status
If the relay is not responding, check its last known status recorded in Medulla:
SELECT *
FROM uptime_machine
WHERE hostname = "RELAY_NAME"
ORDER BY id DESC
LIMIT 1\G;
The status field indicates its last known status:
1: considered online;0: considered offline.
The updowntime field shows how long the relay has been in that state.
4. Verify that the relay is still registered and enabled
To view the relays recognized by Medulla:
SELECT id, nameserver, enabled
FROM relayserver;
Check in particular:
- that the relay in question appears in the list;
- that its name matches the expected server;
- the value in its
enabledfield.
When:
enabled = 1
the operations guide indicates that this relay can be provided to computers by the assessor substitute.
Caution:
A relay that no longer exists but remains enabled in the configuration may continue to be offered to computers and cause deployment errors.
5. Case of a relay that no longer exists
If you have confirmed that a relay has been permanently deleted or is no longer in use, the operations guide provides the following command to disable it:
UPDATE relayserver
SET enabled = 0
WHERE id = <relay_id>;
It also instructs you to check the corresponding entry in the machines table:
SELECT id, enabled
FROM machines
WHERE hostname = 'RELAY_NAME';
Then, if the server has indeed been removed:
UPDATE machines
SET enabled = 0
WHERE id = <id>;
Caution:
These commands directly modify the Medulla database. They should only be used once it has been confirmed that the relay has been permanently removed, after a backup of the database.
6. Diagnose ABORT RELAY DOWN
The status:
ABORT RELAY DOWN
indicates that Medulla cannot use the relay designated for the deployment.
Perform the following checks:
pingthe relay;- test SSH access to it;
- check its latest status in
uptime_machine; - check if it is listed in
relayserver; - check the value of
enabled; - if the relay has been removed, verify that it is no longer offered to computers.
7. Diagnose ABORT ALTERNATIVE RELAYS DOWN
The status:
ABORT ALTERNATIVE RELAYS DOWN
means that none of the usable alternative relays are available.
In this case, the operations guide recommends checking the relays.
For each relay that could be used:
- check its network accessibility;
- check its last known status;
- verify that it is still enabled in
relayserver.
Note:
If multiple relays become unavailable simultaneously, also investigate a common network or infrastructure issue rather than an individual failure of each relay.
8. Diagnose ABORT INFO RELAY MISSING
The status:
ABORT INFO RELAY MISSING
indicates that information regarding the relay associated with the computer is missing.
The operations guide lists the following as a possible cause:
- a relay that was deleted after being assigned to the computer;
- a computer whose
groupdeployfield is empty.
The field:
groupdeploy
in the machines table contains the JID of the relay associated with the computer.
Therefore, verify that the computer does indeed have a value for the groupdeploy field.
For example:
SELECT id, hostname, groupdeploy
FROM machines
WHERE hostname = 'MACHINE_NAME';
If groupdeploy is empty, the computer is not associated with any relay.
9. Verify that the relay specified in groupdeploy exists
A value in groupdeploy is not sufficient on its own: it must correspond to a relay that still exists.
Display the available relays:
SELECT id, nameserver, jid, enabled
FROM relayserver;
Then compare the relay’s JID with the computer’s groupdeploy value.
Caution:
If groupdeploy references an old relay that has been deleted, Medulla may return ABORT INFO RELAY MISSING.
10. Fix a computer with no associated relay
After identifying the correct relay, the operations guide provides the following example:
UPDATE machines
SET groupdeploy = 'agent@medulla-interne/mainrelay'
WHERE id = <machine_id>;
The value used must match the JID of the appropriate relay in the relayserver table.
Caution:
This query directly modifies a computer’s assignment. It must not be executed without first identifying why the automatic assignment failed. Otherwise, the problem may recur during a future reconfiguration.
11. Verify the relay assignment rule
Medulla can assign computers to a relay based on network rules.
The operations guide provides an example for testing whether a subnet matches a regular expression defined in the assignment rules:
SELECT '10.10.19.0/24'
REGEXP '^10\.62\.19\.0\/24$';
In this example:
10.10.19.0/24corresponds to the computer’s subnet;^10\.62\.19\.0\/24$matches the rule to be tested.
The result determines whether or not the subnet matches the rule.
Tip:
If multiple computers on the same network are assigned to the wrong relay, first check for an error in the assignment rule rather than looking for individual issues on each computer.
12. Check the computer’s network information
To understand why a computer is or is not assigned a particular relay, you can view its network information as recorded by Medulla:
SELECT
m.hostname,
m.ippublic,
m.ip_xmpp,
m.macaddress AS machine_mac,
n.ipaddress AS network_ip,
n.broadcast,
n.gateway,
n.mask,
n.mac AS network_mac
FROM xmppmaster.machines AS m
INNER JOIN xmppmaster.network AS n
ON n.machines_id = m.id
WHERE m.hostname = 'MACHINE_NAME';
In particular, verify that the network reported by the computer matches the network expected by the relay assignment rule.
13. Identify relays with no associated computers
To list relays that have no associated clients, the operations guide provides the following query:
SELECT nameserver
FROM relayserver
WHERE relayserver.jid NOT IN (
SELECT groupdeploy
FROM machines
WHERE agenttype = 'machine'
);
This query identifies relays that are currently registered but are not being used by any computer.
Note:
A relay without a client is not necessarily experiencing an error. It may have been installed in anticipation of a new site or be used as an alternative relay. This query should therefore be interpreted within the context of the architecture.
14. Check the availability of packages on the relay
When a relay is accessible but a package cannot be deployed, verify that the package is available on the server used for its distribution.
An error such as:
Transfer error: Package Server does not have this package
indicates that the package server being used does not have the requested package.
In this situation, the problem does not necessarily mean that the relay itself is offline.
It is important to distinguish between:
| Error | Interpretation |
|---|---|
ABORT RELAY DOWN |
The relay is unavailable |
ERROR TRANSFER FAILED |
The server being used does not have the package |
ABORT TRANSFER FAILED |
The package transfer failed |
15. Check Syncthing if you encounter synchronization issues
The Medulla operations guide describes a case in which Syncthing displays an error related to the UDP buffer size:
failed to sufficiently increase receive buffer size
(was: 208 kiB, wanted: 2048 kiB, got: 416 kiB)
In this specific case, it recommends increasing the maximum read and write buffer sizes:
sysctl -w net.core.rmem_max=2500000
sysctl -w net.core.wmem_max=2500000
Caution:
These commands address a specific Syncthing issue, identifiable by the error message above. They should not be applied systematically to every synchronization problem.
16. Determine whether the problem is caused by the relay or the computer
Before taking action on the relay, determine the scope of the problem.
| Findings | Diagnostic guidance |
|---|---|
| Only one computer is affected | Check the computer and its groupdeploy |
| Multiple computers on the same relay are affected | Check the relay |
| All computers on a subnet are incorrectly assigned | Check the assignment rule |
| The computers are communicating, but some packages are missing | Check the availability and synchronization of the packages |
| The relay is inaccessible | Check the server and network flows |
17. Quick troubleshooting procedure
When a Medulla relay server appears to be experiencing a problem, perform the following checks in this order:
- identify the affected computers and functions;
- note the exact deployment status, if there is one;
- test the relay’s accessibility using
pingand SSH; - check its latest status in
uptime_machine; - check its presence and whether it is
enabledinrelayserver; - check the
groupdeployfield for the affected computers; - verify that the JID in
groupdeploycorresponds to an existing relay; - if the assignment appears incorrect, check the assignment rule;
- if only the packages are affected, check their availability and synchronization;
- check the Syncthing logs if a synchronization error is suspected.
18. Quick diagnostic table
| Symptom | Possible cause | First check |
|---|---|---|
ABORT RELAY DOWN |
Relay is off or has been removed | ping, SSH, and uptime_machine |
ABORT ALTERNATIVE RELAYS DOWN |
No alternative relay available | Check the various relays |
ABORT INFO RELAY MISSING |
Relay missing or empty groupdeploy |
Check groupdeploy |
| Relay known but no longer in use | Old entry still enabled | Check relayserver.enabled |
| Computer assigned to the wrong relay | Incorrect assignment rule | Test the subnet with the REGEXP rule |
| Relay with no computers | Unused or backup relay | Query on groupdeploy |
| Package missing on the relay | Distribution or synchronization issue | Check package availability |
| Syncthing buffer error | Insufficient UDP buffer | Search the logs for the exact message |
19. Information to collect if the problem persists
Before continuing with troubleshooting or opening a support ticket, gather the following:
- the relay’s hostname;
- its IP address;
- the result of the reachability test;
- its latest status in
uptime_machine; - its entry in
relayserver; - the value of
enabled; - the hostname of an affected computer;
- the
groupdeployvalue for that computer; - the exact deployment status;
- the full audit message;
- if the issue involves packages, the affected package and relay;
- any Syncthing errors observed.
Recommendation:
To diagnose a Medulla relay, start by distinguishing between three levels: is the relay accessible?, is the computer correctly associated with this relay? and does the relay have the resources needed for deployment? This approach allows you to quickly distinguish between a relay server outage and an assignment problem or a package synchronization problem.