What you're seeing
One destination goes down and calls routed to it ring out for about 30 seconds, then the caller gets 408 or 503. Meanwhile the dead node still shows as active:
SET: {
ID: 1
TARGETS: {
DEST: {
URI: sip:10.0.0.11:5060
FLAGS: AP A = active, P = probing
PRIORITY: 0
}
DEST: {
URI: sip:10.0.0.12:5060
FLAGS: AP
PRIORITY: 0
}
}
}Likely causes
Most common first.
- No failure_route calling ds_next_dst(). Nothing happens after a failure unless t_on_failure() arms a failure route that calls ds_next_dst() and relays again. Without it the first failure goes straight back to the caller.
- Failover support isn't enabled. Dispatcher only keeps the remaining destinations for ds_next_dst() when failover support is on (flags parameter, value 2). A ds_select_dst() limit of 1 has the same effect: there is nothing left to fail over to.
- The timeout is too long. A dead host sends no reply, so the transaction waits for tm's fr_timer, 30 seconds by default, before the failure route runs. Most callers hang up first. Set a short fr_timer per branch with t_set_fr().
- Probing isn't enabled. Without OPTIONS probing (ds_ping_interval and ds_probing_mode), a dead node stays active and every new call pays the timeout again before failing over.
- The failure route ignores the reply codes you get. A failure route that only checks 5xx won't fail over on a timeout, and one that fails over on everything will retry 486 Busy and 404. Fail over on 5xx and on a branch timeout with no reply.
How to fix
1. Enable failover and probing
loadmodule "dispatcher.so"
modparam("dispatcher", "list_file", "/etc/kamailio/dispatcher.list")
modparam("dispatcher", "flags", 2)
modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_probing_mode", 1)
modparam("dispatcher", "ds_probing_threshold", 2)
modparam("dispatcher", "ds_inactive_threshold", 2)
modparam("dispatcher", "ds_ping_from", "sip:[email protected]")
modparam("dispatcher", "ds_ping_reply_codes", "class=2;code=403;code=404;code=405")# setid destination flags priority
1 sip:10.0.0.11:5060 0 0
1 sip:10.0.0.12:5060 0 0ds_ping_reply_codes counts those replies as "alive". Many PBXs and carriers answer OPTIONS with 403 or 404 rather than 200.
2. Select, arm the failure route, shorten the timer
route[DISPATCH] {
# 4 = round-robin; failover list is kept because flags=2
if (!ds_select_dst("1", "4")) {
send_reply("503", "No destination available");
exit;
}
# fr_inv_timer 120s once ringing, fr_timer 2s to get any reply
t_set_fr(120000, 2000);
t_on_failure("DISPATCH_FAILURE");
route(RELAY);
exit;
}
failure_route[DISPATCH_FAILURE] {
if (t_is_canceled()) exit;
if (t_check_status("5[0-9][0-9]") || (t_branch_timeout() && !t_branch_replied())) {
# no reply at all: take it out until probing brings it back.
# a 5xx only moves this call on (a 503 may just mean "busy right now").
if (t_branch_timeout() && !t_branch_replied()) {
ds_mark_dst("ip");
}
if (ds_next_dst()) {
xlog("L_WARN", "DISPATCH: failing over $ci to $du\n");
t_set_fr(120000, 2000);
t_on_failure("DISPATCH_FAILURE");
route(RELAY);
exit;
}
}
}2 seconds suits destinations on your own network. Raise fr_timer for paths with real latency: a trunk that takes 3 seconds to send 100 Trying would otherwise fail over when it shouldn't.
3. Test it before the next outage
# take a destination out and watch calls move
kamcmd dispatcher.set_state ip 1 sip:10.0.0.11:5060
kamcmd dispatcher.list
# after changing dispatcher.list
kamcmd dispatcher.reloadAdd an event_route[dispatcher:dst-down] with an xlog() so a destination going down reaches your alerting, not just the call logs.