Problem: message from Zabbix agent (or to be more specific something which emulates zabbix agent) is not sometimes accepted by the server.
I finally figured out why some messages from our Perl based home grown client were lost. It took me some serious investigation, because when ever I did HexDump of message in my code, it was in correct format, but it was leaving machine in corrupted state.
Let me show you what I mean. Zabbix communicate by sending data in json, prefixed with some header as you can read here. To be more specific, there is ZBXD string, followed by 0x01 character (\1) and then 8B of length of message encoded in little indian. For instance message long 4574 bytes would look like this:
00000000 5A 42 58 44 01 DE 11 00 - 00 00 00 00 00 7B 22 72 ZBXD.........{"r
00000010 65 71 75 65 73 74 22 3A - 22 73 65 6E 64 65 72 20 equest":"sender
00000020 64 61 74 61 22 2C 22 64 - 61 74 61 22 3A 5B 7B 22 data","data":[{"
If you run in terminal
printf %d 0x11DE
you get 4574.
Problem is that when I did strace, I noticed that same message were modified to following data actually send to server:
write(8, "ZBXD\1\303\236\21\0\0\0\0\0\0{\"request\":\"sender"..., 4588) = 4588
You see? There is 9 bytes, not 8!
I spend quite lot of time trying to see what is going on in POE libraries which are below this perl daemon and at the end I more or less figured what is wrong with this my code:
sub _encode_data {
### Encode data to Zabbix JSON protocol format
my ($stub) = @_;
my ($json, $message);
$json = encode_json($stub);
&::trace("$ALIAS encoded metrics to the following format:\n" . JSON::XS->new->pretty->encode(JSON::XS->new->decode($json))); # DEBUG
$message = pack("A5VVA*", "ZBXD\1",length($json),0,$json);
&::trace("$ALIAS message length: " . length($json) . " which translates to message: \n" . HexDump($message)); # DEBUG
return $message;
}
Any guess? HexDump returns data correctly. Still, if you comment debug output „encoded metrics to the following format“, problem disappear. Reason for that is, that $json is passed as a reference and perl optimizer probably kicks in. It do encoding of that ref AFTER HexDump and everything else, but before it is written to the socket buffer (to kernel). What helps here is to do copy of $json variable and pass that one to debug output which do JSON decode and encode (for pretty output).