Hi John Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too. Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network. We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero. The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem. Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __ The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices. Thanks Patrick On 2/7/24, 4:08 PM, "John Stoffel" <john@stoffel.org <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly via WLUG <wlug@lists.wlug.org <mailto:wlug@lists.wlug.org>> writes:
First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses.
Were are the syslog messages coming from? And as people have said, rsyslog is quite fast and should have no trouble pushing packets. Is your box running on two seperate interfaces? How fast are they running? How close to saturation are they? I.e. how busy is your networ?
We tried using https://github.com/sleinen/samplicator <https://github.com/sleinen/samplicator> and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement.
Are you running in virtual hardware for your RHEL8 box? Looking at samplicator, it's old old software, and might not be tuned for newer versions of linux with sendfile and other system calls to speed things up. You might also have firwall and apparmour and selinux overhead. Try turning them all off.
https://github.com/sleinen/samplicator/issues/72 <https://github.com/sleinen/samplicator/issues/72>
Seems at a high rate of logs (which I think we would fall under) there seems to be some issues.
What is a high rate?
We looked at this option -
https://zapier.com/engineering/iptables-replication/ <https://zapier.com/engineering/iptables-replication/>
Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this.
firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1
firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514
firewall-cmd –reload
With no luck, packets are not getting sent to 10.137.241.79.
When it works with samplicator this is what we get.
11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670
11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697
sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
IP Forwarding is enabled.
I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty.
Anyone have any suggestions on what we are doing wrong?
Can you back up and give more details on your setup? Is the traffic coming from a host on the same subnet? Or is the traffic being routed somewhere? I've got rsyslog setup at $WORK and I'm doing all kinds of replication of logs and sending to various places without any problems. I've got probably 10gb of logs per-day going through there. So go back to basics: 1. turn off selinux 2. turn off apparmour 3. give us more details on the source and the destination. 4. Can the source simply send logs to both destinations? 5. How is your CPU on your RHEL8 system? What kind of CPU are you running? John