February 2024 - WLUG - wlug.mailman3.com

Re: Firewalld/iptables/nftables question

by John Stoffel

>>>>> "Patrick" == Patrick McEvilly via WLUG <wlug(a)lists.wlug.org> writes: > First off, I’m in way over my head. $dayjob we have a redhat 8 box. > We are looking to take in syslog messages and sent them out to > one/two different IP addresses. Were are the syslog messages coming from? And as people have said, rsyslog is quite fast and should have no trouble pushing packets. Is your box running on two seperate interfaces? How fast are they running? How close to saturation are they? I.e. how busy is your networ? > We tried using > https://github.com/sleinen/samplicator and while it works perfectly > and a one banana job to setup, we seem to be dropping a significant > amount of traffic on the box. At least 10% of the logs are missing > and we have not loaded up the system yet. We tuned out the network > buffers and added 25MB of memory without any improvement. Are you running in virtual hardware for your RHEL8 box? Looking at samplicator, it's old old software, and might not be tuned for newer versions of linux with sendfile and other system calls to speed things up. You might also have firwall and apparmour and selinux overhead. Try turning them all off. > https://github.com/sleinen/samplicator/issues/72 > Seems at a high rate of logs (which I think we would fall under) > there seems to be some issues. What is a high rate? > We looked at this option - > https://zapier.com/engineering/iptables-replication/ > Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this > method so we tried this. > firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 > -j TEE --gateway 127.0.0.1 > firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j > DNAT --to-destination 10.137.241.79:514 > firewall-cmd –reload > With no luck, packets are not getting sent to 10.137.241.79. > When it works with samplicator this is what we get. > 11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670 > 11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697 > sysctl net.ipv4.ip_forward > net.ipv4.ip_forward = 1 > IP Forwarding is enabled. > I’m not at all familiar with any of this and cutting and pasting > from the internet and chatgpt has come up empty. > Anyone have any suggestions on what we are doing wrong? Can you back up and give more details on your setup? Is the traffic coming from a host on the same subnet? Or is the traffic being routed somewhere? I've got rsyslog setup at $WORK and I'm doing all kinds of replication of logs and sending to various places without any problems. I've got probably 10gb of logs per-day going through there. So go back to basics: 1. turn off selinux 2. turn off apparmour 3. give us more details on the source and the destination. 4. Can the source simply send logs to both destinations? 5. How is your CPU on your RHEL8 system? What kind of CPU are you running? John

Feb. 7, 2024

3 million smart toothbrushes were just used in a DDoS attack. Really

by Tim Keller

My favorite part of the story.. "While the details are scarce, we know that the compromised toothbrushes were running Java." We are officially living in the dumbest time line. My only hope is that the servers at the far end of that DDOS attack at least had a minty smell to them.. Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

Feb. 7, 2024

Re: Firewalld/iptables/nftables question

by Cara Salter

On 2/6/24 13:11, Patrick McEvilly via WLUG wrote: > All > > First off, I’m in way over my head. $dayjob we have a redhat 8 box. We > are looking to take in syslog messages and sent them out to one/two > different IP addresses. We tried using > https://github.com/sleinen/samplicator > <https://github.com/sleinen/samplicator> and while it works perfectly > and a one banana job to setup, we seem to be dropping a significant > amount of traffic on the box. At least 10% of the logs are missing and > we have not loaded up the system yet. We tuned out the network buffers > and added 25MB of memory without any improvement. What syslog server are you using? In rsyslog at least I'm 99% sure you can send to multiple addresses: *.* @$syslog-receiver:514;RSYSLOG_SyslogProtocol23Format Just duplicating this line should do it. Admittedly, I only have one syslog receiver, but the manpages don't offer any issues with that. The issue it does bring up is with high volumes of logs, where UDP can cause logs to get dropped: > Due to the nature of UDP, you will probably lose some messages in transit. If you expect high traffic volume, you can expect to lose a quite noticeable number of messages (the higher the traffic, the more likely and severe is message loss) (from rsyslogd.conf(5)) I'd take a look at that and see if you even need samplicator. --cs

Feb. 6, 2024

Re: Firewalld/iptables/nftables question

by soup

Haven't used either of those. syslog over tcp will ensure delivery but if volume precludes that use I would use nginx for forwarding UDP if you need something quickly and dirty, but fluentd is kind of made for this. It can do parsing and transforms but it's all modularized with a fairly simple DSL, you can just do forwards. There's a gelf for graylog if that's where you're sending but it'll do s3, stdout, http, etc. https://docs.fluentd.org/how-to-guides/graylog2 There's a mess of how-tos for the popular use cases. good hunting, soup On Tue, Feb 6, 2024 at 1:18 PM Patrick McEvilly via WLUG < wlug(a)lists.wlug.org> wrote: > All > > > > First off, I’m in way over my head. $dayjob we have a redhat 8 box. We > are looking to take in syslog messages and sent them out to one/two > different IP addresses. We tried using > https://github.com/sleinen/samplicator and while it works perfectly and a > one banana job to setup, we seem to be dropping a significant amount of > traffic on the box. At least 10% of the logs are missing and we have not > loaded up the system yet. We tuned out the network buffers and added 25MB > of memory without any improvement. > > > > https://github.com/sleinen/samplicator/issues/72 > > > > Seems at a high rate of logs (which I think we would fall under) there > seems to be some issues. > > > > We looked at this option - > > https://zapier.com/engineering/iptables-replication/ > > > > Redhat 8 seems to be using firewalld and backended with nfttables so we > can’t directly use this method so we tried this. > > > > > > firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i > ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1 > > firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i > ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514 > > firewall-cmd –reload > > > > With no luck, packets are not getting sent to 10.137.241.79. > > > > When it works with samplicator this is what we get. > > > > 11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG > local7.notice, length: 670 > > 11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG > local7.notice, length: 697 > > > > sysctl net.ipv4.ip_forward > > net.ipv4.ip_forward = 1 > > > > IP Forwarding is enabled. > > > > I’m not at all familiar with any of this and cutting and pasting from the > internet and chatgpt has come up empty. > > > > Anyone have any suggestions on what we are doing wrong? > > > > Patrick > > > > > > > > > > > > > > > _______________________________________________ > WLUG mailing list -- wlug(a)lists.wlug.org > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > Create Account: https://wlug.mailman3.com/accounts/signup/ > Change Settings: > https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > Web Forum/Archive: > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/M2BHG… >

Feb. 6, 2024

Firewalld/iptables/nftables question

by Patrick McEvilly

All First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses. We tried using https://github.com/sleinen/samplicator and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement. https://github.com/sleinen/samplicator/issues/72 Seems at a high rate of logs (which I think we would fall under) there seems to be some issues. We looked at this option - https://zapier.com/engineering/iptables-replication/ Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this. firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1 firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514 firewall-cmd –reload With no luck, packets are not getting sent to 10.137.241.79. When it works with samplicator this is what we get. 11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670 11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697 sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 IP Forwarding is enabled. I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty. Anyone have any suggestions on what we are doing wrong? Patrick

Feb. 6, 2024

Re: Hey Everybody! WLUG Meeting this week!

by Tim Keller

Currently we're having issues getting the Turing Pi 2 to boot, so doing a show off of Proxmox I could definitely do as well. I've got a three node setup as well as a backup node running at work, but I don't have anything on it that I couldn't show the world. Tim. On Tue, Feb 6, 2024 at 10:00 AM John Stoffel <john(a)stoffel.org> wrote: > >>>>> "Tim" == Tim Keller via WLUG <wlug(a)lists.wlug.org> writes: > > > We've got a WLUG meeting on the 8th! I know weirdly early in the > > month because the 1st landed on a Thursday! > > > If things go well, I *might* have a Pi cluster using compute cards > > in it! I'll keep you all posted! > > That would be awesome to see. I've finally got ProxMox 8.1.3 up and > running at work. It's an interesting beast with some funky > limitations. And alot of flexibility. But the setup can be tough if > you have VLANs on your trunks/bonds feeding the servers. Lots of by > hand networking setup. > > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

Feb. 6, 2024

Re: Hey Everybody! WLUG Meeting this week!

by John Stoffel

>>>>> "Tim" == Tim Keller via WLUG <wlug(a)lists.wlug.org> writes: > We've got a WLUG meeting on the 8th! I know weirdly early in the > month because the 1st landed on a Thursday! > If things go well, I *might* have a Pi cluster using compute cards > in it! I'll keep you all posted! That would be awesome to see. I've finally got ProxMox 8.1.3 up and running at work. It's an interesting beast with some funky limitations. And alot of flexibility. But the setup can be tough if you have VLANs on your trunks/bonds feeding the servers. Lots of by hand networking setup.

Feb. 6, 2024

Hey Everybody! WLUG Meeting this week!

by Tim Keller

Hey Everybody! We've got a WLUG meeting on the 8th! I know weirdly early in the month because the 1st landed on a Thursday! If things go well, I *might* have a Pi cluster using compute cards in it! I'll keep you all posted! As for the meeting details: Time: 7pm Date: 2/8/2024 Location: Techncopia Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

Feb. 5, 2024