Firewalld/iptables/nftables question
All First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses. We tried using https://github.com/sleinen/samplicator and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement. https://github.com/sleinen/samplicator/issues/72 Seems at a high rate of logs (which I think we would fall under) there seems to be some issues. We looked at this option - https://zapier.com/engineering/iptables-replication/ Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this. firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1 firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514 firewall-cmd –reload With no luck, packets are not getting sent to 10.137.241.79. When it works with samplicator this is what we get. 11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670 11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697 sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 IP Forwarding is enabled. I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty. Anyone have any suggestions on what we are doing wrong? Patrick
Haven't used either of those. syslog over tcp will ensure delivery but if volume precludes that use I would use nginx for forwarding UDP if you need something quickly and dirty, but fluentd is kind of made for this. It can do parsing and transforms but it's all modularized with a fairly simple DSL, you can just do forwards. There's a gelf for graylog if that's where you're sending but it'll do s3, stdout, http, etc. https://docs.fluentd.org/how-to-guides/graylog2 There's a mess of how-tos for the popular use cases. good hunting, soup On Tue, Feb 6, 2024 at 1:18 PM Patrick McEvilly via WLUG < wlug@lists.wlug.org> wrote:
All
First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses. We tried using https://github.com/sleinen/samplicator and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement.
https://github.com/sleinen/samplicator/issues/72
Seems at a high rate of logs (which I think we would fall under) there seems to be some issues.
We looked at this option -
https://zapier.com/engineering/iptables-replication/
Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this.
firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1
firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514
firewall-cmd –reload
With no luck, packets are not getting sent to 10.137.241.79.
When it works with samplicator this is what we get.
11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670
11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697
sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
IP Forwarding is enabled.
I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty.
Anyone have any suggestions on what we are doing wrong?
Patrick
_______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/M2BHGY...
On 2/6/24 13:11, Patrick McEvilly via WLUG wrote:
All
First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses. We tried using https://github.com/sleinen/samplicator <https://github.com/sleinen/samplicator> and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement.
Due to the nature of UDP, you will probably lose some messages in
What syslog server are you using? In rsyslog at least I'm 99% sure you can send to multiple addresses: *.* @$syslog-receiver:514;RSYSLOG_SyslogProtocol23Format Just duplicating this line should do it. Admittedly, I only have one syslog receiver, but the manpages don't offer any issues with that. The issue it does bring up is with high volumes of logs, where UDP can cause logs to get dropped: transit. If you expect high traffic volume, you can expect to lose a quite noticeable number of messages (the higher the traffic, the more likely and severe is message loss) (from rsyslogd.conf(5)) I'd take a look at that and see if you even need samplicator. --cs
"Patrick" == Patrick McEvilly via WLUG <wlug@lists.wlug.org> writes:
First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses.
Were are the syslog messages coming from? And as people have said, rsyslog is quite fast and should have no trouble pushing packets. Is your box running on two seperate interfaces? How fast are they running? How close to saturation are they? I.e. how busy is your networ?
We tried using https://github.com/sleinen/samplicator and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement.
Are you running in virtual hardware for your RHEL8 box? Looking at samplicator, it's old old software, and might not be tuned for newer versions of linux with sendfile and other system calls to speed things up. You might also have firwall and apparmour and selinux overhead. Try turning them all off.
Seems at a high rate of logs (which I think we would fall under) there seems to be some issues.
What is a high rate?
We looked at this option -
Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this.
firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1
firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514
firewall-cmd –reload
With no luck, packets are not getting sent to 10.137.241.79.
When it works with samplicator this is what we get.
11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670
11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697
sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
IP Forwarding is enabled.
I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty.
Anyone have any suggestions on what we are doing wrong?
Can you back up and give more details on your setup? Is the traffic coming from a host on the same subnet? Or is the traffic being routed somewhere? I've got rsyslog setup at $WORK and I'm doing all kinds of replication of logs and sending to various places without any problems. I've got probably 10gb of logs per-day going through there. So go back to basics: 1. turn off selinux 2. turn off apparmour 3. give us more details on the source and the destination. 4. Can the source simply send logs to both destinations? 5. How is your CPU on your RHEL8 system? What kind of CPU are you running? John
Hi John Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too. Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network. We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero. The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem. Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __ The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices. Thanks Patrick On 2/7/24, 4:08 PM, "John Stoffel" <john@stoffel.org <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly via WLUG <wlug@lists.wlug.org <mailto:wlug@lists.wlug.org>> writes:
First off, I’m in way over my head. $dayjob we have a redhat 8 box. We are looking to take in syslog messages and sent them out to one/two different IP addresses.
Were are the syslog messages coming from? And as people have said, rsyslog is quite fast and should have no trouble pushing packets. Is your box running on two seperate interfaces? How fast are they running? How close to saturation are they? I.e. how busy is your networ?
We tried using https://github.com/sleinen/samplicator <https://github.com/sleinen/samplicator> and while it works perfectly and a one banana job to setup, we seem to be dropping a significant amount of traffic on the box. At least 10% of the logs are missing and we have not loaded up the system yet. We tuned out the network buffers and added 25MB of memory without any improvement.
Are you running in virtual hardware for your RHEL8 box? Looking at samplicator, it's old old software, and might not be tuned for newer versions of linux with sendfile and other system calls to speed things up. You might also have firwall and apparmour and selinux overhead. Try turning them all off.
https://github.com/sleinen/samplicator/issues/72 <https://github.com/sleinen/samplicator/issues/72>
Seems at a high rate of logs (which I think we would fall under) there seems to be some issues.
What is a high rate?
We looked at this option -
https://zapier.com/engineering/iptables-replication/ <https://zapier.com/engineering/iptables-replication/>
Redhat 8 seems to be using firewalld and backended with nfttables so we can’t directly use this method so we tried this.
firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -i ens192 -p udp --dport 514 -j TEE --gateway 127.0.0.1
firewall-cmd --permanent --direct --add-rule ipv4 nat PREROUTING 1 -i ens192 -p udp --dport 514 -j DNAT --to-destination 10.137.241.79:514
firewall-cmd –reload
With no luck, packets are not getting sent to 10.137.241.79.
When it works with samplicator this is what we get.
11:53:04.382233 IP 10.240.136.4.24277 > 10.240.1.1.syslog: SYSLOG local7.notice, length: 670
11:53:04.382408 IP 10.240.136.4.24277 > 10.137.241.79.syslog: SYSLOG local7.notice, length: 697
sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
IP Forwarding is enabled.
I’m not at all familiar with any of this and cutting and pasting from the internet and chatgpt has come up empty.
Anyone have any suggestions on what we are doing wrong?
Can you back up and give more details on your setup? Is the traffic coming from a host on the same subnet? Or is the traffic being routed somewhere? I've got rsyslog setup at $WORK and I'm doing all kinds of replication of logs and sending to various places without any problems. I've got probably 10gb of logs per-day going through there. So go back to basics: 1. turn off selinux 2. turn off apparmour 3. give us more details on the source and the destination. 4. Can the source simply send logs to both destinations? 5. How is your CPU on your RHEL8 system? What kind of CPU are you running? John
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-) You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right. But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance. John
Our logging folks that operate the $$$$$$syslog server said they were unable to parse our firewall logs once we sent them using tcp. ¯\_(ツ)_/¯ As I indicated in my initial email I'm out over my skis a bit. I'll take a look at rsyslog and see if the docs are dumb enough for my level of skills. Ack on the Host OS, Bus Drivers, etc. We will see how it goes once we load it up. P. On 2/7/24, 5:44 PM, "John Stoffel" <john@stoffel.org <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com <mailto:pmcevilly@gmail.com>> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-) You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right. But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance. John
Syslog over TCP is, relatively, new so if they’re running boat anchor tech or haven’t run into it previous it’ll be an issue def. Nginx’ll do you for a good while, not the ideal thing, but what is. If you need something easy deploy multiplat single-bin down the road caddy with the L4 module built in should work a treat but I haven’t tested it in this kind of bodge. Great project though. Take care, soup On Wednesday, February 7, 2024, Patrick McEvilly via WLUG < wlug@lists.wlug.org> wrote:
Our logging folks that operate the $$$$$$syslog server said they were unable to parse our firewall logs once we sent them using tcp. ¯\_(ツ)_/¯
As I indicated in my initial email I'm out over my skis a bit. I'll take a look at rsyslog and see if the docs are dumb enough for my level of skills.
Ack on the Host OS, Bus Drivers, etc. We will see how it goes once we load it up.
P.
On 2/7/24, 5:44 PM, "John Stoffel" <john@stoffel.org <mailto: john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com <mailto: pmcevilly@gmail.com>> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-)
You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right.
But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance.
John
_______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists. wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists. wlug.org/message/ABJGXWV2Z6TKZBGTNEGMOY7UX3LEXPFC/
Question on Syslog location - Will this be deployed on a server / VM with internal or external storage? Wouldnt you have to partition the drive/s if internal storage is used? I may be over thinking this. On Wednesday, February 7, 2024 at 06:05:48 PM EST, Patrick McEvilly via WLUG <wlug@lists.wlug.org> wrote: Our logging folks that operate the $$$$$$syslog server said they were unable to parse our firewall logs once we sent them using tcp. ¯\_(ツ)_/¯ As I indicated in my initial email I'm out over my skis a bit. I'll take a look at rsyslog and see if the docs are dumb enough for my level of skills. Ack on the Host OS, Bus Drivers, etc. We will see how it goes once we load it up. P. On 2/7/24, 5:44 PM, "John Stoffel" <john@stoffel.org <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com <mailto:pmcevilly@gmail.com>> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-) You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right. But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance. John _______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/ABJGXW...
The systems we are building are just forwarders out to a $$$Splunk cloud syslog thing. So disk space is not a huge issue for us. From: William Grenache <grenache7994@verizon.net> Date: Thursday, February 8, 2024 at 9:57 AM To: John Stoffel <john@stoffel.org>, Patrick McEvilly via WLUG <wlug@lists.wlug.org> Cc: Patrick McEvilly <pmcevilly@gmail.com> Subject: Re: [WLUG] Re: Firewalld/iptables/nftables question Question on Syslog location - Will this be deployed on a server / VM with internal or external storage? Wouldnt you have to partition the drive/s if internal storage is used? I may be over thinking this. On Wednesday, February 7, 2024 at 06:05:48 PM EST, Patrick McEvilly via WLUG <wlug@lists.wlug.org> wrote: Our logging folks that operate the $$$$$$syslog server said they were unable to parse our firewall logs once we sent them using tcp. ¯\_(ツ)_/¯ As I indicated in my initial email I'm out over my skis a bit. I'll take a look at rsyslog and see if the docs are dumb enough for my level of skills. Ack on the Host OS, Bus Drivers, etc. We will see how it goes once we load it up. P. On 2/7/24, 5:44 PM, "John Stoffel" <john@stoffel.org<mailto:john@stoffel.org> <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com<mailto:pmcevilly@gmail.com> <mailto:pmcevilly@gmail.com>> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-) You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right. But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance. John _______________________________________________ WLUG mailing list -- wlug@lists.wlug.org<mailto:wlug@lists.wlug.org> To unsubscribe send an email to wlug-leave@lists.wlug.org<mailto:wlug-leave@lists.wlug.org> Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/ABJGXW...
Understood - Thank you. Bill On Thursday, February 8, 2024 at 11:01:43 AM EST, Patrick McEvilly via WLUG <wlug@lists.wlug.org> wrote: The systems we are building are just forwarders out to a $$$Splunk cloud syslog thing. So disk space is not a huge issue for us. From: William Grenache <grenache7994@verizon.net> Date: Thursday, February 8, 2024 at 9:57 AM To: John Stoffel <john@stoffel.org>, Patrick McEvilly via WLUG <wlug@lists.wlug.org> Cc: Patrick McEvilly <pmcevilly@gmail.com> Subject: Re: [WLUG] Re: Firewalld/iptables/nftables question Question on Syslog location - Will this be deployed on a server / VM with internal or external storage? Wouldnt you have to partition the drive/s if internal storage is used? I may be over thinking this. On Wednesday, February 7, 2024 at 06:05:48 PM EST, Patrick McEvilly via WLUG <wlug@lists.wlug.org> wrote: Our logging folks that operate the $$$$$$syslog server said they were unable to parse our firewall logs once we sent them using tcp. ¯\_(ツ)_/¯ As I indicated in my initial email I'm out over my skis a bit. I'll take a look at rsyslog and see if the docs are dumb enough for my level of skills. Ack on the Host OS, Bus Drivers, etc. We will see how it goes once we load it up. P. On 2/7/24, 5:44 PM, "John Stoffel" <john@stoffel.org <mailto:john@stoffel.org>> wrote:
"Patrick" == Patrick McEvilly <pmcevilly@gmail.com <mailto:pmcevilly@gmail.com>> writes:
Based on a suggestion here we setup Nginx and it looks like it is doing what we need but open to other solutions too. We will check out rsyslog too.
rsyslog should be your goto here, especially since with all those devices, you might want to filter them into different lots based on the source IP, etc.
Logs are from about 2500 networking devices including firewall, routers, AAA servers, wireless controllers on a sizeable campus network.
Ok, so are these devices sending only via UDP or can you turn them on to use TCP? TCP will help keep down message loss, but for your busier devices, you might run into problems.
We are testing with one firewall and its generating about 7,000 log messages a second. This is probably on the high side; some devices will be almost zero.
I would try using rsyslog in this case. It's certainly easier to setup than syslog-ng, which was a damn disaster in my book. Super flexible, lots nad lots of options, but just a total pain in the ass to get working right.
The hosts are VMs (I will have two) that we are just getting the configuration going on, we have not loaded it up with all the logs yet so hard to tell CPU usage. Currently it is almost zero. We can throw more cpu/memory at it if it is a problem.
Make sure you have good networking into these syslog destination VMs.
Having the source send two copies of the same logs is possible though now our gear is going to processing more log data than actual data __
Sure, makes sense. Nice to just point everything at 'log.internal.local' and let that system duplicate the data to multiple destinations or backends.
The real issue if we can't trust the folks we are sending logs to to not move or change to another site/location/cloud whoever is on sale this week. Keeping some central log boxes makes it so we only have to change the target on these two syslog hosts and not on all 2500 devices.
I understand completely! I'm happy to answer questions and give you example configs. The docs for rsyslog are ok, but sometimes the examples are a little lacking for real world examples. Obviously written by people who know the stuff inside out, but can't write for smart but ignorant use cases. :-) You could also have rsyslog save a copy locally, and then send in to another host. If you're looking for fully redundant setup, then you might need a load balancer of some sort. It's hard to get complete logging without doing funky things because having a fully redundant setup is nice simple to do right. But also, check your other system configs at the base OS level. And check your VM's bus drivers. Some network cards will have better performance than others. Depending on how you have your virtualization setup, you might need to tweak your VM's hardware setup to get better network performance. John _______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/ABJGXW... _______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/FPPRL3...
participants (5)
-
Cara Salter -
John Stoffel -
Patrick McEvilly -
soup -
William Grenache