• Steven Mayson

Monitoring IBM AIX Servers with QRadar

Updated: Nov 9, 2020

There are two log source types for IBM AIX Servers: Syslog and Audit. To quickly differentiate the two, Syslog is optional and Audit is mandatory. Developers may opt out of Syslog altogether.

You have two main modes of collection with the AIX Audit subsystem: BIN and STREAM. In the context of QRadar, I have found a few inherent problems that make it unfriendly.

  1. In the case of BIN: Slow conversion rate of the audit.pl script which converts binary data to single line events when faced with huge influx of accumulated activity reaching 2.5 million events in a 20 hour period. These may be caused by application, service ids(human and non-human), or even human users with their own ids. The /etc/security/audit/config design presents a limited ability restricting you to specify user auditing to only a subset of known privileged users or monitor them all. Obvious implications of risk acceptance follow… You are forced to use default for all users with a subset of class objects that you define for the purpose of monitoring with QRadar. As a result of this design inflexibility, it is not practical to manage privileged user ids on-the-fly without more customization.

  2. When the conversion rate cannot keep up, binsize begins to be ignored where audit data is written to either bin1 or bin2 until the entire audit filesystem exhausts all storage capacity.

  3. The DSM Editor allows default event mapping in a canned Log Source or you override it and lose all of the default functionality.

The problem in point 2 is best represented by the following graph:

NOTE: The numbers above in columns A and B are megabyte sizes. The 2d line graph was a time series capture of 19 minutes where each data point is 1 minute in duration.

Alternative Approach

Use auditing in STREAM mode.

The biggest problem I immediately faced: IBM AIX Syslog and IBM AIX Audit log sources do not work well together when they are both using the syslog protocol with the same identifier. In fact, the first log source tends to take dominance and the other never collects. It can flip back and forth. At this point your events do not map correctly because the mapping for the other is in the other log source type!

To workaround this, although very painful and time consuming, you have to create a custom DSM Log Source type. Unfortunately, what this means is you have to manually configure 986 event types to match the functionality of both canned log source types. However, the benefit of this is, if you run into an unknown event type, you’ll be able to identify what it is, create a new custom QID for it with a RegEx mapping and then you may use routing rules in order to drop that event. This helps to conserve your EPS usage greatly overtime as you will have finer control over the types of events being processed by your QRadar appliances.

One such recent example has been the bane of my existence: the usage of Centrify PAM. When AIX is upgraded, Centrify having downlevel components causes millions and millions of events that cap QRadar’s eps limit by a single server until that specific event is dropped or that downlevel component gets upgraded so it stops flooding QRadar. And there were cases where more than one had the same down level components. After QRadar's queue becomes full, it is completely blind in a very short amount of time.

Customizing Log Sources for unknown events above, like Centrify, give you a clearer picture of what's really happening from an auditing perspective. AIX accepts the public key and then Centrify confirms authentication.

TIP: Ensure to deploy full configuration after saving your custom DSM. It was observed that the latest DSM Editor has some bugginess that may cause events to be tagged improperly.

UPDATE: The condition above where events were found to not process as expected was found to be resolved by manipulating what is called parsing order located in the sensordevice table. IBM AIX Audit and IBM AIX Syslog will both work when using the syslog protocol.