logrhythmrulebuilding cheatsheet 6.1

8/14/2012

LogRhythm 6.1

MPE Rule Builder Cheat Sheet

Parsing Fields and Tags The following table provides a list of all the meta-data fields LogRhythm can parse and their associated parsing tag(s). Also provided is the regular expression embedded as part of the tag. This regular expression can be overridden, an explanation of how this done follows the table.

*** All Mapping and Parsing tags are lower case ***

Field Description Tag(s) Default Regex

Origin Host The host from which activity originated (i.e., attacker)

IP Address

(?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])

Host Name

([^\s\.]+\.?)+

IP or Host Name (|)

NAT IP Address

Same as SIP and DIP tags

Impacted Host

The host which was effected by the activity (i.e., target)

IP Address

(?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])\. (?1??\d{1,2}|2[0-4]\d|25[0-5])

Host Name

([^\s\.]+\.?)+

IP or Host Name (|)

NAT IP Address

Same as SIP and DIP tags

Origin Port

The port from which activity originated (i.e., client port).

Source Port \d+

NAT Port \d+

Impacted Port

The port to which activity is targeted (i.e., server port)

Destination Port \d+

NAT Port \d+

Origin MAC Address

The MAC Address which activity originated.

(\w{2}(:|-)?){6}


Copyright 2010 LogRhythm, Inc Page 2 of 11


Impacted MAC Address

The MAC Address which was effected by the activity.

(\w{2}(:|-)?){6}

Origin Interface \w+

Impacted Interface \w+

Protocol

The network protocol used for the activity (i.e., TCP)

use for IANA Protocol Number

1??\d{1,2}|2[0-4]\d|25[0-5]

use for standard protocol abbreviations/names

\w+

Login The user associated with the activity reported in the log.

\w+

Account The user account impacted by activity reported in the log.

\w+

Group The group or role impacted by activity reported in the log.

\w+

Domain The Windows or DNS domain name referenced or impacted by activity reported in the log.

\w+

Object The resource (i.e., file) referenced or impacted by activity reported in the log.

\w+

\w+

URL The URL referenced or impacted by activity reported in the log.

https?://.+




Vendor Message ID The specific vendor log/event identifier for the log.

\w+

Sender The sender of an email or called from number for a VOIP log.

[^\s]+@[^\s]+

Recipient The recipient of an email or called to number for a VOIP log.

[^\s]+@[^\s]+

Subject The subject of an email.

\w+

Session User, system, or application session

\w+

Process System or application process

\w+

\d+

Severity \w+

Version \w+

Command \w+




Bytes In/Out Bytes sent/received from a device, system, or process

Use the appropriate tag based upon the units represented by the log data. , , , , , , , , , , , ,

[0123456789\.]+

Items In/Out Items (ie packets) sent/received from a device, system, or process

Use the following tags if the data represents packet counts , Use the following tags if the data represents anything else. ,

[0123456789\.]+

Duration The duration of a session, job, activity, etc.

If the log contains a specific start and end date/time, use the following tags.

, [0123456789\.]+




[0123456789\.]+

If the log contains a value representing a time, use the following tags to capture the time field elements.

[0123456789\.]+

[0123456789\.]+

[0123456789\.]+

[0123456789\.]+

[0123456789\.]+

[0123456789\.]+

[0123456789\.]+

Size The size of something

[0123456789\.]+

Quantity The quantity of something

[0123456789\.]+

Amount The amount of something

[0123456789\.]+

Rate The rate of something

[0123456789\.]+



Mapping Tags

5 additional tags are available for identifying data in the log specifically for sub-rules. These tags do not parse text into meta-data fields. There sole purpose is to identify portions of the log message that should be used in the development of sub-rules.

Tag Field Type Default Regex

Text .*

Text .*

Text .*

Text .*

Text .*

Overriding the Default Regex

If the default regex for a parsing tag

Will not properly parse the correct data out of the log message, or

Is not the optimal regex from a performance perspective the default should be overridden. To override the default regex, the following syntax should be used. (?[regex]) For example, suppose your regex needs to match file names with a specific extension such as the sample log message below: User joe.blow opened AnnualReport.pdf

If the base-rule was written as: User opened

The value parsed for login would joe and the value for object would be AnnualReport. This is due to the fact that a period is not a word character and the default regex of \w+ would only match up to the period. Instead, the default regexs should be overridden and the base-rule should be: User (?\w+\.?\w*) opened (?\w+\.pdf)

Now, the base-rule will parse anything for login starting with a word character that optionally contains a period followed be additional word characters.

Regular Expression Characters & Practices Following is an overview of regular expression characters and recommend practices.

Match Characters

Notation Characters Matched Example



\d Any digit from 0 to 9 \d\d\d matches 101 but not 10a

\D Any character that is not a numeric digit (0 to 9)

\D\D\D matches abc but not 101

\w Any word character, a-z, A-Z, 0-9 and the underscore character _

\w\w\w matches abc but not ab$

\W Any non-word character \W\W\W matches $#! but not abc

\s Any whitespace character, tab newline, carriage return, form feed and vertical tab.

\s\s\s matches but not abc

\S Matches anything but a space \S\S\S matches a1_ but not

. Any character . matches anything

[ ] Any character between the brackets [abc] matches a or b or c but no other character

[^ ] Matches any character except the characters appearing after the ^ and before the].

[^abc] matches def but not abc

Repetition Characters


{n} Matches n of the previous item \w{4} matches AAAA but not A

{n, } Matches n or more of the previous item \w{4, } matches AAAAAA but not A

{n,m} Matches at least n and at most m of the previous item if n is 0 that makes the character optional ({,9})

A{2,3} matches AA and AAA but not A or AAAA

? Match the previous item 0 or 1 times A? matches A or nothing but not AA

+ Match the previous item 1 or more times A+ matches A, AA, AAA but not nothing

* Match the previous item 0 or more times A* matches nothing, A or any number of As

Positional Characters

Notation Description

^ The following pattern must be at the start of the string, or if its a multi-line string, at the beginning of a line. For multi-line text (string containing a carriage return) the multi-line flag option needs to be set.

$ The preceding pattern must be at the end of the string, or if it is a multi-line string then at the end of a line.

\A The preceding pattern must be at the start of the string; the multi-line flag is ignored

\Z

The preceding pattern must be at the end of the string; the multi-line pattern is ignored

\b The matches a word boundary, essentially the point between a word character (a-z, A-Z, 0-9, _) and a non-word character. The start of a word.

\B This matches a position that is not a word boundary; not the start of a word.

Grouping


()? Matches the pattern inside the brackets 0 or 1 times

(Error)? Matches Error or nothing

()+ Matches the pattern inside the brackets 1 (\w+\s)+ Matches AA AA



or more times

()* Matches the pattern inside the brackets 0 or more times

(\w+\s)* Matches nothing or AA AA

Non-Greedy Qualifier (?)

The non-greedy qualifier is a question mark (?) following a repetition character (?*+). The non-greedy qualifier is used to tell the regex engine that it should stop matching the current match as soon as the next match criterion is met. The non-greedy qualifier is used in combination with a repetition qualifier in order to create a non-greedy match. The non-greedy qualifier improves performance when you want to match any text value up to a specific text value where the specific text value can be uniquely specified within the regex. For example, suppose your regex needs to match the following log: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.blow user account locked out If you use the following regex, incorrect values will be parsed for the login field due to the fact that user occurs twice in the log message. Using this regex will cause account parsed into the login field. ^.*MsgID=1590.*user (?\w+\.?\w*) This is because .* will match everything to the end of the log message. When the regex engine reaches the end of the log message it will begin looking backwards in the log message for the next match. As soon as it finds the last occurrence of user it will match for that portion of the log message. Since the specified regex for login will match account, it will use that match and continue. In order to make the regex take the first occurrence of the next match you use the non-greedy qualifier. The following regex will parse the correct value into the login field because it will stop the previous match (.*) as soon as user is encountered. ^.*?MsgID=1590.*?user (?\w+\.?\w*)



Reserved Characters

The regex engine used by LogRhythm has 12 reserved characters that have special meaning. If any of these characters need to be used as a literal character they will need to be escaped using the backslash (\) character, otherwise known as the escape character. The reserved characters are:

The opening square brackets [

The opening round bracket (

The closing round bracket )

The backslash \

The caret ^

The dollar sign $

The period .

The vertical bar or pipe symbol |

The question mark ?

The asterisk or star *

The plus sign +

The opening squiggly bracket {

The closing squiggly bracket } As a simple example of how to escape reserved characters refer to the following regex which is meant to match any IP address: \d+\.\d+\.\d+\.\d+ As you can see each of the periods of the IP address are escaped meaning the regex engine will look for the actual period (.) character in the string instead of looking for any character, which is the reserved periods special meaning.

Other Special Characters

Other special characters which match special cases that cannot be typed into a regular expression:

Special Character Description

\n Matches newline

\r Matches carriage return

\t Matches tab

\v Matches vertical tab

\f Matches form feed

\nnn Matches ASCII character specified by octal number nnn Ex: \103 matches C

\xnn Matches ASCII character specified by hexadecimal number nn Ex: \x43 matches C

\unnnn Matches the Unicode character specified by the four hexadecimal digits replaced by nnnn

\cV Matches a control character; for example \cV matches Ctrl-V



Regex Recommended Practices

The following are some recommended practices for regex development. All regex examples use the following log. 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.blow user account locked out

Name Recommended Pattern Description

Starting pattern ^.*? This is the best way to start any regex if you want to match any characters until a specific set of characters appear. The ^ tells the regex engine to start from the beginning. The ? tells the engine to perform a non-greedy match. Example: ^.*?MsgID=1590.*?user (?\w+\.?\w*)

Non-greedy Match .*? If you need to match any characters until a specific set of characters appears, use this pattern. Example: ^.*?MsgID=1590.*?user (?\w+\.?\w*)

Overloading Map Tags

(?[regex]) Map Tags should almost always be overloaded. The default regex for map tags is .* which will match everything to the end of the log. Example: ^.*?MsgID=1590.*?user (?\w+\.?\w*)\s(?.*?)$

Preceding and trailing values.

N/A Always match as much constant text as possible. The more information the regex has to evaluate, the faster it will be at identifying non-matching logs. For any parsed field, it is best to have a constant value that is searched for before and after the value being parsed. Example: ^.*?MsgID=1590.*?user (?\w+\.?\w*)\s(?.*?)$

Look Aheads (?=[regex])[regex] (?![regex])[regex]

Positive and negative look ahead allows for an initial check in the regex to see if a case is satisfied in the log messages: Example: (?=^.*?match contains this phrase)^.*?\s\s



Additional Recommended Practices

Rule Names

When the log message the rule will match contains a vendor message ID such as an event ID in Windows Event Logs, it is good to include the ID in the name of the rule. This makes searching for the rule easier and also makes the rule more descriptive of the log it matches.

If the rule matches a log from a logging system that generates logs for a wide variety of services, such as the Windows Application Event Log, the service that generated the log message should be included in the rule name.

All rule names should contain a brief description of the action described by the log. o Ex. EVID 528 : Failed Authentication : Bad Username or Password

Common Event Names

Common Events should be generically named so that they can be re-used for a wide variety of devices. For example, if a common event is being created for a log message that describes a successful connection to an FTP server, the common event should be named so that the FTP server type is irrelevant.

o Good: FTP Connection Succeeded o Bad: Gene6 FTP Connection Succeeded

Common Event names should always have the first letter of each word capitalized. This is to make the viewing of Common Events in analysis tools more consistent.

logrhythmrulebuilding cheatsheet 6.1

Documents