Data Model Overview

Agents provide two primary kinds of normalized output: events and alerts. In the context of this guide an event is a completely neutral transaction that by itself is neither malicious nor benign.

An alert on the other hand is not neutral. Instead, alerts are typically generated by making inferences against acquired traffic, and often highlight activity that seems suspicious or even malicious.

Both events and alerts share several key fields, which make navigation between the two straightforward when it comes time to visualize or search the data.

The Elastic Common Schema

Out-of-the-box DynamiteNSM will normalize all events (and alerts) to the Elastic Common Schema. Elastic Common Schema or ECS for short is an open source specification, developed with the support of the Elastic User Community.

Mapping of Network Events

Network events capture the details of one device communicating with another. The initiator is referred to as the source, and the recipient as the destination. Depending on the data source, a network event can contain details of addresses, protocols, headers, and device roles.

When an event contains details about the sending and receiving hosts, the baseline for capturing these values will be the source and destination fields.

Source Fields

Example

{
  "source": {
    "ip": "192.168.86.222",
    "port": 54162
  }
}

Descriptions

Field	Description	Level
source.address	Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to .ip or .domain, depending on which one it is. type: keyword	extended
source.bytes	Bytes sent from the source to the destination. type: long example: 184	core
source.domain	Source domain. type: keyword	core
source.ip	IP address of the source (IPv4 or IPv6). type: ip	core
source.mac	MAC address of the source. type: keyword	core
source.nat.ip	Translated ip of source based NAT sessions (e.g. internal client to internet) Typically connections traversing load balancers, firewalls, or routers. type: ip	extended
source.nat.port	Translated port of source based NAT sessions. (e.g. internal client to internet) Typically used with load balancers, firewalls, or routers. type: long	extended
source.packets	Packets sent from the source to the destination. type: long example: 12	core
source.port	Port of the source. type: long	core
source.registered_domain	The highest registered source domain, stripped of the subdomain. For example, the registered domain for "foo.example.com" is "example.com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk". type: keyword example: example.com	extended
source.subdomain	The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain. For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period. type: keyword example: east	extended
source.top_level_domain	The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk". type: keyword example: co.uk	extended

Destination Fields

Example

{
  "destination": {
    "ip": "192.168.86.1",
    "port": 53
  }
}

Descriptions

Field	Description	Level
destination.address	Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to .ip or .domain, depending on which one it is. type: keyword	extended
destination.bytes	Bytes sent from the destination to the source. type: long example: 184	core
destination.domain	Destination domain. type: keyword	core
destination.ip	IP address of the destination (IPv4 or IPv6). type: ip	core
destination.mac	MAC address of the destination. type: keyword	core
destination.nat.ip	Translated ip of destination based NAT sessions (e.g. internet to private DMZ) Typically used with load balancers, firewalls, or routers. type: ip	extended
destination.nat.port	Port the source session is translated to by NAT Device. Typically used with load balancers, firewalls, or routers. type: long	extended
destination.packets	Packets sent from the destination to the source. type: long example: 12	core
destination.port	Port of the destination. type: long	core
destination.registered_domain	The highest registered destination domain, stripped of the subdomain. For example, the registered domain for "foo.example.com" is "example.com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk". type: keyword example: example.com	extended
destination.subdomain	The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain. For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period. type: keyword example: east	extended
destination.top_level_domain	The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk". type: keyword example: co.uk	extended

Client/Server Fields

In the examples above an analyst may notice that this event is likely a DNS request. The values mapped under source and destination are copied and mapped under client and server.

Example

{
  "client": {
    "ip": "192.168.86.222",
    "port": 64734
  },
  "server": {
    "ip": "192.168.86.1",
    "port": 53
  }
}

It’s important to note that while the values for the source and destination fields may reverse between events in a single network transaction, the values for client and server typically will not.

Event Categorization Fields

When considering the event categorization fields, the category and type fields are populated using their respective allowed values which best classify the source network event.

Suricata Alert, Event Section

{
  "event": {
    "severity": 3,
    "created": "2021-01-15T03:34:20.671Z",
    "kind": "alert",
    "module": "suricata",
    "start": "2021-01-02T02:49:08.152Z",
    "category": [
      "network",
      "intrusion_detection"
    ],
    "type": [
      "allowed"
    ],
    "dataset": "suricata.eve"
  }
}

Zeek HTTP Event Section

{
  "event": {
    "kind": "event",
    "created": "2021-01-18T19:58:34.280666497Z",
    "module": "zeek",
    "action": "post",
    "id": "CMcIc31sqwZSUKQP5j",
    "category": [
      "network",
      "web"
    ],
    "type": [
      "connection",
      "info",
      "protocol"
    ],
    "dataset": "zeek.http",
    "outcome": "success"
  }
}

Most event.category/event.type ECS pairings are complete on their own. However, the pairing of event.category:network and event.type:protocol is an exception. When these two fields/value pairs both used to categorize an event, the network.protocol field should also be populated

Example

{
  "network": {
    "protocol": "http",
    "type": "ipv4",
    "transport": "tcp"
  }
}