OCTI notifications and digests: a new powerful engine for the knowledge graph

For more and more organizations, OpenCTI is (or starts to be) the nervous system of cyber intelligence knowledge (for both threats and operational data). Multiple CTI feeds, XDR / EDR, SIEMs, logs management systems pieces of data are fully integrated, consolidated and enriched in the platform by all the OpenCTI capabilities (through connectors or built-in engines such as the inference rule manager).

Thus, the volume of information becomes extremely large and difficult to follow as a human being. At some point, each user is only interested by specific parts of this knowledge and would like to be informed as soon as possible when some kinds of data are created in the platform:

• New vulnerability exploited in a given sector.
• New victim in a given country.
• New reports about a given threat actor.
• Update of a subscribed incident response case.
• New bunch of signatures for a given piece of malware.
• New sightings or log entries for a given organization.

Based on this long-awaited need from our community, we decided to completely rewrite the already existing “subscription system” to give more flexibility and power to OpenCTI notifications and digests, relying on our streaming system.

---

A bit of semantic

To be clear about what we’ve designed, let’s start with a semantic definition of the introduced concepts in this new notification system:

• Live Trigger (aka Trigger): A live set of filters that will produce one or multiple notifications.
• Regular Digest (aka Digest): A set of live triggers on a time period (last #hours) that will produce one or multiple notifications
• Notification: The result of a trigger/digest which produces an outcome: user interface entry / bell, emails, etc. Webhooks and other connectors for outcome will be available very soon.

How it works?

All the users in the platform can now define its own live triggers or regular digests to listen for a subset of knowledge in the platform. This concerns data directly created by connectors and/or users but also inferred / auto-generated information: everything is completely accessible.

To illustrate that let’s explain different use cases.

As an analyst, I want to be notified of all reports containing the region “Europe”. But in general, connectors and other analysts only add the concerned countries in the reports and not the region itself. Also, I would like to receive the notifications both in the user interface and by email.

To solve this use case you can simply define a trigger this way:

With this configuration, all reports in the platform that will be linked to Europe (directly or through inference) will generate a notification both in the user interface and by email.

Of course it is possible to adapt the configuration to listen also for update and deletion, pick up the appropriate notification system, etc. As you can see, it’s quite simple to configure a live trigger and start to be notified about pieces of data which matter the most.

Then, let’s move to the next use case.

As an analyst, I want to receive a daily digest of all reports containing the region “Europe”. But in general, connectors and other analysts only add the concerned countries in the reports and not the region itself. Also, I would like to receive this digest only by email at 9AM.

This analyst also would like to receive a daily digest for the trigger he has just created. That can be achieve by defining a regular digest configuration like this:

As you can see, a digest is “just” a combination of live triggers. Then, the user is free to define the period and the list of triggers which makes sense for the wished use case. In this situation, the user will receive this email:

Just to complete the information about those use cases:

• It’s possible to define live triggers without any outcomes to be used only in regular digests and not receive any live notifications.
• A user can also configure a regular digest to be displayed as a user interface notification.

Here is an example of a regular digest directly displayed in the UI:

Of course all the lines are clickable to let the user quickly access to the concerned entities / sightings / relationships.

We hope those examples helped you to understand how we designed the notifications feature and how you can use it by combining filters to create powerful live notifications and reuse them to schedule your regular digests.

How is it technically designed?

Let’s talk a bit about the technical aspect of this notification system. It’s designed around 2 new managers, the “notification manager” and the “publisher manager”.

Each manager is responsible for a specific part of the system.

The notification manager

This manager is responsible for listening the internal platform data stream and applying all triggers defined by users in real time. When a “match” occurs, a new notification will be pushed into another internal platform stream (aka the notification stream). We use this mechanism to split the concerns between the notification generation and the notification distribution.

Some important details about this processing

• In the case of a live trigger, the stream will be filtered and a message will be generated/pushed in the notification stream
• In the case of a regular digest, the manager will periodically consume the notification stream to combine interesting events into a digest notification that will also be pushed in the notification stream.

The publisher manager

This manager is responsible for the distribution of the notifications through multiple outcomes. For now, OpenCTI supports user interface and email outcomes out-of-the-box (templates are hardcoded). But we plan to release very soon built-in webhooks and new connectors able to consume the notification stream (both live and digests) to extend integrations. Also, the templating will be open for configuration.

These 2 managers are started by default but can obviously be managed through static parameters. If the SMTP settings are not correctly configured, the platform now starts normally but each time OpenCTI will try to send an email, it will result in a new error log entry.

Conclusion

We hope that the behavior and the underlying mechanisms of the new notification system is now clear after reading this article. As written above, this is just the first step in a global work about notifications in the platform and multiple enhancements are already scheduled:

• Introduce webhooks and specific connectors as outcomes.
• Be able to customize templates for emails and webhook (even if template already take into account customized theme).
• Have “Subscribe” buttons everywhere to quickly create live triggers to be updated of any activity in specific entities such as incidents, case, etc.

As always, we remain very interested about your feedback and ideas to improve this feature. Don’t hesitate to create some feature requests in our GitHub repositories or join the community slack channel!