OpenCTI data sharing capabilities upgrade

In a previous article, we shared first insights about the multiple data sharing capabilities of the OpenCTI platform. Now is the time to refresh information and provide our users with new statement and explanations about these features.

One of the most important expectations of our community is the ability to disseminate knowledge and make OpenCTI data available to third party systems, platforms or even other OpenCTI instances. In order to fulfill this role, we have developed many channels and features to extract information while keeping in mind that key factors of success are:

• The variety of output formats and standards.
• The needs to provide both push and pull capabilities.
• Real time versus regular updates.

The figure below represents the current landscape of OpenCTI sharing capabilities.

What’s new in OpenCTI 5.3.x?

Menu reorganization

As data sharing features are increasing, we refactored a bit the menu to group these capabilities under the Data sharing menu, in the Data section.

Main goal here was to ease the work of administrators to manage the different ways of sharing the knowledge.

CSV Feeds

Long-awaited feature which was requested several times by the community, OpenCTI is now able to publish data in CSV feeds on a rolling period. To better understand why we have implemented this capability, let’s talk about a simple use case:

I would like to easily ingest all IP addresses and domain names collected by OCTI in my Fortinet firewall blocklist. This firewall is not able to parse STIX, TAXII or any complex format but only CSV.

Based on this use case it could be easy to reply:

Please create a connector, use the GraphQL API and do it yourself 😆

But if one of the main goals of OpenCTI is to disseminate information, it should not be an extra effort of the community to build this kind of integration. So we decided to develop a new data sharing feature to expose the data as CSV feeds with all the power of OCTI.

Let’s configure the described use case to have a better idea of the approach. To cover this problematic, we need to configure a new feed directly in the OCTI user interface, in the menu Data > Data sharing. As we need data from 3 different types of entity (IPv4, IPv6 and domain names) we will adjust the value mapping for each types.

As you can see in the platform screenshot we start by selecting a CSV separator (;), a rolling time to only have the latest creation at each query (here 1 hour), disabling the CSV headers as Fortinet doesn’t support it, and the list of entity types we need. It’s also possible to apply extra filters if needed.

As a reminder, available data in all OCTI sharing endpoints are always segregated with the marking definitions granted to the group of the authenticated user when querying the data. So information in CSV feeds will be filtered depending on the user connecting to the feed.

Then we have to decide the name of the columns we would like to expose and to map the column to an attribute for each type.

Here, we picked up the attribute “value” for each type and mapped it to the column “indicator”, but it could be different depending of your own use case. Of course, you can add more attributes in the CSV feed if needed.

After the creation of the feed, you are able to access the CSV feed URL by clicking on the created line.

And information will be directly displayed in your browser.

And that’s it! You can now feed your firewall with relevant data and format.

Raw & Live streams upgrade

More powerful

Starting from version 5.3.x, we’ve reworked the “live stream” to finally give to the consumers a complete filtered view of the events that happened in the platform. Previously, it was just able to filter creations and deletions only on some circumstances. Basically, it was like a “TAXII collection on steroids” but not at our level of expectations yet.

This was due to lack of information like nested entities (labels, external references, etc.) or relationship sources and targets in the raw stream. It contained only some “diff” and not the “fully resolved” element.

The new live stream implementation is able to publish all event types.

Creation, deletion and merge event types
These types of event haven’t changed, they still contain information in STIX format given at the creation of the element or just prior its deletion. Based on the new implementation, deletions are available in all possible conditions.

Update event type
This type of event has been modified as it now contains many more data. In this context, a custom “STIX patch format” has been developed to represent what has changed in an object. This patch follows and respects the standard used in JSON Patches as specified in RFC 6902.

Example updating the description of a Threat-Actor:

{
  "context": {
    "patch": [
      {
        "op": "replace",
        "path": "/description",
        "value": "This actor represents all Russian state-sponsored offensive activities in the cyberspace. This is a test."
      },
      {
        "op": "replace",
        "path": "/modified",
        "value": "2022-09-11T08:06:31.671Z"
      },
      {
        "op": "replace",
        "path": "/extensions/extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba/updated_at",
        "value": "2022-09-11T08:06:31.671Z"
      }
    ],
    "reverse_patch": [
      {
        "op": "replace",
        "path": "/description",
        "value": "This actor represents all Russian state-sponsored offensive activities in the cyberspace."
      },
      {
        "op": "replace",
        "path": "/modified",
        "value": "2022-08-16T08:33:23.769Z"
      },
      {
        "op": "replace",
        "path": "/extensions/extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba/updated_at",
        "value": "2022-08-16T08:33:23.769Z"
      }
    ]
  }
}

Update events producing creation or deletion
One of the most important things to understand about live streams is that the event type changes depending of the situation. As you can apply filters on top of the stream some events will need a translation when an object enters in the filters criteria or goes out of the filters. Let’s take a simple example.

We create a live stream providing indicators with the label “emotet”.

Here is what will happen in this stream.

Using this new approach, the stream consumers can now have all information required to react precisely to what is happening in the platform.

Embracing STIX 2.1 Extensions

As part of the rework of the stream features we also migrated our stream to embrace STIX 2.1 extensions. If you used the stream before 5.3.x for your own usage you will need to migrate your consumers to access OCTI extended information in the dedicated extensions.

We introduced 3 extension definitions where you will find information instead of the x_ flat attributes.

MITRE Corporation extension
In the MITRE datasets, they have defined some custom attributes like:

• x_mitre_id
• x_mitre_detection
• x_mitre_permissions_required
• x_mitre_platforms

Here is the definition of the new MITRE extension:

STIX_EXT_MITRE
'extension-definition--322b8f77-262a-4cb8-a915-1e441e00329b';

We selected this fixed UUID for now as the MITRE Corporation has not defined it yet. It will be changed in the future if they decide to not use this one.

OpenCTI SDO & SCO extension
We have also introduced multiple custom attributes in OCTI to modelize missing information in the STIX standard. 2 different extensions are now used to represent the information.

STIX_EXT_OCTI_SCO
'extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82';
STIX_EXT_OCTI
'extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba';

To better understand the final format of all of this, see the example below of a STIX data available in the stream.

{
  "id": "threat-actor--7c4952de-b739-53ac-9551-c59825e6f06c",
  "spec_version": "2.1",
  "type": "threat-actor",
  "extensions": {
    "extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba": {
      "extension_type": "property-extension",
      "id": "8d7f5990-d4ce-43c7-9333-f6a5b921865b",
      "type": "Threat-Actor",
      "created_at": "2022-05-28T09:25:21.046Z",
      "is_inferred": false
    }
  },
  "created": "2022-05-28T09:25:21.046Z",
  "modified": "2022-06-05T13:28:17.373Z",
  "revoked": false,
  "confidence": 75,
  "lang": "en",
  "name": "Russian cyber threats",
  "description": "...",
  "threat_actor_types": ["nation-state"],
  "resource_level": "individual",
  "primary_motivation": "coercion",
  "secondary_motivations": ["ideology"],
  "personal_motivations": ["organizational-gain"]
}

Conclusion

As you can see in this article, data sharing is still a challenge we really care about. The new implementation of the stream allowing more use cases to be implemented and the new CSV feeds feature will be completed in the future with even more capabilities.

Obviously, everything is not perfect and we certainly missed some important requirements or improvements. We would like to continue to listen the OCTI community to develop and enhance depending on the needs of organizations using the platform on a daily basis. Please join our community don’t hesitate to share your ideas!

Just a reminder about the other mecanisms available to share data with OCTI.

GraphQL API

GraphQL is a query language for new-generation APIs (please see https://graphql.org). The OCTI API uses a standard implementation of this language for the user interface, allowing us to query exactly the amount of data we need. Anyone can use it to do its own data extraction. It’s quiet simple to use but requires to understand the philosophy behind it. More information in the previous post.

TAXII 2.1 Collections

OpenCTI has an embedded TAXII™ API endpoint which provides valid STIX 2.1 bundles. If you wish to know more about the TAXII™ standard, please read the official introduction.