Breaking change: evolution of the way Connector, Streams and Feeds import data in OpenCTI

Please advise: A change in how Connectors, Feeds, and Streams will use Confidence Level during data import will require a specific action from platform administrators before upgrading to the next major release.

---

How Connectors, Feeds and Streams use Confidence level currently

In OpenCTI, the “deduplication” mechanism prevents the duplication of the same Object when it is created from different sources or users. This helps maintain maximum data consistency over time.

This mechanism relies on the Confidence Level attribute of each STIX Object, as well as identification keys for each Object. If an Object is created with the same identification keys as an existing one, it will replace the existing Object only if the Confidence Level of the new Object is higher or equal to the existing one’s.

Connectors have a Confidence level in their configuration file that can be used for deduplication purpose. Therefore, if a Connector imports an Object already existing in the platform, it will only replace the existing one if its Confidence Level is higher or equal.

Current limits

This deduplication mechanism, although powerful, still has limitations:

• It is not possible to associate a Confidence Level with an OCTI Live stream or a TAXII/RSS Feed. You have no choice but to use the Confidence Level set by the source of the stream or feed. If the source has set a Confidence Level of 100 on an Object, it will overwrite the existing one in your platform, even if you value it highly.
• Connectors have a Confidence Level in their configuration that can be used during data import, if the connector has been programmed accordingly… If not, the limitation explained in the previous point applies.
• The Confidence Level is not mandatory when an Object is created via Connectors, streams or feeds, leading to inconsistency in platforms.

A Confidence level for every User to give you more control

To address these limitations, a “Max Confidence Level” attribute will be implemented for each Groups and Users in the platform in the next major release.

Connectors, Feeds, and OCTI Live Streams can be associated with a user. It is the actual best practice to be able to track what they do on the platform’s data. After the major release, Connectors, Feeds and OCTI Live Streams will need an associated user to work and import their data with their User’s “Max Confidence Level” as a threshold.

As a platform administrator, you will then have complete control over the impact of imported data on the existing content of your platform.

Furthermore, you will also have the ability to adjust Users’ knowledge modification capability. Users will no longer be able to modify an Object if it has a Confidence Level higher than their own max Confidence Level.

What to do before your upgrade to the next major release ?

If your platform administrator doesn’t take action before upgrading to the next major release, every user of your platform will be considered having no max confidence level. Users won’t be able to modify data anymore, and Connectors, Feeds and OCTI streams won’t be able to create and update data anymore either. By adopting this migration strategy, we want to avoid any loss and unwanted modification of data and facilitate the work for platform admin.

To avoid that, platform administrator will have to set a max Confidence level for every Group in the platform. Plus, if you work with connectors importing external data and want them to work exactly as before, you must set a max confidence level for their respective user equal to the confidence level currently set in the connector’s config file, as the later will not be used anymore.

In order to facilitate this work, we have done a minor release (5.12.24) allowing to set the max confidence level of Users and Groups before the next major one. Until the next major release, this attribute will not affect anything in your platform. The user’s “effective” max confidence level will be the highest value among max confidence level of Groups the user is part of. Of course, if users have a specific max confidence level set, it will override the value inherited from their Groups and be used instead.

We very highly recommend you to upgrade your platform to the 5.12.24 before upgrading to the nest major release to give yourself the time to define your confidence level policy and set it with the new max confidence level attribute of Groups.

As always, an important best practice is to make data backup before updating your platform.

GentleCorp’s Use case

Ok! you know now everything about what will happen. You know that you must set max confidence level for Groups in your platform.

But…what value should you set for each group?

That’s a great question and the answer will deeply depend on the different source of data you use, and how different can be your users usage of the platform.

GentleCorp uses OpenCTI on a daily basis for managing their CTI Knowledge from data collection to knowledge dissemination to local information security teams.

Users are organized into 4 groups:

• Connectors: a group containing all users that allows external connectors to create knowledge in the platform
• CTI Lead team: users well versed in CTI analysis and providing investigation on threats
• Local infosec team: mostly junior level CTI analyst embedded in local teams and providing analytics support
• Management: users part of CISO team that mainly consume knowledge but also might provide some strategic insights.

GentleCorp’s CTI’s stakeholders have met up to define how to reflect their CTI production strategy into this new max confidence level setting. Some general rules have been established:

• CTI Lead team will have the last word regarding CTI knowledge. They will be granted the highest max confidence level
• Connectors, by default, must not be able to replace what have been created internally. So they will have the lowest max confidence level of all Groups. But a previous work was done to set confidence level in config files of connectors. So this work must be reflected in the new max confidence level of their respective user.
• Local infosec team should be able to provide better knowledge than default and unvouched connectors. So they will have a higher max confidence level than connectors.
• Management currently add their strategic insights in the form of Notes in the platform, so they don’t really impact Knowledge directly. And other users and connectors do not use Notes currently. Let’s grant them the same max confidence level than connectors.
• Other users are in the “default” group that does not have capability to create or modify knowledge. Default will still get a max confidence level of 0, just to be sure it will not allow any unwanted modification from users that are not part of CTI production’s stakeholders.

Final max confidence levels for those Groups are set as follow:

• Default:0
• Connectors:50
• Management:50
• Local infosec:70
• CTI Lead team:100

GentleCorp is using 3 different connectors to ingest data from external providers. Connectors “CASI”, “NeighborVault” and “Eucalyptus”. Currently, config files of this connectors had been set respectively to 80, 50 and 75. So this value will be now set as the max confidence level of respectively “[c]CASI”, “[c]NeighborVault” and “[c]Eucalyptus” users associated with them. It will ensure these 3 connectors will continue to work as before.

GentleCorp is now ready to upgrade to the next major release! They ensure that their newly and shiny connectors will not wreck havoc in their Knowledge and that their CTI Lead team will be able to improve it with their most valuable investigation!

If you need help, do not hesitate to connect with the community on Slack!

Next steps

We will continue to add Confidence level features in order to bring to platform administrators full control on how imported data can impact existing one. Notably, we will implement:

• a way to differentiate Max Confidence level of users by Entity types
• a way to apply Confidence level at attributes level, to “protect” specific attributes from unwanted modifications or at the contrary, “release” them for modifications.