Preview data ingestion with draft workspaces in OpenCTI 6.6

Being able to preview exactly what your data coming in will look like when ingested is essential for maintaining a healthy and sanitized knowledge base. The introduction of draft workspaces in OpenCTI 6.6 aims to offer a powerful and versatile tool to manipulate data in a draft space, separate of the main knowledge base of the platform, to validate the ingestion of the data when it is deemed acceptable.

---

Introducing draft workspaces

Until now, it was possible to use analyst workbenches for previewing entities that would be sent for ingestion.

However, workbenches had limitations:

• data could only be previewed in a specific screen, and the information displayed was limited.
• adding and removing data in the workbench could only be done through the specific workbench screen, separate from the rest of the platform.
• there could be differences between what the workbench was displaying and the result of the workbench ingestion depending on wether the entity already existed, depending on the result of enrichment connectors etc..

Draft workspaces aim to replace analyst workbenches: it will enable what was possible with workbenches, but will also offer much more than that. The end goal of draft is to offer the same capabilities than the platform can offer, but in a draft environement, where your changes don’t affect the main datbase.

Generating draft workspaces

Draft workspaces can be created in multiple ways.

The first way is to create it manually from the draft menu.

To see this data, you can enter in the draft from the draft menu.

The second, and probably most useful way, it to choose the draft validation mode when importing a file in the ‘step-by-step’ import mode.

When this validation mode is selected, a draft will be created and will contain all of the uploaded files as well as the result of the file imports.

If files are uploaded through the ‘direct/automatic import’ mode, the validation mode used will depend on your platform and connectors configuration. By default, the validation mode used will be workbenches, but this can be changed by changing the APP__VALIDATION_MODE platform parameter to ‘draft’.

To ease the conversion from workbenches to draft, we’ve also introduced the capability to convert an existing workbench into a draft. This will let try out drafts in a smooth manner, and will enable you to see what any existing workbench would look like when imported into a draft.

Navigating in a draft workspace

Once your draft has been created, you can then enter the draft, and see the current changes contained in it.

Entering a draft is a session-wide action: all the pages that you will access in the platform will now be in the context of this draft. You will still be able to see the up to date and latest main knowledge base data, but you will see your draft data in addition.

To remind you of this context data, a header is always visible when in a draft

When first entering a draft, you will land on the draft overview page

This overview page gives you a way to see every entity that was changed within the context of your draft. You will be able to see what kind of operation was applied to your data in the draft: if it was created within this draft, if it was updated, or if it was marked for deletion.

Outside of this overview page, you will be able to browse seamlessly through the platform the same way you would without being in a draft. Only the knowledge related pages are available, and not the settings pages. You are able to create, update or delete data just the same way you would usually do. Your actions will automatically be recognized as draft actions, and will not impact the main knowledge base of the platform.

Other actions are also available in a draft: you can ask for enrichment of your data, just the same way you would normally. The enrichment data will be applied only within your draft.

Managing Your Draft Content

If you decide that you want to import more data to your current draft, this is also available within the File tab of your draft overview

This imported data will be added to your current data already available in your draft.

If you want to revert changes that have been applied in your draft, it is possible to do so: by using the ‘Remove from draft’ mass operation, all the changes applied to these entities will be reverted. If the entity was created in the draft, it will be fully deleted, and if the entity was simply updated, all of those updates will be reverted.

Validating draft workspaces

When you deem that the data in your draft is in a correct state, you can choose to validate your draft. Validating a draft is non-reversible: once validated, the draft state will change, and you will no longer be able to add or modify anything within this draft. When validated, all of the data contained in your draft will be sent for final ingestion, and merged in your main knowledge base.

Once a draft is validated, you can still go back to see what was contained in your draft. When opening a validated draft, you will see a modified version of the draft overview.

When opening the draft, you will no longer enter into the draft context: only the data that was there when it was validated will be visible. You will no longer be able to add any new data to your draft, and you will no longer be able to validate it again.

Future improvements

While already offering a lot more possibilities than workbenches, there are a few features that are not compatible in a draft. Currently, playbooks and rules do not apply to data modified in a draft. This could create situations where the data in your draft will be further modified when ingested. One improvement that will be coming later for drafts is the integration of playbooks and rules into it. This way, your data will look exactly the same way it will be after it has been validated.

An other future improvement will also be the addition of a validation workflow to draft. Currently, any user that can see a draft is able to validate. The introduction of a validation workflow will let users and administrators configure designated validators for draft. This way, users that create and populate a draft will then be able to send their draft for review to a specified validator user, who will be responsible for the final validation of the draft.

Conclusion

Workbenches will remain available to use, but will be deprecated and slowly removed in the coming versions. The current planning is for workbenches to have a first deprecation step in 6.9: you will no longer be able to create new workbenches, but will still be able to manipulate your currently opened workbenches. On the 6.11 release, the second deprecation step will come in, and the workbenches will then be fully removed.

Drafts are a powerful way to preview what data ingestion will really look like. While analyst workbenches were limited in their approach to ingestion preview, drafts unlock a whole new way of verifying and approving your data coming in. We hope that the capabilities unlocked with drafts will ease the way data that data are imported and ingested in the platform, and that it will make the process of data refinement even more smooth.

Join us on our Slack community channel to tell us what you think!