Solve CSV data import with CSV Mapper

Cybersecurity analysts frequently handle lists of Cyber Threat Intelligence (CTI) data in Excel or CSV format. Yet, effective data management is crucial in cybersecurity. To address this need, the OpenCTI team has recently introduced a groundbreaking feature that simplifies the process of importing CTI data from CSV files. This feature empowers users to create customized mappings for their CSV files, making it easier to ingest those table data into STIX entities and relationships in the OpenCTI platform.

This feature is available under the Processing menu in the Data section and with the appropriate capability: Manage CSV mappers.

Mapping CSV data to STIX Entities

The CSV Mapper allows users to create a seamless bridge between their CSV data and the platform’s STIX entities and relationships. Users can map the columns of their CSV files to the properties of all entities in the platform. The mapping of an entity (or relationship) and its properties is called a Representation.

Users have the capability to define one or more Representations, which allows for the import of various data types in one go. Consequently, if your data encompasses multiple CTI elements such as indicators, threat actors, or other related entities within a single CSV file, the CSV Mapper guarantees an efficient and error-free import process.

Establishing Relationships

The CSV Mapper enables the creation of relationship representations by leveraging existing entity representations. This integration allows users to forge connections between diverse entities during the data import phase, facilitating a more coherent analysis of the interrelations among various CTI data components.

Enhanced Mapping Options

To further enhance data quality and consistency, the following advanced options are available:

1. Pattern Date and Timezone: When dealing with date and time data in CSV files, the Mapper allows users to define a pattern and timezone. This feature ensures that date and time properties of entities are correctly formatted according to user preferences.
2. Multiple Entries: CTI datasets often feature entities with several attributes, such as different types of reports that may be listed together. The Mapper facilitates the distinction of these entries by allowing users to define a separator, which helps to parse and import multiple attributes from a single CSV column accurately.

Step-by-step Guide

Step 1: Describe configuration of your CSV file

• Ensure that the first line of your CSV contains the column headers
• Specify which CSV separator (comma or semicolon) is used in your file.

Step 2: Define entities representations

• Add a new entity
• Select the appropriate column index corresponding to each property
• Specify a separator for columns that include multiple entries for a single property
• Define a date pattern for date-related properties, if applicable

Step 3: Describe relationships representations

• Add a new relationship
• Select an appropriate entity representation for each relevant property
• Choose the corresponding column index that applies to each property
• If dealing with properties that include multiple values within a single column, designate a separator
• For date-related properties, establish a date pattern to ensure consistency

Step 4 : Test your CSV mapper with your file

Step 5: Confirm validity of your CSV mapper

All CSV Mappers go through a quick validation that checks if all the representations have all their mandatory fields set. Only valid mappers can be run by the users on their CSV files.

Mapper validity is visible in the list of CSV Mappers as shown below.

Step 6: Use your CSV Mapper with the brand new built in connector

Your csv mapper will be available as setup on the built in ImportCsv connector.
You can use it in your usual data import workflow.

And configure it in your configuration file.

“import_csv_built_in_connector”: {
 “enabled”: true, 
 “interval”: 10000, # connector launch interval
 “validate_before_import”: false
},

Next steps

Work on the CSV Mapper features is ongoing. Future developments will include the ability to map CSV files that have columns with varying meanings. For instance, we could map a CSV file that lists different types of Observables. We also plan to allow users to set default values to complete mappings from CSV files that may not have all the necessary information. Additionally, considering that some CSV files may include comments, we intend to introduce a feature that enables users to ignore specific lines in the CSV.

Conclusion

OpenCTI’s CSV Mapper represents a substantial advancement in CTI data management. It facilitates a streamlined and improved import process by enabling users to meticulously map CSV data to the corresponding entities within the platform. This powerful tool significantly enhances the efficiency of importing CTI data from CSV files.

If you have any question, request, comment or feedback to share with us, don’t hesitate to join us on Slack!