msticpy.context.tilookup module

Module for TILookup classes.

Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.

class msticpy.context.tilookup.TILookup(providers: list[str] | None = None, *, primary_providers: list[Provider] | None = None, secondary_providers: list[Provider] | None = None)

Bases: Lookup

Threat Intel observable lookup from providers.

Initialize TILookup instance.

Parameters:

  • primary_providers (Optional[list[Provider]], optional) – Primary Providers, by default None

  • secondary_providers (Optional[list[Provider]], optional) – Secondary Providers, by default None

  • providers (Optional[list[str]], optional) – List of provider names to load, by default all available providers are loaded. To see the list of available providers call TILookup.list_available_providers(). Note: if primary_provides or secondary_providers is specified This will override the providers list.

CUSTOM_PROVIDERS: ClassVar[dict[str, type[Provider]]] = {}
PACKAGE: ClassVar[str] = 'tiproviders'
PROVIDERS: ClassVar[dict[str, tuple[str, str]]] = {'AbuseIPDB': ('abuseipdb', 'AbuseIPDB'), 'AzSTI': ('azure_sent_byoti', 'AzSTI'), 'BinaryEdge': ('binaryedge', 'BinaryEdge'), 'CrowdSec': ('crowdsec', 'CrowdSec'), 'GreyNoise': ('greynoise', 'GreyNoise'), 'IPQualityScore': ('ip_quality_score', 'IPQualityScore'), 'IntSights': ('intsights', 'IntSights'), 'OPR': ('open_page_rank', 'OPR'), 'OTX': ('alienvault_otx', 'OTX'), 'Pulsedive': ('pulsedive', 'Pulsedive'), 'RiskIQ': ('riskiq', 'RiskIQ'), 'Tor': ('tor_exit_nodes', 'Tor'), 'VirusTotal': ('virustotal', 'VirusTotal'), 'XForce': ('ibm_xforce', 'XForce')}
add_provider(provider: Provider, name: str | None = None, primary: bool = True) → None

Add a provider to the current collection.

Parameters:

  • provider (TIProvider) – Provider instance

  • name (str, optional) – The name to use for the provider (overrides the class name of provider)

  • primary (bool, optional) – “primary” or “secondary” if False, by default “primary”

property available_providers: list[str]

Return a list of builtin and plugin providers.

Returns:

List of TI Provider classes.

Return type:

list[str]

classmethod browse(data: pd.DataFrame, severities: list[str] | None = None, *, height: str = '300px') → SelectItem | None

Return TI Results list browser.

Parameters:

  • data (pd.DataFrame) – TI Results data from TIProviders

  • severities (Optional[list[str]], optional) – A list of the severity classes to show. By default these are [‘warning’, ‘high’]. Pass [‘information’, ‘warning’, ‘high’] to see all results.

  • height (str, Optional) – Height of the widget

Returns:

SelectItem browser for TI Data.

Return type:

SelectItem

classmethod browse_results(data: pd.DataFrame, severities: list[str] | None = None, *, height: str = '300px') → SelectItem | None

Return TI Results list browser.

Parameters:

  • data (pd.DataFrame) – TI Results data from TIProviders

  • severities (Optional[list[str]], optional) – A list of the severity classes to show. By default these are [‘warning’, ‘high’]. Pass [‘information’, ‘warning’, ‘high’] to see all results.

  • height (str, Optional) – Height of the widget

Returns:

SelectItem browser for TI Data.

Return type:

SelectItem

property configured_providers: list[str]

Return a list of available providers that have configuration details present.

Returns:

List of TI Provider classes.

Return type:

list[str]

disable_provider(providers: str | Iterable[str]) → None

Set the provider as secondary (not used by default).

Parameters:

providers (Union[str, Iterable[str]) – Provider name or list of names. Use list_available_providers() to see the list of loaded providers.

Raises:

ValueError – If the provider name is not recognized.

enable_provider(providers: str | Iterable[str]) → None

Set the provider(s) as primary (used by default).

Parameters:

providers (Union[str, Iterable[str]) – Provider name or list of names. Use list_available_providers() to see the list of loaded providers.

Raises:

ValueError – If the provider name is not recognized.

classmethod import_provider(provider: str) → type[Provider]

Import provider class.

classmethod list_available_providers(show_query_types=False, as_list: bool = False) → list[str] | None

Print a list of builtin providers with optional usage.

Parameters:

  • show_query_types (bool, optional) – Show query types supported by providers, by default False

  • as_list (bool, optional) – Return list of providers instead of printing to stdout. Note: if you specify show_query_types this will be printed irrespective of this parameter setting.

Returns:

A list of provider names (if return_list=True)

Return type:

Optional[list[str]]

property loaded_providers: dict[str, Provider]

Return dictionary of loaded providers.

Returns:

[description]

Return type:

dict[str, TIProvider]

lookup_ioc(ioc: str | None = None, ioc_type: str | None = None, ioc_query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary', *, observable: str | None = None, show_not_supported: bool = False, start: dt.datetime | None = None, end: dt.datetime | None = None) → pd.DataFrame

Lookup Threat Intelligence reports for a single IoC in active providers.

Parameters:

  • ioc (str) – IoC observable

  • observable (str) – alias for ioc

  • ioc_type (str, optional) – One of IoCExtract.IoCType, by default None If none, the IoC type will be inferred

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • show_not_supported (boolean, optional) – If True, display result even if provider does not support this type of IOC.

  • start (dt.datetime, optional) – Time since when IOC is considered relevant

  • end (dt.datetime, optional) – Time until when IOC is considered relevant

Returns:

The result returned as a tuple(bool, list): bool indicates whether a TI record was found in any provider list has an entry for each provider result

Return type:

pd.DataFrame

See also

lookup_iocs

Lookup Threat Intelligence reports for a collection of IoCs.

Notes

Queries active Threat Intelligence (TI) providers for a single indicator of compromise (IoC). It returns results as a pandas DataFrame. ioc_type can be used to specify the type (ipv4, ipv6, dns, url, file_hash). If this is not supplied the type is inferred using regular expressions. By default, providers are queried asynchronously, in parallel.

lookup_iocs(data: pd.DataFrame | Mapping[str, str] | Iterable[str], ioc_col: str | None = None, ioc_type_col: str | None = None, ioc_query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary', *, start: dt.datetime | None = None, end: dt.datetime | None = None) → pd.DataFrame

Lookup Threat Intelligence reports for a collection of IoCs in active providers.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in ioc_col parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred

  • ioc_col (str, optional) – DataFrame column to use for observables, by default None (“col” and “column” are also aliases for this parameter)

  • ioc_type_col (str, optional) – DataFrame column to use for IoCTypes, by default None

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • start (dt.datetime, optional) – Time since when IOC is considered relevant

  • end (dt.datetime, optional) – Time until when IOC is considered relevant

Returns:

DataFrame of results

Return type:

pd.DataFrame

See also

lookup_ioc

Lookup Threat Intelligence reports for a single IoC.

Notes

lookup_iocs queries active Threat Intelligence (TI) providers for threat reports. It can accept input as a Python iterable or a pandas dataframe. In the latter case, you also need to supply the ioc_col parameter to indicate which column the IoC value can be found. The ioc_type_col parameter is optional and can be used to manually specify the IoC type for each row. If this is not supplied the ioc types are inferred using regular expressions. The results are returned as a pandas DataFrame.

lookup_iocs_sync(data: pd.DataFrame | Mapping[str, str] | Iterable[str], ioc_col: str | None = None, ioc_type_col: str | None = None, ioc_query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary') → pd.DataFrame

Lookup a collection of IoCs.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in ioc_col parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred

  • ioc_col (str, optional) – DataFrame column to use for observables, by default None (“col” and “column” are also aliases for this parameter)

  • ioc_type_col (str, optional) – DataFrame column to use for IoCTypes, by default None

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_item(item: str, item_type: str | None = None, query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary', *, show_not_supported: bool = False, start: dt.datetime | None = None, end: dt.datetime | None = None) → pd.DataFrame

Lookup single item in active providers.

Parameters:

  • item (str) – item to lookup

  • item_type (str, optional) – One of ItemExtract.ItemType, by default None If none, the Item type will be inferred

  • query_type (str, optional) – The query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_items(data: pd.DataFrame | Mapping[str, str] | Sized, item_col: str | None = None, item_type_col: str | None = None, query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary', *, show_not_supported: bool = False, start: dt.datetime | None = None, end: dt.datetime | None = None) → pd.DataFrame

Lookup a collection of items.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in item_col parameter) 2. Mapping (e.g. a dict) of [item, ItemType] 3. Iterable of items - ItemTypes will be inferred

  • item_col (str, optional) – DataFrame column to use for items, by default None (“col” and “column” are also aliases for this parameter)

  • item_type_col (str, optional) – DataFrame column to use for ItemTypes, by default None

  • query_type (str, optional) – The item query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • progress (bool) – Use progress bar to track completion, by default True

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_items_sync(data: pd.DataFrame | Mapping[str, str] | Iterable[str], item_col: str | None = None, item_type_col: str | None = None, query_type: str | None = None, providers: list[str] | None = None, default_providers: list[str] | None = None, prov_scope: str = 'primary', *, col: str | None = None, column: str | None = None, show_not_supported: bool = False, show_bad_item: bool = False) → pd.DataFrame

Lookup a collection of items.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in item_col parameter) 2. Mapping (e.g. a dict) of [item, ItemType] 3. Iterable of items - ItemTypes will be inferred

  • item_col (str, optional) – DataFrame column to use for items, by default None (“col” and “column” are also aliases for this parameter)

  • item_type_col (str, optional) – DataFrame column to use for ItemTypes, by default None

  • query_type (str, optional) – The item query type (e.g. rep, info, malware)

  • providers (list[str]) – Explicit list of providers to use

  • default_providers (Optional[list[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

Returns:

DataFrame of results

Return type:

pd.DataFrame

property provider_status: Iterable[str]

Return loaded provider status.

Returns:

List of providers and descriptions.

Return type:

Iterable[str]

provider_usage() → None

Print usage of loaded providers.

classmethod reload_provider_settings() → None

Reload provider settings from config.

reload_providers() → None

Reload settings and provider classes.

static result_to_df(item_lookup: DataFrame) → DataFrame

Return DataFrame representation of Lookup response.

Parameters:

item_lookup (pd.DataFrame) – Output from lookup_item

Returns:

The response as a DataFrame with a row for each provider response.

Return type:

pd.DataFrame

set_provider_state(prov_dict: dict[str, bool]) → None

Set a dict of providers to primary/secondary.

Parameters:

prov_dict (dict[str, bool]) – Dictionary of provider name and bool - True if enabled/primary, False if disabled/secondary.