Threat Intel Lookup

This document describes the use of the Threat Intelligence lookup functions in MSTICPy. These of individual or multiple IoCs from one or more TI providers.

The Threat Intelligence component is broken into the TILookup class and multiple provider classes. You can extend TILookup by subclassing either the HttpLookupProvider provider or TIProvider base classes.

Notebook

There is a notebook based on this document available in the MSTICPy GitHub repo: TIProvider Usage Notebook

Features

Supports simultaneous lookup of IoCs against multiple providers.
Providers include:
- VirusTotal
- AlienVault OTX
- RiskIQ
- IBM XForce
- MS Sentinel TI
- GreyNoise
- CrowdSec
- AbuseIPDB
Other pseudo-TI providers are also included:
- TOR exit nodes
- Open Page Rank
- IP Quality Score
- BinaryEdge
Supports common IoC Types
- IP address
- URL
- Domain name
- File hash
- Some providers also support file name/path lookups
Requests to individual providers are run asynchronous so there is little or no performance penalty in querying multiple providers.
TI Results browser allows you to step through and examine details of TI results from different providers.

Introduction/Quickstart

The content in this section is covered in more detail later in the document.

You can do TI Lookups either from the TILookup class or from Entity TI pivot functions.

The latter are more convenient for most cases but one limitation is that pivot functions only work with IoC/Observable type corresponding to the entity type (e.g. Url.ti.lookup_url() cannot be used to look up IpAddresses.)

For bulk lookup of a mixture of IoC types you should use the TILookup class lookup_iocs method.

Note

the following operations require TI Provider configuration in your msticpyconfig.yaml. Please see the following section “Configuring providers”.

Initializing MSTICPy

import msticpy as mp
mp.init_notebook();

Using Pivot TI Functions

IpAddress.tilookup_ip("162.244.80.235")

Observables processed: 100%|██████████| 3/3 [00:00<00:00, 600.44obs/s]

Ioc	IocType	SanitizedValue	Provider	Result	Severity	Details	RawResult	Reference
162.244.80.235	ipv4	162.244.80.235	RiskIQ	True	high	{‘summary’: {‘resolutions’: 12, ‘certific..	{‘summary’: {‘..	https://community.riskiq.com
162.244.80.235	ipv4	162.244.80.235	VirusTotal	True	high	{‘verbose_msg’: ‘IP address in dataset’, ..	{‘asn’: 19624,..	https://www.virustotal.com/vtapi/v2/ip-address/report
162.244.80.235	ipv4	162.244.80.235	OTX	True	high	{‘pulse_count’: 45, ‘names’: [‘Conti Ran..	{‘whois’: ‘htt..	https://otx.alienvault.com/api/v1/indicators/IPv4/1..

For further details see the later section IoC Lookups using Pivot Functions.

For more information on use of pivot functions see Pivot Functions.

TILookup

When using the standalone class TILookup, you need to create an instance of this class before calling lookup_ioc.

Note

We are passing result to TILookup.result_to_df since the single-IoC method by default returns the format in a somewhat verbose format.

For single-IoC lookups, pivot functions are generally recommended.

ti_lookup = mp.TILookup()
# need to pass result to `result_to_df`
# to get results as a DataFrame
results_df = mp.TILookup.result_to_df(
    ti_lookup.lookup_ioc("162.244.80.235")
)
results_df.head(3)

Ioc	IocType	SanitizedValue	Provider	Result	Severity	Details	RawResult	Reference
162.244.80.235	ipv4	162.244.80.235	RiskIQ	True	high	{‘summary’: {‘resolutions’: 12, ‘certific..	{‘summary’: {‘..	https://community.riskiq.com
162.244.80.235	ipv4	162.244.80.235	VirusTotal	True	high	{‘verbose_msg’: ‘IP address in dataset’, ..	{‘asn’: 19624,..	https://www.virustotal.com/vtapi/v2/ip-address/report
162.244.80.235	ipv4	162.244.80.235	OTX	True	high	{‘pulse_count’: 45, ‘names’: [‘Conti Ran..	{‘whois’: ‘htt..	https://otx.alienvault.com/api/v1/indicators/IPv4/1..

For further details see the later section Looking up IoCs.

Multiple observable lookups

The syntax for pivot functions is unchanged but for TILookup you need to use a different method: lookup_iocs.

Using Pivot function for IpAddress

ips = ['162.244.80.235', '185.141.63.120', '82.118.21.1', '85.93.88.165']

results_df = IpAddress.tilookup_ip(ips)
results_df.head(3)

Using the generic lookup_iocs TILookup method. In this case, the result is already formatted as a DataFrame, so you do not need to use the results_to_df function.

results_df = ti_lookup.lookup_iocs(ips)
results_df.head(3)

For further details see the later section Looking up Multiple IoCs.

Browse Results

To make it easier to read through the results, MSTICPy has a TI browser, that allows you to page through individual results.

mp.TILookup.browse(results_df)

For further details see the later section Browsing and Selecting TI Results

Configuring TI providers

All providers require configuration before TILookup will load the drivers for the provider.

Most providers need you to register an account and obtain an API key.

For the MS Sentinel TI provider, the configuration is the workspace and tenant ID of the workspace containing your TI data table.

You can edit your msticpyconfig.yaml in an editor or use the MSTICPy config editor.

We’ll use the configuration editor here and then show an example of what the final msticpyconfig.yaml would look like.

Procedure

Sign up for and obtain an API key (in cases like XForce, you also need a user identifier).
In the MPConfigEdit “TI Providers” tab, select a provider from the Add prov selection list and click the Add button.
Select storage type of “Text” and paste the API/Auth key (and in some cases API ID) into the text box.
Click on Update
Repeat for any other providers that you want to add.
Verify that the Conf File path to your msticpyconfig.yaml is correct.
Click Save Settings

For the current sign-up pages (usually these are in the “Developer” section of the site), go to the relevant provider site.

config = mp.MpConfigEdit()
config.set_tab("TI Providers")
config

After saving the file you should see something like the following in your msticpyconfig.yaml.

Important

We do not recommend keeping your API keys stored as plain text in your configuration file. You have the option to store these as environment variables or as secrets in Azure Key Vault. For more information on how to do this see MSTICPy Settings Editor and MSTICPy Package Configuration

The providers should reflect what you picked in the config editor. The UUIDS for the keys in the example are fictitious - the format of the keys may differ from what is shown

TIProviders:
  OTX:
    Args:
      AuthKey: 9e4c7cbf-6b34-47b5-a1a0-535dbec6e790
    Primary: True
    Provider: "OTX"
  VirusTotal:
    Args:
      AuthKey: 13e5e78a-e59d-4a71-95d1-b3ba87422925
    Primary: True
    Provider: "VirusTotal"
  XForce:
    Args:
      ApiID: 269fe6f8-c41c-4255-a90d-bc5025b0305a
      AuthKey: 5bec9a70-24c3-4a0b-9bba-cc87907d039c
    Primary: True
    Provider: "XForce"
  GreyNoise:
    Args:
      AuthKey: d9dde9d4-b848-4cef-b0ee-40d2b23ba088
    Primary: True
    Provider: "GreyNoise"
  AzureSentinel:
    Args:
      WorkspaceID: c7d6a1ad-357b-48b2-8ee1-a2dcbfa2842b
      TenantID: 228d7b5f-4920-4f8e-872f-52072b92b651
    Primary: True
    Provider: "AzSTI"
  CrowdSec:
    Args:
      AuthKey: [PLACEHOLDER]
    Primary: True
    Provider: "CrowdSec"
  AbuseIPDB:
    Args:
      AuthKey: 1234567890
    Primary: True
    Provider: "AbuseIPDB"

You need to tell TILookup to refresh its configuration.

After reloading the provider settings, you should see a list of providers loaded.

ti_lookup.reload_providers()
ti_lookup.provider_status

['OTX - AlientVault OTX Lookup. (primary)',
'VirusTotal - VirusTotal Lookup. (primary)',
'XForce - IBM XForce Lookup. (primary)',
'GreyNoise - GreyNoise Lookup. (primary)',
'AzSTI - Microsoft Sentinel TI provider class. (primary)',
'CrowdSec - CrowdSec CTI Smoke Lookup. (primary)',
'AbuseIPDB - AbuseIPDB Lookup. (primary)']

Warning

Depending on the type of account that you have with a provider, they will typically impose a limit on the number of requests that you can make each minute or hour. If you see results returning with a status of 403, it is likely that you have temporarily exceed you query quota.

Note

If you have your MS Sentinel workspace and tenant IDs configured as a “Default” entry in the AzureSentinel configuration section of the msticpyconfig.yaml you do not need to set these values for the provider here. They will be inherited from the global configuration. If you want to use a different workspace for your TI lookups then specify the workspace and tenant IDs here. The tenant ID must be the same as in both cases though: the Kqlmagic data query library does not support access workspaces in multiple tenants from the same notebook.

Tip

If you are missing a required parameter for a provider, TILookup will throw an exception. You can use the providers parameter to TILookup

TILookup(providers=[“prov”,…])

to specify which providers to load and avoid loading any that causing problems.

TILookup class

The TILookup class is the main interface to the TI Functions.

Pivot functions also call the lookup_iocs method of this class.

Brief help is shown below. You can read more details about the attributes and functions in the TILookup documentation

To use TILookup, you need to create an instance of the class. Avoid creating lots of instances of this class:

Each instance caches recent results to avoid unnecessary network requests (instances do not share this cache)
The enabled state of providers is not share across instances.

Note

The pivot functions use a single central TILookup instance so are not affected by this.

Input to the lookup methods can be a single IoC observable or a pandas DataFrame (or Python iterable such as a list) containing multiple observables.

Note

the equivalent Pivot functions can accept single values, DataFrames or lists/Python iterables as inputs.

TILookup API documentation

Constructor

See TILookup

Initialize TILookup instance.

Parameters
----------
primary_providers : Optional[List[TIProvider]], optional
    Primary TI Providers, by default None
secondary_providers : Optional[List[TIProvider]], optional
    Secondary TI Providers, by default None
providers: Optional[List[str]], optional
    List of provider names to load, by default all available
    providers are loaded. To see the list of available providers
    call TILookup.list_available_providers().
    Note: if primary_provides or secondary_providers is specified
    This will override the providers list.

Methods

add_provider() - Add a TI provider to the current collection.
available_providers - Return a list of built-in providers.
list_available_providers() - Print a list of built-in providers with optional usage. (class method)
loaded_providers() - Return dictionary of loaded providers.
configured_providers - Return a list of providers with valid configuration
disable_provider() - disable a named provider
enable_provider() - enable a disabled provider
set_provider_state() - configure multiple providers as enabled/disabled
lookup_ioc() - Lookup single IoC in active providers.
lookup_iocs() - Lookup a collection of IoCs.
provider_status - Return loaded provider status.
provider_usage() - Print usage of loaded providers.
reload_provider_settings() - Reload provider settings from config.
reload_providers() - Reload settings and provider classes.
result_to_df() - Return DataFrame representation of IoC Lookup response.

You can change which providers are loaded from the TILookup constructor. However, this is usually not needed.

You can change the providers used in each lookup call by supplying a list of provider names in the `providers` parameter. You can also use the enable_provider, disable_provider and set_provider_state methods to make individual providers active and inactive.

Note

Using either the constructor parameters or the enable/disable methods only affects the current instance of TILookup. If you create a new instance, the changes to providers in previously-created instances have no effect on the state of the new instance. The instance of TILookup used by the Pivot functions is created when MSTICPy is initialized.

Querying and Configuring the Pivot TILookup

Pivot functions use their own instance of TILookup. To access this instance of TILookup use the Pivot providers property

# Note Pivot is only initialized after running mp.init_notebook()
mp.pivot.providers["TILookup"]

{'OTX': <msticpy.context.tiproviders.alienvault_otx.OTX at 0x2494f626490>,
'OPR': <msticpy.context.tiproviders.open_page_rank.OPR at 0x2494f626400>,
'RiskIQ': <msticpy.context.tiproviders.riskiq.RiskIQ at 0x2494f626a60>,
'Tor': <msticpy.context.tiproviders.tor_exit_nodes.Tor at 0x2494f5b0c10>,
'VirusTotal': <msticpy.context.tiproviders.virustotal.VirusTotal at 0x2494eca3850>,
'XForce': <msticpy.context.tiproviders.ibm_xforce.XForce at 0x2494ecae970>}

You can call any of the methods available on standalone TILookup instances - such as enabling and disabling providers - on this instance.

Listing Available Providers

The msticpy TI Provider library can lookup IoCs in multiple providers.

“Available Providers” means providers that have a MSTICPy provider class, not that they are necessarily either configured or loaded.

The list below shows the currently implemented set of MSTICPy TI providers.

The list_available_providers class method shows the current set of providers.

>>> TILookup.list_available_providers()

AzSTI
GreyNoise
OPR
OTX
Tor
VirusTotal
XForce
Intsights
CrowdSec
AbuseIPDB

You can view the list of supported query types for each provider with the show_query_types=True parameter. See

>>> TILookup.list_available_providers(show_query_types=True)

AzSTI
Azure Sentinel TI provider class. Supported query types:
  ioc_type=dns
  ioc_type=file_hash
  ioc_type=hostname
  ioc_type=ipv4
  ioc_type=ipv6
  ioc_type=linux_path
  ...

Loading TI Providers

Calling TILookup with no parameters will load all of the available providers that have a configuration entry in msticpyconfig.yaml (see Configuring TI providers). Pivot functions automatically load all configured providers.

# load all configured providers
ti_lookup = TILookup()

You can provide a list of providers to load when you create an instance of TILookup

# Restricting which providers get loaded
ti_lookup = TILookup(providers=["VirusTotal", "XForce"])
ti_lookup.provider_status

['VirusTotal - VirusTotal Lookup. (primary)',
 'XForce - IBM XForce Lookup. (primary)']

Tip

If you are missing a required parameter for a provider, TILookup will throw an exception. You can use the TILookup(providers=[“prov”,…]) parameter to load only specific providers.

Looking up IoCs

Lookup a single IoC

To lookup a single IoC use lookup_ioc.

Or you can use the pivot ti.lookup_xxx function from the appropriate entity.

Lookup an IoC from a single provider

And show the output

 result, details = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])

 print("Positive" if result else "Negative")
 # the details is a list (since there could be multiple responses for an IoC)
 for provider, detail in details:
     print(provider)
     detail.summary
     print("\nRaw Results")
     detail.raw_result_fmt

Positive
OTX
ioc: 38.75.137.9 ( ipv4 )
result: True
{   'names': ['Underminer EK'],
    'pulse_count': 1,
    'references': [   [   'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']],
    'tags': [[]]}
reference:  https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general

Raw Results
{ 'area_code': 0,
  'asn': 'AS63023 GTHost',
  'base_indicator': { 'access_reason': '',
                      'access_type': 'public',
                      'content': '',
                      'description': '',
                      'id': 2127020821,
                      'indicator': '38.75.137.9',
                      'title': '',
                      'type': 'IPv4'},
  'charset': 0,
  'city': 'Los Angeles',
  'city_data': True,
  'continent_code': 'NA',
  'country_code': 'US',
  'country_code3': 'USA',
  'country_name': 'United States',
  'dma_code': 803,
  'flag_title': 'United States',
  'flag_url': '/assets/images/flags/us.png',
  'indicator': '38.75.137.9',
  'latitude': 34.0584,
  'longitude': -118.278,
  'postal_code': '90017',
  'pulse_info': { 'count': 1,
                  'pulses': [ { 'TLP': 'white',
                                'adversary': '',
                                'attack_ids': [],
                                'author': { 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
                                            'id': '79520',
                                            'is_following': False,
                                            'is_subscribed': False,
                                            'username': 'mattvittitoe'},
                                'cloned_from': None,
                                'comment_count': 0,
                                ....[truncated for brevity]
                                'name': 'Underminer EK',
                                'public': 1,
                                'pulse_source': 'web',
                                'references': [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'],
                                'subscriber_count': 10,
                                'tags': [],
                                'targeted_countries': [],
                                'threat_hunter_scannable': True,
                                'upvotes_count': 0,
                                'validator_count': 0,
                                'vote': 0,
                                'votes_count': 0}],
                  'references': [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']},
  'region': 'CA',
  'reputation': 0,
  'sections': [ 'general',
                'geo',
                'reputation',
                'url_list',
                'passive_dns',
                'malware',
                'nids_list',
                'httpscans'],
  'type': 'IPv4',
  'type_title': 'IPv4',
  'whois': 'http://whois.domaintools.com/38.75.137.9'}

Pivot function syntax

IpAddress.ti.lookup_ip("38.75.137.9", providers=["OTX"])

Lookup using all primary providers

TILookup syntax

result = ti_lookup.lookup_ioc(observable="38.75.137.9")
ti_lookup.result_to_df(result)

	IoC	IoCType	Result	Details	RawResult	Reference	Status
OTX VirusTotal XForce AzSTI GreyNoise CrowdSec AbuseIPDB	38.75.137.9 38.75.137.9 38.75.137.9 38.75.137.9 38.75.137.9 38.75.137.9 38.75.137.9	ipv4 ipv4 ipv4 ipv4 ipv4 ipv4 ipv4	True True True False False False True	“{‘pulse_count’: 1, ‘names’: [‘Underminer EK’], ‘tags’: [[]], ‘references’: [[’https://blog.malw…” “{‘verbose_msg’: ‘IP address in dataset’, ‘response_code’: 1, ‘detected_urls’: [’http://38.75.13…” “{‘score’: 1, ‘cats’: {}, ‘categoryDescriptions’: {}, ‘reason’: ‘Regional Internet Registry’, ‘re…” “0 rows returned.” “Not found.” “{‘Background Noise’: 0, ‘Overall Score’: 0, ‘First Seen’: ‘2021-12-26T18:45:00+00:00’, ‘Last See…” “{‘countryCode’: ‘US’, ‘usage’: ‘Data Center/Web Hosting/Transit’, ‘isp’: ‘GlobalTeleHost Corp.’,…}”	{‘sections’: [‘general’, ‘geo’, ‘reputation’, ‘url_list’, ‘passive_dns’, ‘malware’, ‘nids_list’,…} {‘asn’: 63023, ‘undetected_urls’: [[’http://38.75.137.9:9088/’, ‘3d5edb0e0bb726e414a9b76dac619c…} {‘ip’: ‘38.75.137.9’, ‘history’: [{‘created’: ‘2012-03-22T07:26:00.000Z’, ‘reason’: ‘Regional In…} None <Response [404]> {‘ip_range_score’: 0, ‘ip’: ‘38.75.137.9’, ‘ip_range’: ‘38.75.136.0/23’, ‘as_name’: ‘AS-GLOBALTE…} {‘data’: {‘ipAddress’: ‘38.75.137.9’, ‘isPublic’: True, ‘ipVersion’: 4, ‘isWhitelisted’: None, …}}	https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general https://www.virustotal.com/vtapi/v2/ip-address/report https://api.xforce.ibmcloud.com/ipr/38.75.137.9 None https://api.greynoise.io/v3/community/38.75.137.9 https://cti.api.crowdsec.net/v2/smoke/38.75.137.9 https://api.abuseipdb.com/api/v2/check	200 200 200 -1 404 200 200

Pivot function syntax

IpAddress.ti.lookup_ip("38.75.137.9")

How to convert a raw result to a DataFrame

Note

The pivot functions always return a DataFrame so you do not need to use result_to_df

See result_to_df

result = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
ti_lookup.result_to_df(result).T

	OTX
IoC	38.75.137.9
IoCType	ipv4
QuerySubtype	None
Result	True
Details	{‘pulse_count’: 1, ‘names’: [‘Underminer EK’], ‘tags’: [[]], ‘references’: [[’https://blog.malw…
RawResult	{‘sections’: [‘general’, ‘geo’, ‘reputation’, ‘url_list’, ‘passive_dns’, ‘malware’, ‘nids_list’,…
Reference	https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general
Status	200

# Extract a single field (RawResult) from the dataframe (.iloc[0] is to select the row)
ti_lookup.result_to_df(result)["RawResult"].iloc[0]

{'sections': ['general',
  'geo',
  'reputation',
  'url_list',
  'passive_dns',
  'malware',
  'nids_list',
  'httpscans'],
 'city': 'Los Angeles',
 'area_code': 0,
 'pulse_info': {'count': 1,
  'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'],
  'pulses': [{'indicator_type_counts': {'URL': 16,
     'FileHash-MD5': 5,
     'IPv4': 3},
    'pulse_source': 'web',
    'TLP': 'white',
    'description': '',
    ...

Looking up Multiple IoCs

The pattern for looking up multiple IoCs in a single request is very similar to a single IoC.

Using TILookup.lookup_iocs

lookup_iocs documentation is available at lookup_iocs Example.

ioc_ips = [
    "185.92.220.35",
    "213.159.214.86",
    "77.222.54.202",
    "13.77.161.179",
    "40.76.4.15",
    "40.112.72.205",
    "40.113.200.201",
]

ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI")

	IoC	IoCType	QuerySubtype	Reference	Result	Status	Details	RawResult	Provider
0	213.159.214.86	ipv4	None	ThreatIntelligenceIndicator \| whe…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘0164ADB4A6CB7A79FBAE7BE90A43050B090A18364E3855048AC86B9DA5E0A92B’, ‘TimeGenerat…	AzSTI
1	40.113.200.201	ipv4	None	ThreatIntelligenceIndicator \| whe…	False	-1.0	0 rows returned.	NaN	AzSTI
2	91.219.29.81	ipv4	None	ThreatIntelligenceIndicator \| whe…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘3F458D91A21866C9037B99D997379A6906573766C0C2F8FB45327A6A15676A0D’, ‘TimeGenerat…	AzSTI
3	89.108.83.196	ipv4	None	ThreatIntelligenceIndicator \| whe…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘C3CA82D5B30A34F4BD6188C9DCFAD9E46D3C0CC45CC4FD969DA3A398DC34B1AE’, ‘TimeGenerat…	AzSTI
4	192.42.116.41	ipv4	None	ThreatIntelligenceIndicator \| whe…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘2F321C9D2593B6EF59DEB64B6CB209F375529C429F0DF463D639784E7353AA5D’, ‘TimeGenerat…	AzSTI

IoC Lookups using Pivot functions

Several MSTICPy entities have pivot functions that allow TI lookup. The type of IoC that they look up corresponds to the type of entity. For example, Url.ti.lookup_url() does lookups for URLs, File.ti.lookup_file_hash() does lookups for file hashes, etc.

The functionality of the pivot TI lookup functions is identical to TILookup.lookup_iocs (the pivot functions call lookup_iocs under the covers.). The syntax is also almost identical, with the exception that you should omit the ioc_type parameter, since this parameter is automatically supplied by the pivot subsystem.

Example

IpAddress.ti.lookup_ip(data=ioc_ips, providers="AzSTI")

For more information on use of pivot functions see Pivot Functions.

Asynchronous operation

When using multiple providers, TILookup will send the set of requests to each provider as an asynchronous operation. It splits the lookup job into a group of asychronous jobs (one for each provider) that are run simultaneously and in parallel.

The requests sent to a single provider are sent synchronously - i.e. one item is sent and a response awaited before the next item is sent.

Asynchronous operation means that a lookup using multiple providers should take no more time than the same lookup to a single provider - although the whole job will only complete once the slowest provider has completed.

Progress of the lookup job is shown using a progress bar. The request totals shown in the progres bar are requested_items * num_providers - e.g. a lookup of 10 items using 5 providers will show a total of 50.

Observables processed: 100%|██████████| 50/50 [00:00<00:00, 474.00obs/s]

Multiple IoCs using all providers

Output sorted by IoC

ioc_urls = [
    "http://cheapshirts.us/zVnMrG.php",
    "http://chinasymbolic.com/i9jnrc",
    "http://cetidawabi.com/468fd",
    "http://append.pl/srh9xsz",
    "http://aiccard.co.th/dvja1te",
    "http://ajaraheritage.ge/g7cberv",
    "http://cic-integration.com/hjy93JNBasdas",
    "https://google.com",  # benign
    "https://microsoft.com",  # benign
    "https://python.org",  # benign
]
results = ti_lookup.lookup_iocs(data=ioc_urls)
results.sort_values("IoC")

Observables processed: 100%|██████████| 50/50 [00:00<00:00, 474.00obs/s]

	IoC	IoCType	QuerySubtype	Result	Details	RawResult	Reference	Provider	Status
0	http://aiccard.co.th/dvja1te	url	None	True	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘FAE39C007D6554822504A1E0BDFD788E27DDC748ED63B258651DE52E4FA6D511’, ‘TimeGenerat…	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-21T17:30:41.900764Z) \| w…	AzSTI	0.0
4	http://aiccard.co.th/dvja1te	url	None	True	{‘cats’: None, ‘categoryDescriptions’: None}	{‘result’: {‘url’: ‘aiccard.co.th’, ‘cats’: {}, ‘score’: None, ‘categoryDescriptions’: {}}, ‘ass…	https://api.xforce.ibmcloud.com/url/http://aiccard.co.th/dvja1te	XForce	NaN
4	http://aiccard.co.th/dvja1te	url	None	True	{‘pulse_count’: 3, ‘names’: [‘Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)’, …	{‘indicator’: ‘http://aiccard.co.th/dvja1te’, ‘alexa’: ‘http://www.alexa.com/siteinfo/aiccard.c…	https://otx.alienvault.com/api/v1/indicators/url/http://aiccard.co.th/dvja1te/general	OTX	NaN
4	http://aiccard.co.th/dvja1te	url	None	False	No response from provider.	<Response [403]>	https://www.virustotal.com/vtapi/v2/url/report	VirusTotal	NaN
5	http://ajaraheritage.ge/g7cberv	url	None	True	{‘cats’: None, ‘categoryDescriptions’: None}	{‘result’: {‘url’: ‘ajaraheritage.ge’, ‘cats’: {}, ‘score’: None, ‘categoryDescriptions’: {}}, ‘…	https://api.xforce.ibmcloud.com/url/http://ajaraheritage.ge/g7cberv	XForce	NaN

Note

the URLs in the previous example have been altered to prevent inadvertent navigation to them.

Inferring IoC type vs specifying explicitly

If you do a lookup without specifying a type, TILookup will try to infer the type by matching regexes. There are patterns for all supported types but there are some caveats:

The match is not 100% foolproof - e.g. some URLs and hash types may be misidentified.
The inference adds an overhead to each lookup.

If you know the type that you want to look up, it is always better to explicitly include it.

For single IoC lookup, use the ioc_type parameter.
For multiple IoC lookups (see below), supply either:
- a DataFrame with a column that specifies the type for each entry
- a dictionary of the form {ioc_observable: ioc_type}

Browsing and Selecting TI Results

To make it easier to walk through the returned results msticpy has a browser. This shows you the TI results aggregated by the IoC value (e.g. an individual IP Address or URL) for all providers.

For each provider that returns a result for an IoC, the summarized details will be shown in a table below the browse list.

Click on Raw results from provider... to see all returned data.

Note

the reference URL may not work if you have not authenticated to the TI service.

The value of the selected IoC entry is available as ti_selector.value. You can match this back to the original results DataFrame as follows:

results[results["Ioc"] == ti_selector.value[0]]

Advanced Provider Usage - Query types

Some providers also support special types of sub-query such as geo-ip and passive-dns. To use these, you specify a query_type parameter in addition to the corresponding ioc_type.

See provider_usage

You can see which providers support special query types with the provider_usage method.

ti_lookup.provider_usage()

Primary providers
-----------------

Provider class: OTX
AlientVault OTX Lookup. Supported query types:
  ioc_type=dns
  ioc_type=dns, ioc_query_type=geo
  ioc_type=dns, ioc_query_type=passivedns
  ioc_type=file_hash
  ioc_type=hostname
  ioc_type=ipv4
  ioc_type=ipv4, ioc_query_type=geo
  ioc_type=ipv4, ioc_query_type=passivedns
  ....

Provider class: XForce
IBM XForce Lookup. Supported query types:
  ioc_type=dns, ioc_query_type=info
  ioc_type=dns, ioc_query_type=passivedns
  ioc_type=dns, ioc_query_type=whois
  ioc_type=file_hash
  ioc_type=hostname, ioc_query_type=whois
  ioc_type=ipv4
  ioc_type=ipv4, ioc_query_type=malware
  ioc_type=ipv4, ioc_query_type=passivedns
  ioc_type=ipv4, ioc_query_type=rep
  ioc_type=ipv4, ioc_query_type=whois
  ioc_type=ipv6
  ...

Provider class: GreyNoise
GreyNoise Lookup. Supported query types:
  ioc_type=ipv4
  ioc_type=ipv4, ioc_query_type=full
  ioc_type=ipv4, ioc_query_type=quick

Use to do a passive DNS lookup

 result = ti_lookup.lookup_ioc(
   observable="38.75.137.9",
   ico_type="ipv4",
   ioc_query_type="passivedns",
   providers=["XForce"])
 result

(True,
 [('XForce',
   LookupResult(ioc='38.75.137.9', ioc_type='ipv4', query_subtype='passivedns', result=True, details={'records': 1}, raw_result={'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}, reference='https://api.xforce.ibmcloud.com/resolve/38.75.137.9', status=200))])

Specifying Time Ranges

Some providers (currently only AzSTI - the Sentinel TI provider) support time ranges so that you can specify specific periods to search for.

If a provider does not support time ranges, the parameters will be ignored

 from datetime import datetime
 start = datetime(2021, 8, 5)
 end = datetime(2021, 9, 5)

 # Using this data range returned no results
 ti_lookup.lookup_iocs(
     data=ioc_ips,
     providers="AzSTI",
     start=q_times.start,
     end=q_times.end
 ).head()

	IoC	IoCType	QuerySubtype	Reference	Result	Details	Status	Provider
0 1 2 3 4	213.159.214.86 40.113.200.201 91.219.29.81 89.108.83.196 192.42.116.41	ipv4 ipv4 ipv4 ipv4 ipv4	None None None None None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-08-04T00:00:00Z) \| where Ti… ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-08-04T00:00:00Z) \| where Ti… ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-08-04T00:00:00Z) \| where Ti… ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-08-04T00:00:00Z) \| where Ti… ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-08-04T00:00:00Z) \| where Ti…	False False False False False	0 rows returned. 0 rows returned. 0 rows returned. 0 rows returned. 0 rows returned.	-1 -1 -1 -1 -1	AzSTI AzSTI AzSTI AzSTI AzSTI

from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(
    units="day",
    auto_display=True,
    origin_time=search_origin,
    max_after=24,
    max_before=24
)

# Using a wider ranges produces results
ti_lookup.lookup_iocs(
    data=ioc_ips,
    providers="AzSTI",
    start=q_times.start,
    end=q_times.end
)

	IoC	IoCType	QuerySubtype	Reference	Result	Status	Details	RawResult	Provider
0	213.159.214.86	ipv4	None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-12T00:00:00Z) \| where Ti…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘0164ADB4A6CB7A79FBAE7BE90A43050B090A18364E3855048AC86B9DA5E0A92B’, ‘TimeGenerat…	AzSTI
1	40.113.200.201	ipv4	None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-12T00:00:00Z) \| where Ti…	False	-1.0	0 rows returned.	NaN	AzSTI
2	91.219.29.81	ipv4	None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-12T00:00:00Z) \| where Ti…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘3F458D91A21866C9037B99D997379A6906573766C0C2F8FB45327A6A15676A0D’, ‘TimeGenerat…	AzSTI
3	89.108.83.196	ipv4	None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-12T00:00:00Z) \| where Ti…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘C3CA82D5B30A34F4BD6188C9DCFAD9E46D3C0CC45CC4FD969DA3A398DC34B1AE’, ‘TimeGenerat…	AzSTI
4	192.42.116.41	ipv4	None	ThreatIntelligenceIndicator \| where TimeGenerated >= datetime(2019-07-12T00:00:00Z) \| where Ti…	True	0.0	{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description…	{‘IndicatorId’: ‘2F321C9D2593B6EF59DEB64B6CB209F375529C429F0DF463D639784E7353AA5D’, ‘TimeGenerat…	AzSTI