Threat Intel Lookup
This document describes the use of the Threat Intelligence lookup class in msticpy. The class allows lookup of individual or multiple IoCs from one or more TI providers.
TILookup is also extensible - you can subclass TIProvider to implement your own custom lookups. You can also subclass the HTTPProvider or KqlProvider classes, which provide support for querying a REST endpoint or Log Analytics table respectively.
Notebook
TILookup class
Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.
Constructor
See TILookup
Initialize TILookup instance. Parameters ---------- primary_providers : Optional[List[TIProvider]], optional Primary TI Providers, by default None secondary_providers : Optional[List[TIProvider]], optional Secondary TI Providers, by default None providers: Optional[List[str]], optional List of provider names to load, by default all available providers are loaded. To see the list of available providers call TILookup.list_available_providers(). Note: if primary_provides or secondary_providers is specified This will override the providers list.
Methods
add_provider() - Add a TI provider to the current collection.
available_providers() - Return a list of built-in providers.
list_available_providers() - Print a list of built-in providers with optional usage. (class method)
loaded_providers() - Return dictionary of loaded providers.
lookup_ioc() - Lookup single IoC in active providers.
lookup_iocs() - Lookup a collection of IoCs.
provider_status() - Return loaded provider status.
provider_usage() - Print usage of loaded providers.
reload_provider_settings() - Reload provider settings from config.
reload_providers() - Reload providers based on current settings in config.
result_to_df() - Return DataFrame representation of IoC Lookup response.
Available Providers
The msticpy TI Provider library can lookup IoCs in multiple providers.
The list_available_providers
class method shows the current set of providers.
>>> TILookup.list_available_providers()
AzSTI
GreyNoise
OPR
OTX
Tor
VirusTotal
XForce
You can view the list of supported query types for each provider
with the show_query_types=True
parameter.
See
>>> TILookup.list_available_providers(show_query_types=True)
AzSTI
Azure Sentinel TI provider class. Supported query types:
ioc_type=dns
ioc_type=file_hash
ioc_type=hostname
ioc_type=ipv4
ioc_type=ipv6
ioc_type=linux_path
...
Loading TI Providers
Calling TILookup with no parameters will load all of the available providers
that have a configuration entry in msticpyconfig.yaml
(see next section).
# load all configured providers
ti_lookup = TILookup()
ti_lookup.provider_status
# Restricting which providers get loaded
#ti_lookup = TILookup(providers=["VirusTotal", "XForce"])
['VirusTotal - VirusTotal Lookup. (primary)',
'XForce - IBM XForce Lookup. (primary)']
Tip
If you are missing a required parameter for a provider, TILookup will throw an exception. You can use the TILookup(providers=[“prov”,…]) parameter to load only specific providers.
Configuration File
You must have a correctly configured msticpyconfig.yaml in order to use the TILookup module. In this file you specify the providers you want to load, any API keys that the provider services require. You can configure primary and secondary providers.
Primary providers are those used by default, when you query for IoCs without specifying any specific provider names. You may want to add some providers as secondary so that they are not used for every query but are available if you want to search more widely.
You will usually need to supply an authorization (API) key and in some cases a user ID for each provider.
Note
For LogAnalytics/Azure Sentinel providers, you will need the workspace ID and tenant ID and will need to authenticate in order to access the data (although if you have an existing authenticated connection with the same workspace/tenant, this connection will be re-used).
The configuration file is read from the current directory.
Alternatively, you can specify a location for this file in an
environment variable MSTICPYCONFIG
.
If you need to create a config file, use the content shown below.
The Provider
key must correspond to an available Python class.
The names of the built-in provider classes can be obtained using the
list_available_providers
function.
Delete any provider entries from the example below that you do not want
to use and add the missing parameters for your providers.
Save the file as msticpyconfig.yaml
or insert the TIProviders
section into your existing msticpyconfig.yaml
QueryDefinitions:
TIProviders:
OTX:
Args:
AuthKey: "your-otx-key"
Primary: True
Provider: "OTX" # Explicitly name provider to override
VirusTotal:
Args:
AuthKey: "your-vt-key"
Primary: True
Provider: "VirusTotal"
XForce:
Args:
ApiID: "your-xforce-id"
AuthKey: "your-xforce-key"
Primary: True
Provider: "XForce"
GreyNoise:
Args:
AuthKey: ""
Primary: true
Provider: GreyNoise
AzureSentinel:
# Note if you do not specify any settings in the Args key for the AzureSentinel
# provider, it will default to using your default Azure Sentinel workspace.
Args:
WorkspaceID: "your-azure-sentinel-workspace-id"
TenantID: "your-azure-sentinel-tenant-id"
Primary: True
Provider: "AzSTI"
Note
If you have your Azure Sentinel workspace and tenant IDs configured either in a config.json file or in the AzureSentinel configuration section of the msticpyconfig.yaml you do not need to set these values for the provider here. They will be inherited from the global configuration. If you want to use a different workspace for your TI lookups then specify the workspace and tenant IDs here. The tenant ID must be the same as in both cases though: the Kqlmagic data query library does not support access workspaces in multiple tenants from the same notebook.
You can also specify that the Args values as environment variables as follows:
XForce:
Args:
ApiID:
EnvironmentVar: "XFORCE_ID"
AuthKey:
EnvironmentVar: "XFORCE_KEY"
Primary: False
Provider: "XForce"
Note
You can also use Key Vault storage with optional local caching of the secrets using keyring. See msticpy Package Configuration for more details.
When you have made a configuration change you can reload the providers and check the status like this.
ti_lookup.reload_providers()
ti_lookup.provider_status
['OTX - AlientVault OTX Lookup. (primary)',
'VirusTotal - VirusTotal Lookup. (primary)',
'XForce - IBM XForce Lookup. (primary)',
'AzSTI - Azure Sentinel TI provider class. (primary)']
Tip
If you are missing a required parameter for a provider, TILookup will throw an exception. You can use the providers parameter to TILookup
TILookup(providers=[“prov”,…])
to specify which providers to load and avoid loading any that causing problems.
The providers parameter to TILookup affects which providers are loaded. When you are querying TI (using lookup_ioc or lookup_iocs) you can also specify a list of providers to use for the query. In the second case the providers list selects which of the loaded providers to send queries to. It does not cause additional providers to load.
Looking up IoCs
lookup_ioc
See lookup_ioc
To lookup a single IoC.
ti_lookup.lookup_ioc(
observable: str = None,
ioc_type: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> Tuple[bool, List[Tuple[str, msticpy.sectools.tiproviders.ti_provider_base.LookupResult]]]
Lookup single IoC in active providers.
Parameters
----------
observable : str
IoC observable
(`ioc` is also an alias for observable)
ioc_type : str, optional
One of IoCExtract.IoCType, by default None
If none, the IoC type will be inferred
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
Tuple[bool, List[Tuple[str, LookupResult]]]
The result returned as a tuple(bool, list):
bool indicates whether a TI record was found in any provider
list has an entry for each provider result
Lookup an IoC from a single provider
And show the output
result, details = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
print("Positive" if result else "Negative")
# the details is a list (since there could be multiple responses for an IoC)
for provider, detail in details:
print(provider)
detail.summary
print("\nRaw Results")
detail.raw_result_fmt
Positive
OTX
ioc: 38.75.137.9 ( ipv4 )
result: True
{ 'names': ['Underminer EK'],
'pulse_count': 1,
'references': [ [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']],
'tags': [[]]}
reference: https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general
Raw Results
{ 'area_code': 0,
'asn': 'AS63023 GTHost',
'base_indicator': { 'access_reason': '',
'access_type': 'public',
'content': '',
'description': '',
'id': 2127020821,
'indicator': '38.75.137.9',
'title': '',
'type': 'IPv4'},
'charset': 0,
'city': 'Los Angeles',
'city_data': True,
'continent_code': 'NA',
'country_code': 'US',
'country_code3': 'USA',
'country_name': 'United States',
'dma_code': 803,
'flag_title': 'United States',
'flag_url': '/assets/images/flags/us.png',
'indicator': '38.75.137.9',
'latitude': 34.0584,
'longitude': -118.278,
'postal_code': '90017',
'pulse_info': { 'count': 1,
'pulses': [ { 'TLP': 'white',
'adversary': '',
'attack_ids': [],
'author': { 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
'id': '79520',
'is_following': False,
'is_subscribed': False,
'username': 'mattvittitoe'},
'cloned_from': None,
'comment_count': 0,
'created': '2019-07-31T18:01:29.744000',
'description': '',
'downvotes_count': 0,
'export_count': 0,
'follower_count': 0,
'groups': [ { 'id': 614,
'name': 'DCT Security Team'}],
'id': '5d41d77901a2f8c6e9b650e9',
'in_group': True,
'indicator_count': 24,
'indicator_type_counts': { 'FileHash-MD5': 5,
'IPv4': 3,
'URL': 16},
'industries': [],
'is_author': False,
'is_modified': False,
'is_subscribing': None,
'locked': 0,
'malware_families': [],
'modified': '2019-07-31T18:01:29.744000',
'modified_text': '19 days ago ',
'name': 'Underminer EK',
'public': 1,
'pulse_source': 'web',
'references': [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'],
'subscriber_count': 10,
'tags': [],
'targeted_countries': [],
'threat_hunter_scannable': True,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0}],
'references': [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']},
'region': 'CA',
'reputation': 0,
'sections': [ 'general',
'geo',
'reputation',
'url_list',
'passive_dns',
'malware',
'nids_list',
'httpscans'],
'type': 'IPv4',
'type_title': 'IPv4',
'whois': 'http://whois.domaintools.com/38.75.137.9'}
Or convert result to a DataFrame and let pandas do the display work
See result_to_df
result = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
ti_lookup.result_to_df(result).T
OTX |
|
IoC |
38.75.137.9 |
IoCType |
ipv4 |
QuerySubtype |
None |
Result |
True |
Details |
{‘pulse_count’: 1, ‘names’: [‘Underminer EK’], ‘tags’: [[]], ‘references’: [[’https://blog.malw… |
RawResult |
{‘sections’: [‘general’, ‘geo’, ‘reputation’, ‘url_list’, ‘passive_dns’, ‘malware’, ‘nids_list’,… |
Reference |
https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general |
Status |
200 |
# Extract a single field (RawResult) from the dataframe (.iloc[0] is to select the row)
ti_lookup.result_to_df(result)["RawResult"].iloc[0]
{'sections': ['general',
'geo',
'reputation',
'url_list',
'passive_dns',
'malware',
'nids_list',
'httpscans'],
'city': 'Los Angeles',
'area_code': 0,
'pulse_info': {'count': 1,
'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'],
'pulses': [{'indicator_type_counts': {'URL': 16,
'FileHash-MD5': 5,
'IPv4': 3},
'pulse_source': 'web',
'TLP': 'white',
'description': '',
...
Lookup using all primary providers
result = ti_lookup.lookup_ioc(observable="38.75.137.9")
ti_lookup.result_to_df(result)
IoC |
IoCType |
QuerySubtype |
Result |
Details |
RawResult |
Reference |
Status |
|
---|---|---|---|---|---|---|---|---|
OTX |
38.75.137.9 |
ipv4 |
None |
True |
{‘pulse_count’: 1, ‘names’: [‘Underminer EK’], ‘tags’: [[]], ‘references’: [[’https://blog.malw… |
{‘sections’: [‘general’, ‘geo’, ‘reputation’, ‘url_list’, ‘passive_dns’, ‘malware’, ‘nids_list’,… |
https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general |
200 |
VirusTotal |
38.75.137.9 |
ipv4 |
None |
True |
{‘verbose_msg’: ‘IP address in dataset’, ‘response_code’: 1, ‘detected_urls’: [’http://38.75.13… |
{‘asn’: 63023, ‘undetected_urls’: [[’http://38.75.137.9:9088/’, ‘3d5edb0e0bb726e414a9b76dac619c… |
https://www.virustotal.com/vtapi/v2/ip-address/report |
200 |
XForce |
38.75.137.9 |
ipv4 |
None |
True |
{‘score’: 1, ‘cats’: {}, ‘categoryDescriptions’: {}, ‘reason’: ‘Regional Internet Registry’, ‘re… |
{‘ip’: ‘38.75.137.9’, ‘history’: [{‘created’: ‘2012-03-22T07:26:00.000Z’, ‘reason’: ‘Regional In… |
https://api.xforce.ibmcloud.com/ipr/38.75.137.9 |
200 |
AzSTI |
38.75.137.9 |
ipv4 |
None |
False |
0 rows returned. |
None |
None |
-1 |
GreyNoise |
38.75.137.9 |
ipv4 |
None |
False |
Not found. |
<Response [404]> |
404 |
Provider Usage
This shows the supported IoC Types.
In some cases an IoC type will also support special types of sub-query such as geo-ip and passive-dns
See provider_usage
ti_lookup.provider_usage()
Primary providers
-----------------
Provider class: OTX
AlientVault OTX Lookup. Supported query types:
ioc_type=dns
ioc_type=dns, ioc_query_type=geo
ioc_type=dns, ioc_query_type=passivedns
ioc_type=file_hash
ioc_type=hostname
ioc_type=ipv4
ioc_type=ipv4, ioc_query_type=geo
ioc_type=ipv4, ioc_query_type=passivedns
ioc_type=ipv6
ioc_type=ipv6, ioc_query_type=geo
ioc_type=ipv6, ioc_query_type=passivedns
ioc_type=md5_hash
ioc_type=sha1_hash
ioc_type=sha256_hash
ioc_type=url
Provider class: VirusTotal
VirusTotal Lookup. Supported query types:
ioc_type=dns
ioc_type=file_hash
ioc_type=ipv4
ioc_type=md5_hash
ioc_type=sha1_hash
ioc_type=sha256_hash
ioc_type=url
Provider class: XForce
IBM XForce Lookup. Supported query types:
ioc_type=dns, ioc_query_type=info
ioc_type=dns, ioc_query_type=passivedns
ioc_type=dns, ioc_query_type=whois
ioc_type=file_hash
ioc_type=hostname, ioc_query_type=whois
ioc_type=ipv4
ioc_type=ipv4, ioc_query_type=malware
ioc_type=ipv4, ioc_query_type=passivedns
ioc_type=ipv4, ioc_query_type=rep
ioc_type=ipv4, ioc_query_type=whois
ioc_type=ipv6
ioc_type=ipv6, ioc_query_type=malware
ioc_type=ipv6, ioc_query_type=passivedns
ioc_type=ipv6, ioc_query_type=rep
ioc_type=ipv6, ioc_query_type=whois
ioc_type=md5_hash
ioc_type=sha1_hash
ioc_type=sha256_hash
ioc_type=url
ioc_type=url, ioc_query_type=malware
Provider class: AzSTI
Azure Sentinel TI provider class. Supported query types:
ioc_type=dns
ioc_type=file_hash
ioc_type=hostname
ioc_type=ipv4
ioc_type=ipv6
ioc_type=linux_path
ioc_type=md5_hash
ioc_type=sha1_hash
ioc_type=sha256_hash
ioc_type=url
ioc_type=windows_path
Provider class: GreyNoise
GreyNoise Lookup. Supported query types:
ioc_type=ipv4
ioc_type=ipv4, ioc_query_type=full
ioc_type=ipv4, ioc_query_type=quick
Secondary providers
-------------------
none
Use to do a passive DNS lookup
result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="passivedns", providers=["XForce"])
result
(True, [('XForce', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', query_subtype='passivedns', result=True, details={'records': 1}, raw_result={'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}, reference='https://api.xforce.ibmcloud.com/resolve/38.75.137.9', status=200))])
Inferring IoC type vs specifying explicitly
If you do a lookup without specifying a type, TILookup will try to infer the type by matching regexes. There are patterns for all supported types but there are some caveats:
The match is not 100% foolproof - e.g. some URLs and hash types may be misidentified.
The inference adds an overhead to each lookup.
If you know the type that you want to look up, it is always better to
explicitly include it. - For single IoC lookup, use the ioc_type
parameter. - For multiple IoC lookups (see below), supply either: - a
DataFrame with a column that specifies the type for each entry - a
dictionary of the form {ioc_observable: ioc_type}
Looking up Multiple IoCs
See lookup_iocs
lookup_iocs
Signature:
ti_lookup.lookup_iocs(
data: Union[pandas.core.frame.DataFrame, Mapping[str, str], Iterable[str]],
obs_col: str = None,
ioc_type_col: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> pandas.core.frame.DataFrame
Lookup a collection of IoCs.
Parameters
----------
data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]]
Data input in one of three formats:
1. Pandas dataframe (you must supply the column name in
`obs_col` parameter)
2. Mapping (e.g. a dict) of [observable, IoCType]
3. Iterable of observables - IoCTypes will be inferred
obs_col : str, optional
DataFrame column to use for observables, by default None
ioc_type_col : str, optional
DataFrame column to use for IoCTypes, by default None
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
pd.DataFrame
DataFrame of results
# View the docstring (as above)
ti_lookup.lookup_iocs?
ioc_ips = [
"185.92.220.35",
"213.159.214.86",
"77.222.54.202",
"91.219.29.81",
"193.9.28.254",
"89.108.83.196",
"91.219.28.44",
"188.127.231.124",
"192.42.116.41",
"91.219.31.18",
"46.4.239.76",
"188.166.168.250",
"195.154.241.208",
"51.255.172.55",
"93.170.169.52",
"104.215.148.63",
"13.77.161.179",
"40.76.4.15",
"40.112.72.205",
"40.113.200.201",
]
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI")
IoC |
IoCType |
QuerySubtype |
Reference |
Result |
Status |
Details |
RawResult |
Provider |
|
---|---|---|---|---|---|---|---|---|---|
0 |
213.159.214.86 |
ipv4 |
None |
ThreatIntelligenceIndicator | whe… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘0164ADB4A6CB7A79FBAE7BE90A43050B090A18364E3855048AC86B9DA5E0A92B’, ‘TimeGenerat… |
AzSTI |
1 |
40.113.200.201 |
ipv4 |
None |
ThreatIntelligenceIndicator | whe… |
False |
-1.0 |
0 rows returned. |
NaN |
AzSTI |
2 |
91.219.29.81 |
ipv4 |
None |
ThreatIntelligenceIndicator | whe… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘3F458D91A21866C9037B99D997379A6906573766C0C2F8FB45327A6A15676A0D’, ‘TimeGenerat… |
AzSTI |
3 |
89.108.83.196 |
ipv4 |
None |
ThreatIntelligenceIndicator | whe… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘C3CA82D5B30A34F4BD6188C9DCFAD9E46D3C0CC45CC4FD969DA3A398DC34B1AE’, ‘TimeGenerat… |
AzSTI |
4 |
192.42.116.41 |
ipv4 |
None |
ThreatIntelligenceIndicator | whe… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘2F321C9D2593B6EF59DEB64B6CB209F375529C429F0DF463D639784E7353AA5D’, ‘TimeGenerat… |
AzSTI |
Multiple IoCs using all providers
Output sorted by IoC
ioc_urls = [
"http://cheapshirts.us/zVnMrG.php",
"http://chinasymbolic.com/i9jnrc",
"http://cetidawabi.com/468fd",
"http://append.pl/srh9xsz",
"http://aiccard.co.th/dvja1te",
"http://ajaraheritage.ge/g7cberv",
"http://cic-integration.com/hjy93JNBasdas",
"https://google.com", # benign
"https://microsoft.com", # benign
"https://python.org", # benign
]
results = ti_lookup.lookup_iocs(data=ioc_urls)
results.sort_values("IoC")
IoC |
IoCType |
QuerySubtype |
Result |
Details |
RawResult |
Reference |
Provider |
Status |
|
---|---|---|---|---|---|---|---|---|---|
0 |
http://aiccard.co.th/dvja1te |
url |
None |
True |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘FAE39C007D6554822504A1E0BDFD788E27DDC748ED63B258651DE52E4FA6D511’, ‘TimeGenerat… |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-21T17:30:41.900764Z) | w… |
AzSTI |
0.0 |
4 |
http://aiccard.co.th/dvja1te |
url |
None |
True |
{‘cats’: None, ‘categoryDescriptions’: None} |
{‘result’: {‘url’: ‘aiccard.co.th’, ‘cats’: {}, ‘score’: None, ‘categoryDescriptions’: {}}, ‘ass… |
https://api.xforce.ibmcloud.com/url/http://aiccard.co.th/dvja1te |
XForce |
NaN |
4 |
http://aiccard.co.th/dvja1te |
url |
None |
True |
{‘pulse_count’: 3, ‘names’: [‘Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)’, … |
{‘indicator’: ‘http://aiccard.co.th/dvja1te’, ‘alexa’: ‘http://www.alexa.com/siteinfo/aiccard.c… |
https://otx.alienvault.com/api/v1/indicators/url/http://aiccard.co.th/dvja1te/general |
OTX |
NaN |
4 |
http://aiccard.co.th/dvja1te |
url |
None |
False |
No response from provider. |
<Response [403]> |
https://www.virustotal.com/vtapi/v2/url/report |
VirusTotal |
NaN |
5 |
http://ajaraheritage.ge/g7cberv |
url |
None |
True |
{‘cats’: None, ‘categoryDescriptions’: None} |
{‘result’: {‘url’: ‘ajaraheritage.ge’, ‘cats’: {}, ‘score’: None, ‘categoryDescriptions’: {}}, ‘… |
https://api.xforce.ibmcloud.com/url/http://ajaraheritage.ge/g7cberv |
XForce |
NaN |
Note
the URLs in the previous example have been altered to prevent inadvertent navigation to them.
Browsing and Selecting TI Results
To make it easier to walk through the returned results msticpy has a browser. This shows you the TI results aggregated by the IoC value (e.g. an individual IP Address or URL) for all providers.
For each provider that returns a result for an IoC, the summarized details will be shown in a table below the browse list.
Click on Raw results from provider...
to see all returned data.
Note
the reference URL may not work if you have not authenticated to the TI service.
The value of the selected IoC entry is available as ti_selector.value. You can match this back to the original results DataFrame as follows:
results[results["Ioc"] == ti_selector.value[0]]
Specifying Time Ranges
Some providers (currently only AzSTI) support time ranges so that you can specify specific periods to search for.
If a provider does not support time ranges, the parameters will be ignored
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="hour", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
# Using this data range returned no results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end).head()
IoC |
IoCType |
QuerySubtype |
Reference |
Result |
Details |
Status |
Provider |
|
---|---|---|---|---|---|---|---|---|
0 1 2 3 4 |
213.159.214.86 40.113.200.201 91.219.29.81 89.108.83.196 192.42.116.41 |
ipv4 ipv4 ipv4 ipv4 ipv4 |
None None None None None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-08-04T00:00:00Z) | where Ti… ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-08-04T00:00:00Z) | where Ti… ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-08-04T00:00:00Z) | where Ti… ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-08-04T00:00:00Z) | where Ti… ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-08-04T00:00:00Z) | where Ti… |
False False False False False |
0 rows returned. 0 rows returned. 0 rows returned. 0 rows returned. 0 rows returned. |
-1 -1 -1 -1 -1 |
AzSTI AzSTI AzSTI AzSTI AzSTI |
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="day", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
# Using a wider ranges produces results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end)
IoC |
IoCType |
QuerySubtype |
Reference |
Result |
Status |
Details |
RawResult |
Provider |
|
---|---|---|---|---|---|---|---|---|---|
0 |
213.159.214.86 |
ipv4 |
None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-12T00:00:00Z) | where Ti… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘0164ADB4A6CB7A79FBAE7BE90A43050B090A18364E3855048AC86B9DA5E0A92B’, ‘TimeGenerat… |
AzSTI |
1 |
40.113.200.201 |
ipv4 |
None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-12T00:00:00Z) | where Ti… |
False |
-1.0 |
0 rows returned. |
NaN |
AzSTI |
2 |
91.219.29.81 |
ipv4 |
None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-12T00:00:00Z) | where Ti… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘3F458D91A21866C9037B99D997379A6906573766C0C2F8FB45327A6A15676A0D’, ‘TimeGenerat… |
AzSTI |
3 |
89.108.83.196 |
ipv4 |
None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-12T00:00:00Z) | where Ti… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘C3CA82D5B30A34F4BD6188C9DCFAD9E46D3C0CC45CC4FD969DA3A398DC34B1AE’, ‘TimeGenerat… |
AzSTI |
4 |
192.42.116.41 |
ipv4 |
None |
ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2019-07-12T00:00:00Z) | where Ti… |
True |
0.0 |
{‘Action’: ‘alert’, ‘ThreatType’: ‘Malware’, ‘ThreatSeverity’: nan, ‘Active’: True, ‘Description… |
{‘IndicatorId’: ‘2F321C9D2593B6EF59DEB64B6CB209F375529C429F0DF463D639784E7353AA5D’, ‘TimeGenerat… |
AzSTI |