IoC Extraction
==============


This class allows you to extract IoC patterns from a string or a
DataFrame. Several patterns are built in to the class and you can
override these or supply new ones.


.. code:: ipython3

    # Imports
    import sys
    MIN_REQ_PYTHON = (3,6)
    if sys.version_info < MIN_REQ_PYTHON:
        print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')
        print('or later is selected as the active kernel.')
        sys.exit("Python %s.%s or later is required.\n" % MIN_REQ_PYTHON)

    from IPython.display import display, HTML
    import matplotlib.pyplot as plt
    import seaborn as sns
    sns.set()
    import pandas as pd
    pd.set_option('display.max_rows', 500)
    pd.set_option('display.max_columns', 50)
    pd.set_option('display.max_colwidth', 100)

.. code:: ipython3

    # Load test data
    process_tree = pd.read_csv('data/process_tree.csv')
    process_tree[['CommandLine']].head()


.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>CommandLine</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>.\ftp  -s:C:\RECYCLER\xxppyy.exe</td>
        </tr>
        <tr>
          <th>1</th>
          <td>.\reg  not /domain:everything that /sid:shines is /krbtgt:golden !</td>
        </tr>
        <tr>
          <th>2</th>
          <td>cmd  /c "systeminfo &amp;&amp; systeminfo"</td>
        </tr>
        <tr>
          <th>3</th>
          <td>.\rundll32  /C 42424.exe</td>
        </tr>
        <tr>
          <th>4</th>
          <td>.\rundll32  /C c:\users\MSTICAdmin\42424.exe</td>
        </tr>
      </tbody>
    </table>
    </div>
    <br/>


Looking for IoC in a String
---------------------------

Just pass the string as a parameter to the extract() method.


Get a commandline from our data set.

.. code:: ipython3

    # get a commandline from our data set
    cmdline = process_tree['CommandLine'].loc[78]
    cmdline


.. parsed-literal::

    'netsh  start capture=yes IPv4.Address=1.2.3.4 tracefile=C:\\\\Users\\\\user\\\\AppData\\\\Local\\\\Temp\\\\bzzzzzz.txt'


Instantiate an IoCExtract instance and pass the string to the extract() method.

.. code:: ipython3

    # Instantiate an IoCExtract object
    from msticpy.transform import IoCExtract
    ioc_extractor = IoCExtract()

    # any IoCs in the string?
    iocs_found = ioc_extractor.extract(cmdline)

    if iocs_found:
        print('\nPotential IoCs found in alert process:')
        display(iocs_found)


.. parsed-literal::


    Potential IoCs found in alert process:


.. parsed-literal::

    defaultdict(set,
                {'ipv4': {'1.2.3.4'},
                 'windows_path': {'C:\\\\Users\\\\user\\\\AppData\\\\Local\\\\Temp\\\\bzzzzzz.txt'}})


The following IoC patterns are searched for:

* ipv4
* ipv6
* dns
* url
* windows_path
* linux_path
* md5_hash
* sha1_hash
* sha256_hash


Using a DataFrame as Input
--------------------------

You can use the ``data=`` parameter to
IoCExtract.extract() to pass a DataFrame. Use the ``columns``
parameter to specify which column or columns that you want to search.

.. note:: When searching a DataFrame
    the following types are not included in the search by default
    ``windows_path`` and ``linux_path`` because of the likely high volume
    of results and number of false positive matches. You can
    include them by specifing ``include_paths=True`` as a parameter to
    ``extract()``.

    You can also use the ``ioc_types`` parameter to explicitly list the
    ioc_types that you want to search for. This should be a list of
    strings of valid types.
    See :py:meth:`ioc_types<msticpy.transform.ioc_extractor.IoCExtract.ioc_types>`


.. code:: ipython3

    ioc_extractor = IoCExtract()
    ioc_df = ioc_extractor.extract(data=process_tree, columns=['CommandLine'])
    if len(ioc_df):
        display(HTML("<h3>IoC patterns found in process tree.</h3>"))
        display(ioc_df)


.. raw:: html

    <h3>IoC patterns found in process tree.</h3>


.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>IoCType</th>
          <th>Observable</th>
          <th>SourceIndex</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>48</th>
          <td>windows_path</td>
          <td>.\powershell</td>
          <td>36</td>
        </tr>
        <tr>
          <th>49</th>
          <td>url</td>
          <td>http://somedomain/best-kitten-names-1.jpg'</td>
          <td>37</td>
        </tr>
        <tr>
          <th>53</th>
          <td>windows_path</td>
          <td>.\pOWErS^H^ElL^.eX^e^</td>
          <td>37</td>
        </tr>
        <tr>
          <th>58</th>
          <td>md5_hash</td>
          <td>81ed03caf6901e444c72ac67d192fb9c</td>
          <td>44</td>
        </tr>
        <tr>
          <th>59</th>
          <td>url</td>
          <td>http://badguyserver/pwnme"</td>
          <td>46</td>
        </tr>
        <tr>
          <th>68</th>
          <td>windows_path</td>
          <td>.\reg  query add mscfile\\\\open</td>
          <td>59</td>
        </tr>
        <tr>
          <th>72</th>
          <td>windows_path</td>
          <td>\system\CurrentControlSet\Control\Terminal</td>
          <td>63</td>
        </tr>
        <tr>
          <th>92</th>
          <td>ipv4</td>
          <td>1.2.3.4</td>
          <td>78</td>
        </tr>
        <tr>
          <th>108</th>
          <td>ipv4</td>
          <td>127.0.0.1</td>
          <td>102</td>
        </tr>
        <tr>
          <th>109</th>
          <td>url</td>
          <td>http://127.0.0.1/</td>
          <td>102</td>
        </tr>
        <tr>
          <th>110</th>
          <td>windows_path</td>
          <td>\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost\MyNastySvcHostConfig</td>
          <td>103</td>
        </tr>
      </tbody>
    </table>
    </div>
    <br/>


IoCExtractor API
----------------

See :py:class:`IoCExtract<msticpy.transform.ioc_extractor.IoCExtract>`
and See :py:func:`IoCExtract<msticpy.transform.ioc_extractor.IoCExtract.extract>`


Predefined Regex Patterns
-------------------------

.. code:: ipython3

    from html import escape
    extractor = IoCExtract()

    for ioc_type, pattern in extractor.ioc_types.items():
        esc_pattern = escape(pattern.comp_regex.pattern)
        display(HTML(f'<b>{ioc_type}</b>'))
        display(HTML(f'<div style="margin-left:20px"><pre>{esc_pattern}</pre></div>)'))


.. raw:: html

    <table border="1">
      <thead>
        <tr style="text-align: right;">
          <th>IoCType</th>
          <th>Regex</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>ipv4</td>
          <td><pre>(?P&lt;ipaddress&gt;(?:[0-9]{1,3}\\.){3}[0-9]{1,3})</pre></td>
        </tr>
        <tr>
          <td>ipv6</td>
          <td><pre>(?&lt;![:.\\w])(?:[A-F0-9]{1,4}:){7}[A-F0-9]{1,4}(?![:.\\w])</pre></td>
        </tr>
        <tr>
          <td>dns</td>
          <td><pre>((?=[a-z0-9-]{1,63}\\.)[a-z0-9]+(-[a-z0-9]+)*\\.){2,}[a-z]{2,63}</pre></td>
        </tr>
        <tr>
          <td>url</td>
          <td>
            <pre>(?P&lt;protocol&gt;(https?|ftp|telnet|ldap|file)://)&#10;</pre>
            <pre>(?P&lt;userinfo&gt;([a-z0-9-._~!$&amp;\\'()*+,;=:]|%[0-9A-F]{2})*@)?&#10;</pre>
            <pre>(?P&lt;host&gt;([a-z0-9-._~!$&amp;\\'()*+,;=]|%[0-9A-F]{2})*)</pre>
          </td>
        </tr>
        <tr>
          <td>windows_path</td>
          <td>
            <pre>&#13;&#10;(?P&lt;root&gt;[a-z]:|\\\\\\\\[a-z0-9_.$-]+||[.]+)&#10;</pre>
            <pre>(?P&lt;folder&gt;\\\\(?:[^\\/:*?&quot;\\\'&lt;&gt;|\\r\\n]+\\\\)*)&#10;></pre>
            <pre>(?P&lt;file&gt;[^\\\\/*?&quot;&quot;&lt;&gt;|\\r\\n ]+)</pre>
          </td>
        </tr>
        <tr>
          <td>linux_path</td>
          <td>
            <pre>(?P&lt;root&gt;/+||[.]+)&#10;</pre>
            <pre>(?P&lt;folder&gt;/(?:[^\\\\/:*?&lt;&gt;|\\r\\n]+/)*)&#10;</pre>
            <pre>(?P&lt;file&gt;[^/\\0&lt;&gt;|\\r\\n ]+)</pre>
          </td>
        <tr>
          <td>md5_hash</td>
          <td><pre>(?:^|[^A-Fa-f0-9])(?P&lt;hash&gt;[A-Fa-f0-9]{32})(?:$|[^A-Fa-f0-9])</pre></td>
        </tr>
        <tr>
          <td>sha1_hash</td>
          <td><pre>(?:^|[^A-Fa-f0-9])(?P&lt;hash&gt;[A-Fa-f0-9]{40})(?:$|[^A-Fa-f0-9])</pre></td>
        </tr>
          <tr>
          <td>ipv6</td>
          <td><pre>(?:^|[^A-Fa-f0-9])(?P&lt;hash&gt;[A-Fa-f0-9]{64})(?:$|[^A-Fa-f0-9])</pre></td>
        </tr>
      </table>
      <br>


Adding your own pattern(s)
--------------------------


See :py:func:`add_ioc_type<msticpy.transform.ioc_extractor.IoCExtract.add_ioc_type>`


Add an IoC type and regular expression to use to the built-in set.

.. warning:: Adding an ioc_type that exists in the internal set will overwrite that item

Regular expressions are compiled with re.I | re.X | re.M (Ignore case, Verbose
and MultiLine)

add_ioc_type parameters:

-  ioc_type{str} - a unique name for the IoC type
-  ioc_regex{str} - a regular expression used to search for the type


.. code:: ipython3

    import re
    rcomp = re.compile(r'(?P<pipe>\\\\\.\\pipe\\[^\s\\]+)')

.. code:: ipython3

    extractor.add_ioc_type(ioc_type='win_named_pipe', ioc_regex=r'(?P<pipe>\\\\\.\\pipe\\[^\s\\]+)')

    # Check that it added ok
    print(extractor.ioc_types['win_named_pipe'])

    # Use it in our data set
    ioc_extractor.extract(data=process_tree, columns=['CommandLine']).query('IoCType == \'win_named_pipe\'')


.. parsed-literal::

    IoCPattern(ioc_type='win_named_pipe', comp_regex=re.compile('(?P<pipe>\\\\\\\\\\.\\\\pipe\\\\[^\\s\\\\]+)', re.IGNORECASE|re.MULTILINE|re.VERBOSE), priority=0)


.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>IoCType</th>
          <th>Observable</th>
          <th>SourceIndex</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>116</th>
          <td>win_named_pipe</td>
          <td>\\.\pipe\blahtest"</td>
          <td>107</td>
        </tr>
      </tbody>
    </table>
    </div>
    <br>

extract_df()
~~~~~~~~~~~~

``extract_df`` functions identically to ``extract`` with a ``data``
parameter. It may be more convenient to use this when you know that your
input is a DataFrame

.. code:: ipython3

    ioc_extractor.extract_df(process_tree, columns=['NewProcessName', 'CommandLine']).head(10)


Merging output with source data
-------------------------------

The SourceIndex column allows you to merge the
results with the input DataFrame Where an input row has multiple IoC
matches the output of this merge will result in duplicate rows from the
input (one per IoC match). The previous index is preserved in the second
column (and in the SourceIndex column).

Note: you will need to set the type of the SourceIndex column. In the
example below case we are matching with the default numeric index so we
force the type to be numeric. In cases where you are using an index of a
different dtype you will need to convert the SourceIndex (dtype=object)
to match the type of your index column.

.. code:: ipython3

    input_df = data=process_tree.head(20)
    output_df = ioc_extractor.extract(data=input_df, columns=['NewProcessName', 'CommandLine'])
    # set the type of the SourceIndex column. In this case we are matching with the default numeric index.
    output_df['SourceIndex'] = pd.to_numeric(output_df['SourceIndex'])
    merged_df = pd.merge(left=input_df, right=output_df, how='outer', left_index=True, right_on='SourceIndex')
    merged_df.head()


====  ====================================  ==========================  =========  =======================  ===============  ============================================  =================  ===================  ================  ==============  ===================================  ====================  ===========  ==================================================================  ===========================  ===============  ====================================  =======================  ==========  =======  ============  ===============  =========  ============  =============
  ..  TenantId                              Account                       EventID  TimeGenerated            Computer         SubjectUserSid                                SubjectUserName    SubjectDomainName    SubjectLogonId    NewProcessId    NewProcessName                       TokenElevationType    ProcessId    CommandLine                                                         ParentProcessName            TargetLogonId    SourceComputerId                      TimeCreatedUtc           NodeRole      Level    ProcessId1    NewProcessId1    IoCType    Observable    SourceIndex
====  ====================================  ==========================  =========  =======================  ===============  ============================================  =================  ===================  ================  ==============  ===================================  ====================  ===========  ==================================================================  ===========================  ===============  ====================================  =======================  ==========  =======  ============  ===============  =========  ============  =============
   0  802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\MSTICAdmin       4688  2019-01-15 05:15:15.677  MSTICAlertsWin1  S-1-5-21-996632719-2361334927-4038480536-500  MSTICAdmin         MSTICAlertsWin1      0xfaac27          0x1580          C:\Diagnostics\UserTmp\ftp.exe       %%1936                0xbc8        .\ftp  -s:C:\RECYCLER\xxppyy.exe                                    C:\Windows\System32\cmd.exe  0x0              46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:15.677  source            0           nan              nan        nan           nan              0
   1  802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\MSTICAdmin       4688  2019-01-15 05:15:16.167  MSTICAlertsWin1  S-1-5-21-996632719-2361334927-4038480536-500  MSTICAdmin         MSTICAlertsWin1      0xfaac27          0x16fc          C:\Diagnostics\UserTmp\reg.exe       %%1936                0xbc8        .\reg  not /domain:everything that /sid:shines is /krbtgt:golden !  C:\Windows\System32\cmd.exe  0x0              46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:16.167  sibling           1           nan              nan        nan           nan              1
   2  802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\MSTICAdmin       4688  2019-01-15 05:15:16.277  MSTICAlertsWin1  S-1-5-21-996632719-2361334927-4038480536-500  MSTICAdmin         MSTICAlertsWin1      0xfaac27          0x1700          C:\Diagnostics\UserTmp\cmd.exe       %%1936                0xbc8        cmd  /c "systeminfo && systeminfo"                                  C:\Windows\System32\cmd.exe  0x0              46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:16.277  sibling           1           nan              nan        nan           nan              2
   3  802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\MSTICAdmin       4688  2019-01-15 05:15:16.340  MSTICAlertsWin1  S-1-5-21-996632719-2361334927-4038480536-500  MSTICAdmin         MSTICAlertsWin1      0xfaac27          0x1728          C:\Diagnostics\UserTmp\rundll32.exe  %%1936                0xbc8        .\rundll32  /C 42424.exe                                            C:\Windows\System32\cmd.exe  0x0              46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:16.340  sibling           1           nan              nan        nan           nan              3
   4  802d39e1-9d70-404d-832c-2de5e2478eda  MSTICAlertsWin1\MSTICAdmin       4688  2019-01-15 05:15:16.400  MSTICAlertsWin1  S-1-5-21-996632719-2361334927-4038480536-500  MSTICAdmin         MSTICAlertsWin1      0xfaac27          0x175c          C:\Diagnostics\UserTmp\rundll32.exe  %%1936                0xbc8        .\rundll32  /C c:\users\MSTICAdmin\42424.exe                        C:\Windows\System32\cmd.exe  0x0              46fe7078-61bb-4bed-9430-7ac01d91c273  2019-01-15 05:15:16.400  sibling           1           nan              nan        nan           nan              4
====  ====================================  ==========================  =========  =======================  ===============  ============================================  =================  ===================  ================  ==============  ===================================  ====================  ===========  ==================================================================  ===========================  ===============  ====================================  =======================  ==========  =======  ============  ===============  =========  ============  =============


IPython magic
-------------

You can use the line magic ``%ioc`` or cell magic ``%%ioc`` to extract
IoCs from text pasted directly into a cell

The ioc magic supports the following options:

::

   --out OUT, -o OUT
       The variable to return the results in the variable `OUT`
       Note: the output variable is a dictionary iocs grouped by IoC Type
   --ioc_types IOC_TYPES, -i IOC_TYPES
       The types of IoC to search for (comma-separated string)

.. code:: ipython3

    %%ioc --out ioc_capture
    netsh  start capture=yes IPv4.Address=1.2.3.4 tracefile=C:\Users\user\AppData\Local\Temp\bzzzzzz.txt
    hostname	customers-service.ddns.net		Feb 5, 2020, 2:20:35 PM		7
    URL	\https://two-step-checkup.site/securemail/secureLogin/challenge/url?ucode=d50a3eb1-9a6b-45a8-8389-d5203bbddaa1&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;service=mailservice&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;type=password		Feb 5, 2020, 2:20:35 PM		1
    hostname	mobile.phonechallenges-submit.site		Feb 5, 2020, 2:20:35 PM		8
    hostname	youtube.service-activity-checkup.site		Feb 5, 2020, 2:20:35 PM		8
    hostname	www.drive-accounts.com		Feb 5, 2020, 2:20:35 PM		7
    hostname	google.drive-accounts.com		Feb 5, 2020, 2:20:35 PM		7
    domain	niaconucil.org		Feb 5, 2020, 2:20:35 PM		11
    domain	isis-online.net		Feb 5, 2020, 2:20:35 PM		11
    domain	bahaius.info		Feb 5, 2020, 2:20:35 PM		11
    domain	w3-schools.org		Feb 5, 2020, 2:20:35 PM		12
    domain	system-services.site		Feb 5, 2020, 2:20:35 PM		11
    domain	accounts-drive.com		Feb 5, 2020, 2:20:35 PM		8
    domain	drive-accounts.com		Feb 5, 2020, 2:20:35 PM		10
    domain	service-issues.site		Feb 5, 2020, 2:20:35 PM		8
    domain	two-step-checkup.site		Feb 5, 2020, 2:20:35 PM		8
    domain	customers-activities.site		Feb 5, 2020, 2:20:35 PM		11
    domain	seisolarpros.org		Feb 5, 2020, 2:20:35 PM		11
    domain	yah00.site		Feb 5, 2020, 2:20:35 PM		4
    domain	skynevvs.com		Feb 5, 2020, 2:20:35 PM		11
    domain	recovery-options.site		Feb 5, 2020, 2:20:35 PM		4
    domain	malcolmrifkind.site		Feb 5, 2020, 2:20:35 PM		8
    domain	instagram-com.site		Feb 5, 2020, 2:20:35 PM		8
    domain	leslettrespersanes.net		Feb 5, 2020, 2:20:35 PM		11
    domain	software-updating-managers.site		Feb 5, 2020, 2:20:35 PM		8
    domain	cpanel-services.site		Feb 5, 2020, 2:20:35 PM		8
    domain	service-activity-checkup.site		Feb 5, 2020, 2:20:35 PM		7
    domain	inztaqram.ga		Feb 5, 2020, 2:20:35 PM		8
    domain	unirsd.com		Feb 5, 2020, 2:20:35 PM		8
    domain	phonechallenges-submit.site		Feb 5, 2020, 2:20:35 PM		7
    domain	acconut-verify.com		Feb 5, 2020, 2:20:35 PM		11
    domain	finance-usbnc.info		Feb 5, 2020, 2:20:35 PM		8
    FileHash-MD5	542128ab98bda5ea139b169200a50bce		Feb 5, 2020, 2:20:35 PM		3
    FileHash-MD5	3d67ce57aab4f7f917cf87c724ed7dab		Feb 5, 2020, 2:20:35 PM		3
    hostname	x09live-ix3b.account-profile-users.info		Feb 6, 2020, 2:56:07 PM		0
    hostname	www.phonechallenges-submit.site		Feb 6, 2020, 2:56:07 PM


.. parsed-literal::

    [('ipv4', ['1.2.3.4']),
     ('dns',
      ['malcolmrifkind.site',
       'w3-schools.org',
       'niaconucil.org',
       'software-updating-managers.site',
       'isis-online.net',
       'accounts-drive.com',
       'cpanel-services.site',
       'service-activity-checkup.site',
       'service-issues.site',
       'recovery-options.site',
       'instagram-com.site',
       'mobile.phonechallenges-submit.site',
       'youtube.service-activity-checkup.site',
       'google.drive-accounts.com',
       'phonechallenges-submit.site',
       'drive-accounts.com',
       'www.phonechallenges-submit.site',
       'yah00.site',
       'seisolarpros.org',
       'customers-activities.site',
       'bahaius.info',
       'system-services.site',
       'two-step-checkup.site',
       'x09live-ix3b.account-profile-users.info',
       'customers-service.ddns.net',
       'leslettrespersanes.net',
       'www.drive-accounts.com',
       'acconut-verify.com',
       'finance-usbnc.info',
       'unirsd.com',
       'skynevvs.com',
       'inztaqram.ga']),
     ('url',
      ['\https://two-step-checkup.site/securemail/secureLogin/challenge/url?ucode=d50a3eb1-9a6b-45a8-8389-d5203bbddaa1&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;service=mailservice&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;type=password']),
     ('windows_path', ['C:\\Users\\user\\AppData\\Local\\Temp\\bzzzzzz.txt']),
     ('linux_path',
      ['//two-step-checkup.site/securemail/secureLogin/challenge/url?ucode=d50a3eb1-9a6b-45a8-8389-d5203bbddaa1&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;service=mailservice&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;type=password\t\tFeb']),
     ('md5_hash',
      ['3d67ce57aab4f7f917cf87c724ed7dab', '542128ab98bda5ea139b169200a50bce'])]


.. code:: ipython3

    %%ioc --ioc_types "ipv4, ipv6, linux_path, md5_hash"
    netsh  start capture=yes IPv4.Address=1.2.3.4 tracefile=C:\Users\user\AppData\Local\Temp\bzzzzzz.txt
    tracefile2=/usr/localbzzzzzz.sh
    hostname	customers-service.ddns.net		Feb 5, 2020, 2:20:35 PM		7
    URL	\https://two-step-checkup.site/securemail/secureLogin/challenge/url?ucode=d50a3eb1-9a6b-45a8-8389-d5203bbddaa1&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;service=mailservice&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;type=password		Feb 5, 2020, 2:20:35 PM		1
    hostname	mobile.phonechallenges-submit.site		Feb 5, 2020, 2:20:35 PM		8
    hostname	youtube.service-activity-checkup.site		Feb 5, 2020, 2:20:35 PM		8
    hostname	www.drive-accounts.com		Feb 5, 2020, 2:20:35 PM		7
    hostname	google.drive-accounts.com		Feb 5, 2020, 2:20:35 PM		7
    domain	niaconucil.org		Feb 5, 2020, 2:20:35 PM		11
    domain	isis-online.net		Feb 5, 2020, 2:20:35 PM		11
    domain	bahaius.info		Feb 5, 2020, 2:20:35 PM		11
    domain	w3-schools.org		Feb 5, 2020, 2:20:35 PM		12
    domain	system-services.site		Feb 5, 2020, 2:20:35 PM		11
    domain	accounts-drive.com		Feb 5, 2020, 2:20:35 PM		8
    domain	drive-accounts.com		Feb 5, 2020, 2:20:35 PM		10
    domain	service-issues.site		Feb 5, 2020, 2:20:35 PM		8
    domain	two-step-checkup.site		Feb 5, 2020, 2:20:35 PM		8
    domain	customers-activities.site		Feb 5, 2020, 2:20:35 PM		11
    domain	seisolarpros.org		Feb 5, 2020, 2:20:35 PM		11
    domain	yah00.site		Feb 5, 2020, 2:20:35 PM		4
    domain	skynevvs.com		Feb 5, 2020, 2:20:35 PM		11
    domain	recovery-options.site		Feb 5, 2020, 2:20:35 PM		4
    domain	malcolmrifkind.site		Feb 5, 2020, 2:20:35 PM		8
    domain	instagram-com.site		Feb 5, 2020, 2:20:35 PM		8
    domain	leslettrespersanes.net		Feb 5, 2020, 2:20:35 PM		11
    domain	software-updating-managers.site		Feb 5, 2020, 2:20:35 PM		8
    domain	cpanel-services.site		Feb 5, 2020, 2:20:35 PM		8
    domain	service-activity-checkup.site		Feb 5, 2020, 2:20:35 PM		7
    domain	inztaqram.ga		Feb 5, 2020, 2:20:35 PM		8
    domain	unirsd.com		Feb 5, 2020, 2:20:35 PM		8
    domain	phonechallenges-submit.site		Feb 5, 2020, 2:20:35 PM		7
    domain	acconut-verify.com		Feb 5, 2020, 2:20:35 PM		11
    domain	finance-usbnc.info		Feb 5, 2020, 2:20:35 PM		8
    FileHash-MD5	542128ab98bda5ea139b169200a50bce		Feb 5, 2020, 2:20:35 PM		3
    FileHash-MD5	3d67ce57aab4f7f917cf87c724ed7dab		Feb 5, 2020, 2:20:35 PM		3
    hostname	x09live-ix3b.account-profile-users.info		Feb 6, 2020, 2:56:07 PM		0
    hostname	www.phonechallenges-submit.site		Feb 6, 2020, 2:56:07 PM


.. parsed-literal::

    [('ipv4', ['1.2.3.4']),
     ('linux_path',
      ['//two-step-checkup.site/securemail/secureLogin/challenge/url?ucode=d50a3eb1-9a6b-45a8-8389-d5203bbddaa1&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;service=mailservice&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;type=password\t\tFeb',
       '/usr/localbzzzzzz.sh']),
     ('md5_hash',
      ['3d67ce57aab4f7f917cf87c724ed7dab', '542128ab98bda5ea139b169200a50bce'])]


Pandas Extension
----------------

The decoding functionality is also available in a pandas extension
``mp_ioc``. This supports a single method ``extract()``.

This supports the same syntax as ``extract`` (described earlier).

.. code:: ipython3

    process_tree.mp_ioc.extract(columns=['CommandLine'])


+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
|     | IoCType         | Observable                                                                                           | SourceIndex  |
+=====+=================+======================================================================================================+==============+
| 0   | dns             | microsoft.com                                                                                        | 24           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 1   | url             | \http://server/file.sct                                                                              | 31           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 2   | dns             | server                                                                                               | 31           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 3   | dns             | evil.ps                                                                                              | 35           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 4   | url             | \http://somedomain/best-kitten-names-1.jpg'                                                          | 37           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 5   | dns             | somedomain                                                                                           | 37           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 6   | dns             | blah.ps                                                                                              | 40           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 7   | md5_hash        | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                     | 40           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 8   | dns             | blah.ps                                                                                              | 41           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 9   | md5_hash        | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa                                                                     | 41           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 10  | md5_hash        | 81ed03caf6901e444c72ac67d192fb9c                                                                     | 44           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 11  | url             | \http://badguyserver/pwnme                                                                           | 46           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 12  | dns             | badguyserver                                                                                         | 46           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 13  | url             | \http://badguyserver/pwnme                                                                           | 47           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 14  | dns             | badguyserver                                                                                         | 47           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 15  | dns             | Invoke-Shellcode.ps                                                                                  | 48           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 16  | dns             | Invoke-ReverseDnsLookup.ps                                                                           | 49           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 17  | dns             | Wscript.Shell                                                                                        | 67           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 18  | url             | \http://system.management.automation.amsiutils').getfield('amsiinitfailed','nonpublic,static').s...  | 77           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 19  | dns             | system.management.automation.amsiutils').getfield('amsiinitfailed','nonpublic,static').setvalue(...  | 77           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 20  | ipv4            | 1.2.3.4                                                                                              | 78           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 21  | dns             | wscript.shell                                                                                        | 81           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 22  | dns             | abc.com                                                                                              | 90           |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 23  | ipv4            | 127.0.0.1                                                                                            | 102          |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 24  | url             | \http://127.0.0.1/                                                                                   | 102          |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+
| 25  | win_named_pipe  | \\.\pipe\blahtest"                                                                                   | 107          |
+-----+-----------------+------------------------------------------------------------------------------------------------------+--------------+

.. note:: the URLs in the previous table have been altered to prevent
   inadvertent navigation to them.
	CommandLine
0	.\ftp -s:C:\RECYCLER\xxppyy.exe
1	.\reg not /domain:everything that /sid:shines is /krbtgt:golden !
2	cmd /c "systeminfo && systeminfo"
3	.\rundll32 /C 42424.exe
4	.\rundll32 /C c:\users\MSTICAdmin\42424.exe
	IoCType	Observable	SourceIndex
48	windows_path	.\powershell	36
49	url	http://somedomain/best-kitten-names-1.jpg'	37
53	windows_path	.\pOWErS^H^ElL^.eX^e^	37
58	md5_hash	81ed03caf6901e444c72ac67d192fb9c	44
59	url	http://badguyserver/pwnme"	46
68	windows_path	.\reg query add mscfile\\\\open	59
72	windows_path	\system\CurrentControlSet\Control\Terminal	63
92	ipv4	1.2.3.4	78
108	ipv4	127.0.0.1	102
109	url	http://127.0.0.1/	102
110	windows_path	\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost\MyNastySvcHostConfig	103
IoCType	Regex
ipv4	(?P<ipaddress>(?:[0-9]{1,3}\\.){3}[0-9]{1,3})
ipv6	(?<![:.\\w])(?:[A-F0-9]{1,4}:){7}[A-F0-9]{1,4}(?![:.\\w])
dns	((?=[a-z0-9-]{1,63}\\.)[a-z0-9]+(-[a-z0-9]+)*\\.){2,}[a-z]{2,63}
url	(?P<protocol>(https?\|ftp\|telnet\|ldap\|file)://) (?P<userinfo>([a-z0-9-._~!$&\\'()+,;=:]\|%[0-9A-F]{2})@)? (?P<host>([a-z0-9-._~!$&\\'()+,;=]\|%[0-9A-F]{2}))
windows_path	(?P<root>[a-z]:\|\\\\\\\\[a-z0-9_.$-]+\|\|[.]+) (?P<folder>\\\\(?:[^\\/:?"\\\'<>\|\\r\\n]+\\\\)) > (?P<file>[^\\\\/*?""<>\|\\r\\n ]+)
linux_path	(?P<root>/+\|\|[.]+) (?P<folder>/(?:[^\\\\/:?<>\|\\r\\n]+/)) (?P<file>[^/\\0<>\|\\r\\n ]+)
md5_hash	(?:^\|[^A-Fa-f0-9])(?P<hash>[A-Fa-f0-9]{32})(?:$\|[^A-Fa-f0-9])
sha1_hash	(?:^\|[^A-Fa-f0-9])(?P<hash>[A-Fa-f0-9]{40})(?:$\|[^A-Fa-f0-9])
ipv6	(?:^\|[^A-Fa-f0-9])(?P<hash>[A-Fa-f0-9]{64})(?:$\|[^A-Fa-f0-9])