Azure Data Enrichment ===================== Description ----------- This package contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. By providing an Azure Resource or Azure Subscription the package returns key contextual information regarding the Subscription or Resource. This package is primarily aimed at Azure IaaS resources but will work with any Azure Resource type. This feature is currently a work in progress and additional data enrichment capabilities will be added over time. :py:mod:`Azure Data API documentation` Instantiating and Connecting with an Azure Data Connector --------------------------------------------------------- See :py:class:`Azure Data ` In order to connect to the Azure API and retrieve the required data we need to instantiate an Azure Data Connector and connect to the API. Authentication to the Azure API is handled via the azure_auth package. By default this package will attempt to use a prioritized list of authentication options. Available options are: * 'env' - This checks for credentials stored as environment variables. If this option is selected valid credentials in msticpyconfig.yaml will be written as environment variable values and used. * 'cli' - This attempts to use credentials generated by logging in via the `Azure CLI `__ on the host running the notebook kernel. * 'msi' - This attempts to use an `Azure Managed Identity. `__ * 'interactive' - This prompts the browser to interactively login using the device's browser. By default ``['cli', 'msi', 'devicecode']`` is used but you can provide an alternative list to ``.connect`` via the auth_methods parameter. .. code:: ipython3 az = AzureData() az.connect(auth_methods=['cli','interactive']) For more details on Azure authentication see :doc:`../getting_started/AzureAuthentication`. Get Azure Subscription Details ------------------------------ See :py:meth:`get_subscriptions ` Details about the subscription a resource is a member of can provide vital context to a security analyst when conducting an investigation. This package contains 2 functions to support this. AZURE_DATA_CONNECTOR.list_subscriptions() returns a pandas DataFrame with details of all the subscriptions within the tenant. .. code:: ipython3 az.get_subscriptions() .. raw:: html
Subscription ID Display Name State
0 c690adc4-ec82-41fa-ad36-80f3c8899bd0 Visual Studio Enterprise SubscriptionState.enabled
| See :py:meth:`get_subscription_info ` AZURE_DATA_CONNECTOR.get_subscription_info() gets information on a specific subscription ID. .. code:: ipython3 az.get_subscription_info("c690adc4-ec82-41fa-ad36-80f3c8899bd0") .. parsed-literal:: {'Subscription ID': 'c690adc4-ec82-41fa-ad36-80f3c8899bd0', 'Display Name': 'Visual Studio Enterprise', 'State': 'SubscriptionState.enabled', 'Subscription Location Limits': 'Public_2014-09-01', 'Subscription Quota': 'MSDN_2014-09-01', 'Spending Limit': } Get Azure Resource Details -------------------------- See :py:meth:`get_resources ` As well as subscriptions we can return details on a specific Azure resource. AZURE_DATA_CONNECTOR.get_resources() returns a pandas DataFrame with details on all resources within a Subscription or Resource Group. In addition, you can request full properties on each Resource with the get_props = True parameter. However, this can take some time to return results. .. code:: ipython3 resources = az.get_resources(sub_id="bca22c36-a158-44ff-8cbb-23fa92236a55") resources.head() +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ | | resource_id | name | resource_type | location | tags | plan | properties | kind | managed_by | sku | identity | +===+====================================================+====================================================+====================================+=============+=============================================+=======+=============+==========+====================================================+====================================================+===========+ | 0 | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | cloud-shell-storage-westeurope-vnet | Microsoft.Network/virtualNetworks | centralus | {} | None | None | None | None | None | None | +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ | 1 | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | csb3b701f84d04bx4479x89b | Microsoft.Storage/storageAccounts | westeurope | {'ms-resource-usage': 'azure-cloud-shell'} | None | None | Storage | None | {'additional_properties': {}, 'name': 'Standar... | None | +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ | 2 | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | bluepot-01_OsDisk_1_ad7a7c0383444f02830ba46418... | Microsoft.Compute/disks | westus | None | None | None | None | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | None | None | +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ | 3 | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | bluepot-02_OsDisk_1_dce988e082e54617ae3622eca0... | Microsoft.Compute/disks | westus | None | None | None | None | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | None | None | +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ | 4 | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | CentOS-Test_OsDisk_1_7ee38d36b893481e8a68405c0... | Microsoft.Compute/disks | westus | None | None | None | None | /subscriptions/3b701f84-d04b-4479-89b1-fa8827e... | {'additional_properties': {}, 'name': 'Premium... | None | +---+----------------------------------------------------+----------------------------------------------------+------------------------------------+-------------+---------------------------------------------+-------+-------------+----------+----------------------------------------------------+----------------------------------------------------+-----------+ See :py:meth:`get_resource_details` You can return full details on a single resource with AZURE_DATA_CONNECTOR.get_resource_details() and passing a Resource ID. .. code:: ipython3 az.get_resource_details(resource_id="/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv") .. parsed-literal:: {'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv', 'name': 'UbuntuDevEnv', 'resource_type': 'Microsoft.Compute/virtualMachines', 'location': 'northeurope', 'tags': {}, 'plan': None, 'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87', 'hardwareProfile': {'vmSize': 'Standard_B1s'}, 'storageProfile': {'imageReference': {'publisher': 'Canonical', 'offer': 'UbuntuServer', 'sku': '18.04-LTS', 'version': 'latest', 'exactVersion': '18.04.201812040'}, 'osDisk': {'osType': 'Linux', 'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5', 'createOption': 'FromImage', 'caching': 'ReadWrite', 'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}}, 'dataDisks': [{'lun': 0, 'name': 'UbuntuDevEnv_DataDisk_0', 'createOption': 'Attach', 'caching': 'None', 'writeAcceleratorEnabled': False, 'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'}, 'toBeDetached': False}]}, 'osProfile': {'computerName': 'UbuntuDevEnv', 'adminUsername': 'user', 'linuxConfiguration': {'disablePasswordAuthentication': True, 'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys', 'keyData': ''}]}, 'provisionVMAgent': True}, 'secrets': [], 'allowExtensionOperations': True}, 'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]}, 'provisioningState': 'Succeeded'}, 'kind': None, 'managed_by': None, 'sku': None, 'identity': None} .. note:: You can also provide a dictionary of resource details if you don't have a complete Resource ID. The details dictionary must contain: * resource_group_name * resource_provider_namespace * parent_resource_path (if there isn't one leave as a empty string). * resource_type * resource_name .. code:: ipython3 resource_details = {"resource_group_name":"Contoso", "resource_provider_namespace":"Microsoft.Compute", "parent_resource_path":"", "resource_type":"virtualMachines", "resource_name":"UbuntuDevEnv"} az.get_resource_details(resource_details=resource_details) .. parsed-literal:: {'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv', 'name': 'UbuntuDevEnv', 'resource_type': 'Microsoft.Compute/virtualMachines', 'location': 'northeurope', 'tags': {}, 'plan': None, 'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87', 'hardwareProfile': {'vmSize': 'Standard_B1s'}, 'storageProfile': {'imageReference': {'publisher': 'Canonical', 'offer': 'UbuntuServer', 'sku': '18.04-LTS', 'version': 'latest', 'exactVersion': '18.04.201812040'}, 'osDisk': {'osType': 'Linux', 'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5', 'createOption': 'FromImage', 'caching': 'ReadWrite', 'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}}, 'dataDisks': [{'lun': 0, 'name': 'UbuntuDevEnv_DataDisk_0', 'createOption': 'Attach', 'caching': 'None', 'writeAcceleratorEnabled': False, 'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'}, 'toBeDetached': False}]}, 'osProfile': {'computerName': 'UbuntuDevEnv', 'adminUsername': 'user', 'linuxConfiguration': {'disablePasswordAuthentication': True, 'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys', 'keyData': ''}]}, 'provisionVMAgent': True}, 'secrets': [], 'allowExtensionOperations': True}, 'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]}, 'provisioningState': 'Succeeded'}, 'kind': None, 'managed_by': None, 'sku': None, 'identity': None} Get Azure Network Details ------------------------- See :py:meth:`get_network_details ` If your Azure resources has a network interface associated with it (for example a VM) you can return details on the interface as associated Network Security Group (NSG). Calling this function is very similar to getting resource details however instead of passing it a resource ID you provide the network interface ID for the network device you want details for. .. code:: ipython3 az.get_network_details(networkID=NETWORK_INTERFACE_ID, sub_id=SUBSCRIPTION_ID) .. note:: If youa are looking for a VM network interface ID you can use get_resource_details to get details on the VM. The network interface will be under properties > networkProfile > networkInterfaces > id This will return a DataFrame containing details of all IP addresses and subnets associated with the network interface. Get Azure Metrics ----------------- See :py:meth:`get_metrics ` Azure provides a range of metrics for resources. The types of metrics available depends on the Azure resource in question, a full list of metrics can be found `here `__. You can return all of these metrics with get_metrics. In order to call this function you need to provide the metrics you want to retrieve in a comma separated string e.g. ""Percentage CPU,Disk Read Bytes,Disk Write Bytes", along with the resource ID of the item you wish to retrieve the metrics for, and the subscription ID that resource is part of. You can also choose to get the metrics sampled at either the minute or the hour interval, and for how many days preceding you want metrics for. By default the function returns hourly metrics for the last 30 days. .. code:: ipython3 az.get_metrics(metrics="Percentage CPU", resource_id=resource_details['resource_id'], sub_id=sub_details['Subscription ID'], sample_time="hour", start_time=15) This returns a dictionary of items with the metric name as they key and a DataFrame of the metrics as the value. .. note:: get_metrics is resource specific, so if you want to get metrics from more than one resource you will need separate function calls.