Jupyter, msticpy and Azure Sentinel¶
Creating a notebooks project within Azure Notebooks is directly supported by Azure Sentinel. Click on the notebook icon in the Azure Sentinel main navigation menu. From here you have the option to create a new project from our GitHub repo or just open your existing Azure Notebooks project. Azure Notebooks is a Jupyterhub implementation and has a free tier that you can use for any notebook tasks.
If you have a local installation of Python 3.6 or later, you can also download the notebooks and run these locally. My personal recommendation is to use the Anaconda distribution since it contains the Jupyter packages and many others needed for the Azure Sentinel notebooks.
Further reading: Using Notebooks in Azure Sentinel and .
Open one of the Sample Notebooks¶
Note: you do not need to have Python or any of the dependencies installed to view notebooks in nbviewer or GitHub.
When it comes to running one of the notebooks in against real data, you will need some preparatory steps.
Permissions in your Azure Sentinel/Log Analytics Workspace¶
In order to read any data, you will need to have at least LogAnalytics Reader role for your account.
Configuring your Python Environment for the First Time¶
You will need to carry out this procedure every time you start working in a fresh Python environment.
If you are using Notebooks for Azure using free computer, creating a new project is effectively starting a new environment (although there are ways to automate this setup). The exception to this is if you are using a dedicated Compute resource such as a Data Science Virtual Machine. Since this machine is persisted and linked to your Notebooks for Azure account, all the configuration will be there next time you come to use it.
If you are working locally or using another Jupyterhub hosted environment, you will only need to do this environment configuration for each fresh install or when you create a new python or conda virtual environment.
- Ensure that you have a version of Python 3.6 or later.
- Install the two main packages used by the notebook: Kqlmagic and msticpy (see references at end of document). These will install most of the dependencies needed by the notebooks if they are not already installed.
- Install one or two additional python packages – these vary depending on the notebook.
If you are running on a Windows machine where Python is installed for All Users, you may have to add the –user flag to the pip install commands. You will see permission failures when trying to install if this is the case.
pip install --user <pkg_name>
Notes for Conda users.
If you are running in a Conda environment (an Anaconda distribution) run the pip commands from a Conda prompt, ideally in a dedicated Conda virtual environment. Just start an Anaconda prompt shell, paste the pip install commands into it and execute them, rather than running them from the notebook. You will need to run Jupyter from the same environment. More details of can be found here.
Keeping the packages up-to-date¶
It is a good idea to force an update of packages at regular intervals using
pip install --upgrade <pkg_name>
to ensure that you have the latest features and fixes (including fixes for security vulnerabilities).
There are two main pieces of housekeeping here that you need each time a notebook is started:
- Importing python libraries (this is reading in the installed versions of the libraries so that they become accessible in your python session). I try to keep all of the imports at the start of my notebooks so that you have an early warning of missing dependencies.
- Authenticating to Azure Sentinel/Log Analytics with Azure Active Directory. This is a complex topic but there are two main methods of authentication:
- Interactive device/user authentication - this prompts you for user credentials and a one-time device code. While this frees you from having to worry about saving/pasting in credentials each time, you do suffer a multi-prompt authentication experience. If you happen to be working a long time in a single notebook this is not too onerous but can be frustrating if you are hopping between multiple notebooks.
- AppId authentication - this uses an App account, created in your Azure Active Directory tenant, and granted read access to your Log Analytics workspace. This is a smoother authentication experience but means that you need to manage the app client secret (and, hopefully, avoid leaving a copy of it in a notebook uploaded to GitHub!).
On successful authentication you should see a button displayed. Clicking this brings up a pop-up of the schema of all the tables your workspace and is a useful reference feature. This feature is also accessible from the notebook Help menu.
- The msticpy Python package containing tools used in these notebooks developed by engineers on the Microsoft Threat Intelligence team. It is available on GitHub along with several notebooks documenting the use of the tools and on PyPi.
- Kqlmagic is a Jupyter-friendly package developed by Azure’s Michael Binstock.
- Using Notebooks in Azure Sentinel is the official documentation for using Jupyter notebooks in Azure Sentinel.
- Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks by John Lambert
- Azure Sentinel Jupyter notebooks can be found here on GitHub. - Account Explorer - Domain and URL Explorer - IP Explorer - Linux Host Explorer - Windows Host Explorer
Other sample notebooks with saved data are in the Sample-Notebooks folder: