The Velociraptor provider

Velociraptor driver documentation

The Velociraptor data provider can read Velociraptor offline collection log files (see Velociraptor Offline Collection) and provide convenient query functions for each data set in the output logs.

The provider can read files from one or more hosts, stored in in separate folders. The files are read, converted to pandas DataFrames and grouped by table/event. Multiple log files of the same type (when reading in data from multiple hosts) are concatenated into a single DataFrame.

The query provider query functions will ignore parameters and do no further filtering. You can use pandas to do additional filtering and sorting of the data, or use it directly with other MSTICPy functionality.

Note

The examples used in this document were from data provided by Blue Team Village at Defcon 30. You can find this data at the Project-Obsidian-DC30 GitHub and more about Project Obsidian here.

Velociraptor Configuration

You can (optionally) store your connection details in msticpyconfig.yaml, instead of supplying the data_paths parameter to the QueryProvider class.

For more information on using and configuring msticpyconfig.yaml see msticpy Package Configuration and MSTICPy Settings Editor

The Velociraptor settings in the file should look like the following:

DataProviders:
    ...
    Velociraptor:
        data_paths:
            - /home/user1/sample_data
            - /home/shared/sample_data

Expected log file format

The log file format must be a text file of JSON records. An example is shown below

{"Pid":1664,"Ppid":540,"Name":"spoolsv.exe","Path":"C:\Windows\System32\spoolsv.exe","CommandLine":"C:\Windows\System32\spoolsv.exe","Hash":{"MD5":"c111e3d38c71808a8289b0e49db40c96","SHA1":"e56df979d776fe9e8c3b84e6fef8559d6811898d","SHA256":"0ed0c6f4ddc620039f05719d783585d69f03d950be97b49149d4addf23609902"},"Username":"NT AUTHORITY\SYSTEM","Authenticode":{"Filename":"C:\Windows\System32\spoolsv.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002ed2c45e4c145cf48440000000002ed","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows","Timestamp":null,"Trusted":"trusted","_ExtraInfo":{"Catalog":"C:\Windows\system32\CatRoot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Package_6350_for_KB5007192~31bf3856ad364e35~amd64~~10.0.1.8.cat"}},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49697,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"}
{"Pid":548,"Ppid":416,"Name":"lsass.exe","Path":"C:\Windows\System32\lsass.exe","CommandLine":"C:\Windows\system32\lsass.exe","Hash":{"MD5":"93212fd52a9cd5addad2fd2a779355d2","SHA1":"49a814f72292082a1cfdf602b5e4689b0f942703","SHA256":"95888daefd187fac9c979387f75ff3628548e7ddf5d70ad489cf996b9cad7193"},"Username":"NT AUTHORITY\SYSTEM","Authenticode":{"Filename":"C:\Windows\System32\lsass.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002f49e469c54137b85e00000000002f4","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49722,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:54Z"}
{"Pid":540,"Ppid":416,"Name":"services.exe","Path":"C:\Windows\System32\services.exe","CommandLine":"C:\Windows\system32\services.exe","Hash":{"MD5":"fefc26105685c70d7260170489b5b520","SHA1":"d9b2cb9bf9d4789636b5fcdef0fdbb9d8bc0fb52","SHA256":"930f44f9a599937bdb23cf0c7ea4d158991b837d2a0975c15686cdd4198808e8"},"Username":"NT AUTHORITY\SYSTEM","Authenticode":{"Filename":"C:\Windows\System32\services.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002a5e1a081b7c895c0ed0000000002a5","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49728,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:57Z"}

The columns in each JSON will be used to create the pandas DataFrame columns.

Using the Velociraptor provider

To use the Velociraptor provider you need to create an QueryProvider instance, passing the string “VelociraptorLogs” as the data_environment parameter. If you have not configured data_paths in msticpyconfig.yaml, you also need to add the data_paths parameter to specify specific folders or files that you want to read.

Calling the connect method triggers the provider to register the paths of the log files to be read (although the log files are not read and parsed until the related query is run - see below).

Listing Velociraptor tables

Until you run connect no queries will be available. After running connect you can list the available queries using the list_queries

qry_prov.list_queries()

['velociraptor.Custom_Windows_NetBIOS',
'velociraptor.Custom_Windows_Patches',
'velociraptor.Custom_Windows_Sysinternals_PSInfo',
'velociraptor.Custom_Windows_Sysinternals_PSLoggedOn',
'velociraptor.Custom_Windows_System_Services',
'velociraptor.Windows_Applications_Chrome_Cookies',
'velociraptor.Windows_Applications_Chrome_Extensions',
'velociraptor.Windows_Applications_Chrome_History',
'velociraptor.Windows_Applications_Edge_History',
'velociraptor.Windows_Forensics_Lnk',
'velociraptor.Windows_Forensics_Prefetch',
'velociraptor.Windows_Forensics_ProcessInfo',
'velociraptor.Windows_Forensics_Usn',
...]

Querying Velociraptor table schema

The schema of the log tables is built by sampling the first record from each log file type, so is relatively fast to retrieve even if you have large numbers and sizes of logs.

vc_prov.schema["Windows_Network_InterfaceAddresses"]

{'Index': 'int64',
'MTU': 'int64',
'Name': 'object',
'HardwareAddr': 'object',
'Flags': 'int64',
'IP': 'object',
'Mask': 'object'}

Running a Velociraptor query

Each query returns a pandas DataFrame retrieved from the logs of that type (potentially containing records from multiple hosts depending on the data_paths you specified).

qry_prov.vc_prov.velociraptor.Windows_Forensics_ProcessInfo()

	Name	PebBaseAddress	Pid	ImagePathName	CommandLine	CurrentDirectory	Env
10	LogonUI.exe	0x95bd3d2000	804	C:Windowssystem32LogonUI.exe	“LogonUI.exe” /flags:0x2 /state0:0xa3b92855 /state1:0x41c64e6d	C:Windowssystem32	{‘ALLUSERSPROFILE’: ‘C:\ProgramD..
11	dwm.exe	0x6cf4351000	848	C:Windowssystem32dwm.exe	“dwm.exe”	C:Windowssystem32	{‘ALLUSERSPROFILE’: ‘C:\ProgramD..
12	svchost.exe	0x6cd64d000	872	C:WindowsSystem32svchost.exe	C:WindowsSystem32svchost.exe -k termsvcs	C:Windowssystem32	{‘ALLUSERSPROFILE’: ‘C:\ProgramD..
13	svchost.exe	0x7d18e99000	912	C:WindowsSystem32svchost.exe	C:WindowsSystem32svchost.exe -k LocalServiceNetworkRestricted	C:Windowssystem32	{‘ALLUSERSPROFILE’: ‘C:\ProgramD..
14	svchost.exe	0x5c762eb000	920	C:Windowssystem32svchost.exe	C:Windowssystem32svchost.exe -k LocalService	C:Windowssystem32	{‘ALLUSERSPROFILE’: ‘C:\ProgramD..