Using the new Archiving, Basic Logs, and Retention Workbook

by

Back in February of 2022 several new features for Azure Monitor were released to support long term data retention, data search, data recovery and data ingestion. More specifically these features are built into Azure Log Analytics which is the data lake for Microsoft Sentinel.

While we are still waiting for many of these features to be better integrated into the Sentinel UI, the new Archiving, Basic Logs, and Retention workbook provides the ability to view and configure Sentinel Data Archiving and Basic Logs. The workbook will also allow you to manage Search and Restore tables and estimate costs associated with Data Archiving and Basic Logs.

See the list of features announced in February

Data Retention and Archive

By default Sentinel allows for 90 days free of data retention with the ability to retain data for up to two years. The new Data Archiving feature aka Archived Logs will allow you to retain data on per table basis for up to 7 years at a much cheaper cost. There are some caveats to be aware of, data retained in archived logs can’t be used interactively like regular data. To access archived data you must first run a search job and then restore the data to a custom table. I would recommend reading more about this feature from the Azure Monitor Docs

Be aware that the Azure Monitor docs do not include Sentinel UI enhancements that present this integration in a better manner. Please see the below links on the UI enhancements with Sentinel

Search across long time spans in large datasets
Restore archived logs from search

Basic Logs

Basic Logs are a new log ingestion plan intended for high volume but low security value logs such as netflow, vpc flow logs, and cloud storage access logs. Similar to the data archiving feature the ingestion price is significantly cheaper, however there are also some caveats.

  • Basic Logs are applied on a per table basis
  • Not all tables are supported with Basic Logs. See Which tables support Basic Logs?
    • Generally speaking tables created with Data Collection Rules are supported
  • You cannot use Basic Logs with Analytics Rules
  • There is a limited subset of supported KQL operators
  • Basic Logs have fixed retention of 8 days. However you can configure longer retention using the Data Archiving feature.
Take a look at the FAQ: Search, Basic Ingestion, Archive, and Data Restoration - Microsoft Tech Community docs for more information on Basic Logs, Search, Archiving, and Restore

Using the Workbook

Installation

You can find the workbook in the Sentinel Workbooks Templates section under the name Archiving, Basic Logs, and Retention. Simply save the workbook to get started. You can also find this in the Sentinel github repo.

Adjusting Base Pricing

The pricing parameters under the Update Pricing Based on your Region section can be updated to reflect your current region’s base pricing. You can get those values from the Azure Pricing Calculator. A couple of notes:

  • The Ingestion Price is the sum of Sentinel and Log Analytics ingestion price
  • The Workspace Retention Period (Months) is automatically obtained from your current workspace’s retention period. However you can adjust this value to predict costs. 
Adjusting the workbook Time Range will only impact costs estimates on the Cost Estimation tab. All other cost estimates are based on the last 30 days.

Working with Data Archiving

Under the Data Archive tab you will find information on all tables in your Sentinel workspace. You can use the option selectors to filter further by the ingestion plan and archive tier.

Selecting a table from the grid results will show an additional section where you can update the table’s retention period and see the estimated monthly cost for enabling archiving. You can also reset the retention period to your workspace’s retention period.

Currently the API doesn't support configuring multiple tables at once. You can leverage the Configure-Long-Term-Retention PowerShell script in the meantime.

Working with Basic Logs

Under the Basic Logs tab you will find information on current basic log tables and available tables that can be enabled for basic logs. Selecting a table from the grid results will show an additional section where you can either enable or disable basic logs. You will also see a section for estimating monthly costs.

Working with Search and Restore

Under the Search and Restore tab you will find information on current tables associated with search and restore. You will notice there is not any information on table size due to current limitations with not being able to retrieve this information from the Log Analytics API.

Selecting a table from the grid results will show an additional section where you can update the table’s retention period, reset the retention period, or delete the table. You will also see a section for estimating costs.

Working with Cost Estimation

Under the Costs Information tab you will find information you can use to estimate data retention costs across all tables. You will also find a section for estimating basic logs costs.

Leverage the Total Retention in Days input box to estimate Data Archiving Costs. You can also update the Workspace Retention Period (Months) parameter under the Update Pricing Based on your Region section to estimate workspace data retention costs.

%d