I am seeking your assistance regarding a challenge we are facing in our current project to present events generated by the EPM-W Agent in PowerBI.
As you are aware, the GUI allows us to download up to 5 million records, but this process is often time-consuming, especially when the data size reaches several gigabytes. Additionally, I recently learned that the API has a limit of only 10,000 events per request, which is insufficient for our environment. We generate approximately 455,882,549 events over a 30-day period. While I am actively working to reduce the number of events, this is our current situation.
Your advice on how to efficiently process and visualize such large datasets in PowerBI would be greatly appreciated. Specifically, if there are ways to increase the API limit or alternative best practices for handling this scale of data, please let me know.
Best answer by Jens Hansen
Dear all,
I am seeking your assistance regarding a challenge we are facing in our current project to present events generated by the EPM-W Agent in PowerBI.
As you are aware, the GUI allows us to download up to 5 million records, but this process is often time-consuming, especially when the data size reaches several gigabytes. Additionally, I recently learned that the API has a limit of only 10,000 events per request, which is insufficient for our environment. We generate approximately 455,882,549 events over a 30-day period. While I am actively working to reduce the number of events, this is our current situation.
Your advice on how to efficiently process and visualize such large datasets in PowerBI would be greatly appreciated. Specifically, if there are ways to increase the API limit or alternative best practices for handling this scale of data, please let me know.
1st off you have to start doing some allow listing to remove the known noise from your reporting, else it will be impossible to find the unknown in you environment.
Then you can workaround the API restriction by for calling batches of time. and configure a warning of you exceed 10K events in one batch, using below
# Process events in 1-hour batches $startTime = [DateTime]::Parse($TimePeriod_StartDate) $endTime = [DateTime]::Parse($TimePeriod_EndDate) $batchSize = 1 # 1 hour batches # Loop through each hour in the 24-hour period
# Add the check here If ($totalRecordCount -eq 10000) { Write-Error "Maximum record limit reached (10,000 records)" Break
Again. Allow-listing is crucial, I bet if you filter on Allowed application that will be 95% of the events on you system, number one app is Chrome, Slack and others that run without admin rights from the users profile. Set the filter to 30 days, which is the retention for Allowed Events. (Passive events 106)
I am seeking your assistance regarding a challenge we are facing in our current project to present events generated by the EPM-W Agent in PowerBI.
As you are aware, the GUI allows us to download up to 5 million records, but this process is often time-consuming, especially when the data size reaches several gigabytes. Additionally, I recently learned that the API has a limit of only 10,000 events per request, which is insufficient for our environment. We generate approximately 455,882,549 events over a 30-day period. While I am actively working to reduce the number of events, this is our current situation.
Your advice on how to efficiently process and visualize such large datasets in PowerBI would be greatly appreciated. Specifically, if there are ways to increase the API limit or alternative best practices for handling this scale of data, please let me know.
