Storage Management

[Mattm261]

Hi All,

Been using LME for a bit now with about ~15 endpoints for testing, this past weekend the drive filled up and now I can't login (working on extending the drive). I was reading the system requirements and it does say that the system will purge old logs, is there a reason why this didn't happen?

Thank you!

[aarz-snl]

Tuning and space management is for sure one of the biggest challenges with selfhosting a SIEM

Couple things I would recommend doing.

Free up space to get logged back in:

From LME server

login to the elasticsearch container:

sudo -i podman exec -it lme-elasticsearch bash

list your indices by size:

curl -k -u elastic:${ELASTIC_PASSWORD} 'https://localhost:9200/_cat/indices?v&s=store.size:desc&h=index,store.size,docs.count,creation.date.string'

You can then delete some of them using a command like so:

curl -k -X DELETE -u elastic:${ELASTIC_PASSWORD} https://localhost:9200/.ds-metrics-windows.service-default-2025.02.24-000001

The end of the command will be the name of the index you want to delete.

After that I would login to Elastic and go to Fleet -> Agent Policies -> endpoint policy

If you added the "windows" integration.. select it. Scroll down and turn off "collect windows perfmon and service metrics.

Go back and select "system" if you have that integration. You may want to turn off metrics in this integration as well.

Your indices dont just automatically delete when space starts filling up you have to set index polices.

See the example here:

https://github.com/cisagov/LME/blob/main/docs/markdown/maintenance/index-management.md

You may need to go into

Stack Management -> Index Lifecycle policies - Click Deprecated -> click "logs" (theres also one called metrics)

This are where you create the rules for when this data is delete. The default is 50gb or 30 days it will 'rollover' -- this doesn't mean delete. It just starts a new named index. By default they were NEVER delete. You need to click to turn on the cold phase, set a day... and then you can also set a delete day limit.

You can also click "Use recommended defaults" and edit that as need if you need to delete beforehand.

You may find however, once you turn off all those metrics you dont need to do such changes other than deleting after X amount of days.

[Mattm261]

Thank you so much for the detailed response that helped clear up some space and was able to tweak the index policy as well to delete of X amount of days.

Thank you!

[SomewhereinSC]

I didn't know if I should start a new thread since this topic hits on my question.

I just installed LME for the first time on a Dell server that has 2 RAID setups, the OS drive and what I thought would be the LME or DATA drive, but it looks like LME only used the OS drive (/sda) and /sdb isn't even shown when doing df command.

I've only setup a single test Windows client with Sysmon and Elastic installed, so a complete rebuilds is not a problem, just need to know if it is possible to install LME to the larger RAID drive in my Dell? I sadly am Linux deficient, but can google the heck out of things for sure. I appreciate any pointers

[SomewhereinSC]

hmm for now tested using this information https://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-adding-a-new-disk/#google_vignette to extend my setup to the 2nd drive... not as elegant as I'd like... but it worked.