ESXi SD card faulty, host errors (ql_ima.log files filling up ramdisk on ESXi SD card)

We came across the above problem shortly after updating our ESXi hosts from 5.5 to 6.5. The host was reporting an issue with storage and on looking it reported the SD card was dead.

Checked and it looked as though the SD card was faulty so was going to change it out but after trying a host reboot first it resolved the problem and came back up with no errors.

Then again a few weeks later the same problem appeared, looking into the issue we found the tmp directory on the SD card was filling up to 100%. This stopped things like updates occurring also.

I found an article which sounded like exactly the same problem we were having here and it seems it was the same issue.

https://www.blackmanticore.com/918a7e3d3af84da8fd796d839a7726bf

We did not have the issue before upgrading to ESXI 6.5 using the custom HP ESXi image. So this upgrade has introduced this bug into the environment.

To check if you have this issue you need to enable SSH on the ESXi host with the issue.

Click on the host then go to the configure tab;

Go down to System and then under the security profile click Edit on the services pane and start the SSH service.

You will now be able to SSH using Putty onto the ESXi host.

Login to the SSH session using the root credentials for your ESXi host.

As per the details on the URL above run the command vdf -h which will show you all the directories and you should be able to see if any are at capacity as per below;

Here we can see the tmp directory is full

If you issue the change directory command “cd tmp” and then issue the “ls” command you can see the contents of the directory

Here we can see the offending files the ql****** log files

As per the following URL download the updated driver which corrects this problem and either update via CLI or alternatively add into update manager and remediate the hosts.

https://www.blackmanticore.com/becfb055a261e92e2151b4d3fd760b3c

You will need to first reboot your host to clear the tmp directory then run the update.

I went down the route of adding the updated driver to update manager and remediating the hosts.

Since running the update and leaving it a few days on checking our hosts again the QLogic logs have not reappeared.

Many thanks to BlackManticore for the article it really helped!