Longhorn PVC expansion blocked by filesystem errors
A PVC can show its new capacity while an application still sees the old filesystem size. This can happen when Kubernetes and Longhorn have expanded the volume’s block device, but the filesystem could not be grown.
This guide explains how to identify which layer is stuck, why filesystem errors block an online resize, and how to repair and grow an ext4 filesystem on a Longhorn volume. Filesystem repair is a maintenance operation: take a backup, stop all writers, and verify that the filesystem is unmounted before running repair tools.
The three layers of a PVC expansion
A filesystem-backed PVC has three relevant sizes:
| Layer | What it represents | How to inspect it |
|---|---|---|
| PVC/PV capacity | The size Kubernetes requested and recorded | kubectl get pvc and kubectl get pv |
| Longhorn volume | The configured size of the virtual block device and its replicas | Longhorn UI or the Volume custom resource; verify the attached device with blockdev |
| Filesystem | The usable space mounted in the workload | df -hT in the pod |
These sizes normally converge during expansion. The PVC request is increased, Longhorn expands the block device, and the CSI node expansion step grows the filesystem. If the third step fails, the first two can still show the new size while df continues to show the old one.
Longhorn supports online expansion, but availability and behavior depend on the Longhorn version and data engine. Check the documentation for the version in use. Longhorn cannot safely grow a filesystem that the operating system considers inconsistent. For ext4, a filesystem error can set an error state in its superblock. The kernel then refuses an online resize rather than risk worsening filesystem damage.
Identify the stuck layer
Set the namespace and PVC name, then find the PV name. For dynamically provisioned Longhorn volumes, the PV name is also normally the Longhorn volume name.
NAMESPACE=<namespace>
PVC=<claim-name>
PV=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
kubectl -n "$NAMESPACE" get pvc "$PVC" -o wide
kubectl get pv "$PV"
kubectl -n "$NAMESPACE" describe pvc "$PVC"
Check that the StorageClass permits expansion:
SC=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.storageClassName}')
kubectl get storageclass "$SC" -o yaml
The StorageClass should have allowVolumeExpansion: true. Inspect the matching Longhorn volume as well:
kubectl -n longhorn-system get volumes.longhorn.io "$PV" \
-o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,NODE:.status.currentNodeID,SIZE:.spec.size
Compare the values:
- PVC
spec.resources.requests.storage: the desired size. - PVC
status.capacity.storageand PVspec.capacity.storage: Kubernetes’ recorded capacity. - Longhorn volume
spec.size: the configured block-device size. When the volume is attached, verify the actual device size on its node withblockdev --getsize64. df -hT <mount-path>in the pod: the filesystem size the application can use.
For example, if the PVC and Longhorn volume configuration show 6Gi, blockdev confirms a 6Gi device, but df shows about 2Gi, the block device grew and filesystem expansion did not. The ROBUSTNESS field is about Longhorn’s volume/replica health; it is not an ext4 filesystem-health check.
Look at the PVC events and the CSI/Longhorn logs for the first failure. Exact event text varies by Kubernetes and Longhorn version. A resource-version conflict such as “the object has been modified” is an optimistic concurrency conflict; it may be transient if a controller retries successfully. A persistent resize failure, especially one mentioning filesystem errors, needs investigation rather than being dismissed as noise.
On the node where the volume is attached, kernel messages can explain why the filesystem resize failed:
sudo dmesg -T | grep -Ei 'EXT4-fs|longhorn|filesystem' | tail -100
Messages such as There are errors in the filesystem, so online resizing is not allowed or a block-bitmap/group-descriptor mismatch point to filesystem inconsistency.
Repair an ext4 filesystem safely
Do not run e2fsck on a mounted filesystem. First plan for downtime, take or verify a usable backup, and identify every workload that can write to the PVC. For a StatefulSet with several replicas, stop all replicas using the claim; stopping only one pod is not sufficient if another can mount the same data.
Stop consumers and detach the volume
Scale down the owning Deployment or StatefulSet, saving its original replica count:
kubectl -n "$NAMESPACE" get deployment <deployment-name> \
-o jsonpath='{.spec.replicas}{"\n"}'
kubectl -n "$NAMESPACE" scale deployment <deployment-name> --replicas=0
Use the corresponding StatefulSet command if that owns the workload. Also stop any jobs, maintenance pods, or other consumers using the PVC. Wait for pods to terminate and for Longhorn to report the volume as detached:
kubectl -n longhorn-system get volumes.longhorn.io "$PV" -o yaml
Do not proceed while the volume is mounted or still in use. For Longhorn RWX volumes, stopping the application pods is not by itself proof that the filesystem is free: RWX is provided through a Longhorn Share Manager using NFS. Verify that there are no remaining clients and that the Share Manager is no longer using the volume before attempting a filesystem check.
Attach for maintenance and check the device
Use the Longhorn UI to attach the detached volume to a maintenance node. Confirm the attached node in the UI or the Longhorn volume resource, then connect to that node. The device is normally exposed as /dev/longhorn/<PV-name>.
Set and verify the device path before using it:
DEVICE="/dev/longhorn/$PV"
sudo ls -l "$DEVICE"
sudo lsblk -f "$DEVICE"
grep -F "$DEVICE" /proc/mounts
sudo blockdev --getsize64 "$DEVICE"
The mount check must show no mount for this filesystem. blockdev --getsize64 reports bytes; for a 6Gi device, the expected value is 6442450944. If the device path is missing, the device is mounted, or the Longhorn volume is not in the state you expect, stop and resolve that before continuing.
Run a forced, read-only ext4 check first:
sudo e2fsck -f -n "$DEVICE"
echo "e2fsck exit code: $?"
With -n, e2fsck answers “no” to repair questions and does not change the filesystem. If it reports errors, do not run resize2fs yet. Review the findings and repair plan, and make sure a recoverable backup exists. Then run an interactive repair and review each proposed change:
sudo e2fsck -f "$DEVICE"
The -y option automatically answers “yes” to every repair prompt. Avoid using it as a shortcut unless you understand and accept all proposed changes; repairs can affect orphaned files or directory entries.
Grow the filesystem
Once the filesystem check completes without remaining errors, and the block device is larger than the filesystem, grow ext4 to fill the device while it remains unmounted:
sudo resize2fs "$DEVICE"
sudo dumpe2fs -h "$DEVICE" 2>/dev/null | grep -E 'Filesystem state|Block count|Block size'
With ext4, Block count multiplied by Block size gives the filesystem size in bytes. Check that it is consistent with the device size.
This procedure is specifically for ext4. XFS uses different tools and must be repaired with xfs_repair while unmounted; growing XFS uses xfs_growfs on a mounted filesystem. Do not use e2fsck or resize2fs on XFS.
Detach and restore the workload
After the filesystem is clean and has the expected size, detach the maintenance attachment through the Longhorn UI. Then restore the original replica count:
kubectl -n "$NAMESPACE" scale deployment <deployment-name> --replicas=<original-count>
kubectl -n "$NAMESPACE" rollout status deployment <deployment-name>
For a StatefulSet, use kubectl rollout status statefulset/<name>. Once the application is ready, check the mounted filesystem from the pod:
kubectl -n "$NAMESPACE" exec <pod-name> -- df -hT <mount-path>
The capacity shown by df should now match the expanded volume. Also check the PVC events and Longhorn volume state for any remaining resize or replica problems.
Comparison with a Portworx incident
The troubleshooting pattern comes from a real Portworx PVC expansion incident. The names and environment are omitted, but the useful symptoms were:
- The PVC, PV, and storage-system block device all reported 6Gi.
- The ext4 filesystem still reported 2Gi and was full.
- Kernel logs showed that ext4 had errors and would not allow an online resize.
- A filesystem check found inconsistent free-block metadata. A conservative repair left errors unresolved; a more comprehensive repair corrected them.
- After checking the filesystem, an administrator ran
resize2fsoffline on the storage node. The filesystem then grew to fill the 6Gi device.
The diagnostic reasoning is the same with Longhorn: compare Kubernetes capacity, the storage-system block device, and df inside the workload; then use kernel logs and filesystem tools to determine why growth stopped.
The vendor operations differ. Portworx uses pxctl for volume inspection, attachment, and filesystem checks. Longhorn exposes volume state through its Volume custom resource and UI, and presents an attached block device as /dev/longhorn/<volume-name>. Do not copy Portworx commands or assume that a Longhorn volume’s robustness field checks the filesystem; it does not.
Longhorn-specific notes
- Longhorn does not report ext4 filesystem health through the volume’s
robustnessfield. A Longhorn volume markedrobustness: healthydoes not prove that ext4 is clean. Use filesystem tools on the unmounted device and inspect node kernel messages. - Do not delete or manually edit the Longhorn
Volumecustom resource to resolve a filesystem resize failure. Use the supported PVC expansion and Longhorn UI workflows. - Longhorn RWX volumes use a Share Manager. Longhorn documents automatic RWX filesystem expansion from version 1.8.0 when the Longhorn Manager, CSI plugin, and Share Manager are all at the required version. An older Share Manager left running after an upgrade can prevent that expansion. Check the version and Share Manager before attempting manual repair.
- For a normal expansion, let Longhorn and CSI perform filesystem expansion automatically. The manual offline
resize2fsstep here is for the case where the block device is already larger but an ext4 consistency error prevented the filesystem from growing.