|
Description
|
Category
kernel
Sub-Category
zfs
Description
When removing large files (around 800GB) on a ZFS filesystem, the space does not get freed after the file is removed. Scrub, export/import does not help the situation. zdb still sees the object:
Object lvl iblk dblk lsize asize type
5 5 16K 128K 1T 1.00T ZFS plain file
264 bonus ZFS znode
path???<object#5>
( I tried to file this bug a moth ago, but i never got any indication that it was processed )
Frequency
Always
Regression
No
Steps to Reproduce
# zpool create zpool01 c1t1d0 c1t2d0 c1t4d0
# zfs list -t all zpool01
NAME USED AVAIL REFER MOUNTPOINT
zpool01 72K 4.01T 18K /zpool01
# mkfile 1024G /zpool01/testfile
# zfs list zpool01
NAME USED AVAIL REFER MOUNTPOINT
zpool01 1.00T 3.01T 1.00T /zpool01
# ls -hl /zpool01
total 2147892153
-rw------T 1 root root 1.0T Dec 16 20:00 testfile
# zpool status zpool01
pool: zpool01
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zpool01 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
errors: No known data errors
# rm /zpool01/testfile
# sync
# zpool export zpool01
# zpool import zpool01
# zfs list -t all zpool01
NAME USED AVAIL REFER MOUNTPOINT
zpool01 1.00T 3.01T 1.00T /zpool01
# ls -al /zpool01
total 6
drwxr-xr-x 2 root root 2 Dec 16 20:22 .
drwxr-xr-x 24 root root 25 Dec 16 20:24 ..
Expected Result
# zfs list -t all zpool01
NAME USED AVAIL REFER MOUNTPOINT
zpool01 72K 4.01T 18K /zpool01
Actual Result
# zfs list -t all zpool01
NAME USED AVAIL REFER MOUNTPOINT
zpool01 1.00T 3.01T 1.00T /zpool01
Error Message(s)
zdb output:
# zdb -dddd zpool01
Dataset mos [META], ID 0, cr_txg 4, 1.07M, 158 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=<2:560000ee00:200> DVA[1]=<0:560000ee00:200> DVA[2]=<1:560000ce00:200> fletcher4 lzjb LE contiguous birth=1259 fill=158 cksum=a33ac32de:431a0555d5b:dfb1a673bb02:1f7e3b446d91a2
Object lvl iblk dblk lsize asize type
0 2 16K 16K 80.0K 70.5K DMU dnode
Object lvl iblk dblk lsize asize type
1 1 16K 512 512 1.50K object directory
microzap: 512 bytes, 5 entries
history = 22
sync_bplist = 21
deflate = 1
root_dataset = 2
config = 20
Object lvl iblk dblk lsize asize type
2 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Tue Dec 16 18:07:04 2008
head_dataset_obj = 16
parent_dir_obj = 0
origin_obj = 14
child_dir_zapobj = 4
used_bytes = 1.00T
compressed_bytes = 1.00T
uncompressed_bytes = 1.00T
quota = 0
reserved = 0
props_zapobj = 3
deleg_zapobj = 0
flags = 1
used_breakdown[HEAD] = 1.00T
used_breakdown[SNAP] = 0
used_breakdown[CHILD] = 1.07M
used_breakdown[CHILD_RSRV] = 0
used_breakdown[REFRSRV] = 0
Object lvl iblk dblk lsize asize type
3 1 16K 512 512 1.50K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
4 1 16K 512 512 1.50K DSL directory child map
microzap: 512 bytes, 2 entries
$MOS = 5
$ORIGIN = 8
Object lvl iblk dblk lsize asize type
5 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Tue Dec 16 18:07:04 2008
head_dataset_obj = 0
parent_dir_obj = 2
origin_obj = 0
child_dir_zapobj = 7
used_bytes = 1.07M
compressed_bytes = 365K
uncompressed_bytes = 365K
quota = 0
reserved = 0
props_zapobj = 6
deleg_zapobj = 0
flags = 1
used_breakdown[HEAD] = 1.07M
used_breakdown[SNAP] = 0
used_breakdown[CHILD] = 0
used_breakdown[CHILD_RSRV] = 0
used_breakdown[REFRSRV] = 0
Object lvl iblk dblk lsize asize type
6 1 16K 512 512 1.50K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
7 1 16K 512 512 1.50K DSL directory child map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
8 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Tue Dec 16 18:07:04 2008
head_dataset_obj = 11
parent_dir_obj = 2
origin_obj = 0
child_dir_zapobj = 10
used_bytes = 0
compressed_bytes = 0
uncompressed_bytes = 0
quota = 0
reserved = 0
props_zapobj = 9
deleg_zapobj = 0
flags = 1
used_breakdown[HEAD] = 0
used_breakdown[SNAP] = 0
used_breakdown[CHILD] = 0
used_breakdown[CHILD_RSRV] = 0
used_breakdown[REFRSRV] = 0
Object lvl iblk dblk lsize asize type
9 1 16K 512 512 1.50K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
10 1 16K 512 512 1.50K DSL directory child map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
11 1 16K 512 512 0 DSL dataset
320 bonus DSL dataset
dir_obj = 8
prev_snap_obj = 14
prev_snap_txg = 1
next_snap_obj = 0
snapnames_zapobj = 12
num_children = 0
creation_time = Tue Dec 16 18:07:04 2008
creation_txg = 1
deadlist_obj = 15
used_bytes = 0
compressed_bytes = 0
uncompressed_bytes = 0
unique = 0
fsid_guid = 23643114179568212
guid = 1763794244041509443
flags = 4
next_clones_obj = 0
props_obj = 0
bp = <hole>
Object lvl iblk dblk lsize asize type
12 1 16K 512 512 1.50K DSL dataset snap map
microzap: 512 bytes, 1 entries
$ORIGIN = 14
Object lvl iblk dblk lsize asize type
13 1 16K 128K 128K 0 bplist
32 bonus bplist header
Object lvl iblk dblk lsize asize type
14 1 16K 512 512 0 DSL dataset
320 bonus DSL dataset
dir_obj = 8
prev_snap_obj = 0
prev_snap_txg = 0
next_snap_obj = 11
snapnames_zapobj = 0
num_children = 2
creation_time = Tue Dec 16 18:07:04 2008
creation_txg = 1
deadlist_obj = 13
used_bytes = 0
compressed_bytes = 0
uncompressed_bytes = 0
unique = 0
fsid_guid = 21670240190795124
guid = 3211932234295913091
flags = 4
next_clones_obj = 19
props_obj = 0
bp = <hole>
Object lvl iblk dblk lsize asize type
15 1 16K 128K 128K 0 bplist
32 bonus bplist header
Object lvl iblk dblk lsize asize type
16 1 16K 512 512 0 DSL dataset
320 bonus DSL dataset
dir_obj = 2
prev_snap_obj = 14
prev_snap_txg = 1
next_snap_obj = 0
snapnames_zapobj = 17
num_children = 0
creation_time = Tue Dec 16 18:07:04 2008
creation_txg = 1
deadlist_obj = 18
used_bytes = 1.00T
compressed_bytes = 1.00T
uncompressed_bytes = 1.00T
unique = 1.00T
fsid_guid = 51673161574950800
guid = 8417046356846952011
flags = 4
next_clones_obj = 0
props_obj = 0
bp = [L0 DMU objset] 400L/200P DVA[0]=<2:560000c600:200> DVA[1]=<0:560000c600:200> fletcher4 lzjb LE contiguous birth=1259 fill=5 cksum=93c841e3e:3c768102bfa:c8ea55c8f7a2:1c3fa0f9f9340a
Object lvl iblk dblk lsize asize type
17 1 16K 512 512 1.50K DSL dataset snap map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
18 1 16K 128K 128K 0 bplist
32 bonus bplist header
Object lvl iblk dblk lsize asize type
19 1 16K 512 512 1.50K DSL dataset next clones
microzap: 512 bytes, 1 entries
10 = 16
Object lvl iblk dblk lsize asize type
20 1 16K 16K 16K 4.50K packed nvlist
8 bonus packed nvlist size
version=14
name='zpool01'
state=0
txg=1257
pool_guid=6871687200964306102
hostid=13281026
hostname='tank'
vdev_tree
type='root'
id=0
guid=6871687200964306102
children[0]
type='disk'
id=0
guid=10174457158611201760
path='/dev/dsk/c1t1d0s0'
devid='id1,sd@SATA_____ST31500341AS________________9VS04BQL/a'
phys_path='/pci@0,0/pci1043,82f2@9/disk@1,0:a'
whole_disk=1
metaslab_array=27
metaslab_shift=33
ashift=9
asize=1500288647168
is_log=0
children[1]
type='disk'
id=1
guid=2528046875237128862
path='/dev/dsk/c1t2d0s0'
devid='id1,sd@SATA_____ST31500341AS________________9VS0311L/a'
phys_path='/pci@0,0/pci1043,82f2@9/disk@2,0:a'
whole_disk=1
metaslab_array=25
metaslab_shift=33
ashift=9
asize=1500288647168
is_log=0
children[2]
type='disk'
id=2
guid=12400481499280375100
path='/dev/dsk/c1t4d0s0'
devid='id1,sd@SATA_____ST31500341AS________________9VS00GFT/a'
phys_path='/pci@0,0/pci1043,82f2@9/disk@4,0:a'
whole_disk=1
metaslab_array=23
metaslab_shift=33
ashift=9
asize=1500288647168
is_log=0
Object lvl iblk dblk lsize asize type
21 1 16K 16K 16K 3.00K bplist (Z=uncompressed)
32 bonus bplist header
Object lvl iblk dblk lsize asize type
22 1 16K 128K 128K 13.5K SPA history
40 bonus SPA history offsets
Object lvl iblk dblk lsize asize type
23 1 16K 512 512 1.50K object array
Object lvl iblk dblk lsize asize type
24 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
25 1 16K 512 512 1.50K object array
Object lvl iblk dblk lsize asize type
26 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
27 1 16K 512 512 1.50K object array
Object lvl iblk dblk lsize asize type
28 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
29 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
30 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
31 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
32 1 16K 4K 8K 9.00K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
33 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
34 1 16K 4K 8K 9.00K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
35 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
36 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
37 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
38 1 16K 4K 8K 9.00K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
39 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
40 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
41 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
42 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
43 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
44 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
45 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
46 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
47 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
48 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
49 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
50 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
51 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
52 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
53 1 16K 4K 8K 9.00K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
54 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
55 1 16K 4K 8K 9.00K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
56 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
57 1 16K 4K 8K 7.50K SPA space map
24 bonus SPA space map header
Object lvl iblk dblk lsize asize type
58 1 16K 4K 8K 7.50K SPA space map
This issue is also being seen by xxxxx@xxxxx.com on nfsrdma2-t2k.central, dataset diskpool/JUNK.
I'm seeing the issue with s10u7 build 8 as well.
# uname -a
SunOS ncs7 5.10 Generic_139555-08 sun4u sparc SUNW,UltraAX-i2
# cat /etc/release
Solaris 10 5/09 s10s_u7wos_08 SPARC
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 30 March 2009
#
# zfs list diskpool <== after some testing to this pool
NAME USED AVAIL REFER MOUNTPOINT
diskpool 7.79G 1.99G 7.79G /diskpool
# ls /diskpool <== there's nothing in there
# ls -Ra /diskpool
/diskpool:
./ ../
#
# df -h /diskpool <== but shown 80% used
Filesystem size used avail capacity Mounted on
diskpool 9.8G 7.8G 2.0G 80% /diskpool
# zfs umount diskpool <== umount/remount didn't help
# zfs list diskpool
NAME USED AVAIL REFER MOUNTPOINT
diskpool 7.79G 1.99G 7.79G /diskpool
# zfs mount diskpool
du -s -h /diskpool
1K /diskpool
#
I have been hit by this bug with a couple of non-holey files.
# zdb -dd bw
Dataset mos [META], ID 0, cr_txg 4, 2.77M, 139 objects
--snip--
Dataset bw [ZPL], ID 16, cr_txg 1, 768G, 11 objects
ZIL header: claim_txg 0, claim_seq 0 replay_seq 0, flags 0x0
Object lvl iblk dblk lsize asize type
0 7 16K 16K 48.0K 18.0K DMU dnode
-1 1 16K 512 512 1K ZFS user/group used
-2 1 16K 512 512 1K ZFS user/group used
1 1 16K 512 512 1K ZFS master node
2 1 16K 512 512 1K ZFS delete queue
3 1 16K 512 512 1K ZFS directory
4 1 16K 512 512 1K ZFS directory
5 1 16K 3.00K 3.00K 1K ZFS directory
6 1 16K 512 512 512 ZFS plain file
74 4 16K 128K 256G 256G ZFS plain file
75 5 16K 128K 512G 512G ZFS plain file
79 1 16K 1.50K 1.50K 1.50K ZFS plain file
80 1 16K 512 512 512 ZFS plain file
82 1 16K 6.50K 6.50K 6.50K ZFS plain file
If you would like access to the machine it's itfc-temp.uk.sun.com. Let me know and I will power it on.
|