Docker内部存储结构(devicemapper)解析(续)
dm.fs
参数dm.fs可以指定容器的rootfs的文件系统,但只支持ext4/xfs:
func NewDeviceSet(root string, doInit bool, options []string) (*DeviceSet, error) {
...
case "dm.fs":
if val != "ext4" && val != "xfs" {
return nil, fmt.Errorf("Unsupported filesystem %s\n", val)
}
devices.filesystem = va
这是为什么呢?下面是docker dm的维护者Alexander Larsson的描述:
Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.
参考这里
一句话,因为ext4/xfs支持DISCARD。这样,如果容器中删除了文件,空间就会马上还给Thin pool,因为Thin provisioning是支持DISCARD操作的。但是,默认情况下Thin pool是底层是稀疏文件/var/lib/docker/devicemapper/devicemapper/data,所以,只有Host的文件系统支持DISCARD,才能保证稀疏文件空间释放。
Host为ext3
# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dbyin/tlinux1.2 latest 8297f05d459f 41 hours ago 399.6 MB
dbyin/httpd latest 93e711fab1c1 7 weeks ago 412.7 MB
centos latest 61038e6e3195 3 months ago 236.4 MB
我们可以查看稀疏文件的真正大小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
yy_pool: 0 409600 thin-pool 0 13/65536 0/3200 - rw no_discard_passdown
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 633/524288 26115/3276800 - rw no_discard_passdown
我们删除一个image
# docker rmi dbyin/httpd
可以看到稀疏文件并没有变小:
# ls -lsh /var/lib/docker/devicemapper/devicemapper/data
1.6G -rw------- 1 root root 200G Nov 12 11:52 /var/lib/docker/devicemapper/devicemapper/data
# dmsetup status
docker-8:1-696417-base: 0 41943040 thin 928768 41943039
docker-8:1-696417-pool: 0 419430400 thin-pool 73 490/524288 18758/3276800 - rw no_discard_passdown
no_discard_passdown表示dm层不会将DISCARD传给底层的设备(loopback device),只删除映射关系。
Host为ext4
[root@yy3 ~]# dmsetup status
docker-253:1-8790943-pool: 0 209715200 thin-pool 339 876/524288 32432/1638400 - rw discard_passdown queue_if_no_space
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAABAwAAACtCAIAAAC7jnp5AAAgAElEQVR4nO1d27XbPK90Wf9SQapH1bAZlXIe9nmwLYHkDC6yne18mVl5SGgaBAcXEqQc3W6CIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAh/K9b2c8e+/bYq/2ks297W29r2bfltVQRBEARBEIS/B8u2P3brf2wbuWz7c7i1/bR1/nRsu4q1vVCHbPvbaHmS/DPM7Gh+bZhHyQWELNve1nulUBao+k0QBEEQBOGfxbLtn6sP4O60G/Hjh9xre2Gz+x5u1nZsuG1RtLZjW7+2sXwoA9txbT9XioTbi7wxXFHkN8b9LT0FQRAEQRC+BfPm8n68vW/L8/z73DAdJ9/Dnha0L6at/8KzLhjPv+0XTpWWQ53uX8GUzkG7zS7T/3j66WffLB/3v3fyhqlNwmfeujqIbNgv7eNHCYMdz0kln+yaebu37Ht7tlF/2Pf9kA/5If5wb77//S7+PovHNc5mDJOo1oA+3A8tQZO/ZfX0eBAEQRAEQfh7QU+gjx3e2u57o/MhoVt3Os7ab+RE9r5pfpypw5uEobH/Z7SdNgf0/WMzTE97jL9su93+mSLB7oz7LaXdFCLexiIBbCJzx/ZW+PRQFrPjvi252xrC2zH5J01HPWNHXJvlmfLDbpbsxr0vGq1hgjKK6cPGva2rrXZsj5KezriCIAiCIAh/K5zN5div3zg9urD22+3mFgmPYTNFgpVzfjQelD82lMN3j0051dPboB+/1+gEwnGJ5mPbXCQMJQjFYxPPfkMC7XhvTBUJDm/Pmue4RzmNMZPg8lMtEmzvxGUL0oePO6h6uUhwxhUEQRAEQfhb8VtFwrlFm/fo8+jPlsxG8b1FQmvD7wXcU39cJJjv92oMFxevYLLj/LSRO8yVIqEf7UGLz2epSBg1KmzA4xste3MyCi8WCXRcQRAEQRCEvxXZImHctJ17QdbefWSe6Yl/Ko1Hvz8nnzgSHx47j/Tsdov9b4iP647hUXaqBtRw2Zp5HMX+cPmNG0p2k9DW5M9wCW+8SOgGNN92+An8oa+Z7v8w3w1mwfTB49ornekyp6gnHVcQBEEQBOFvRXgC3Z2Cl9o7YcP5+eOf+Elz8gxS9lmO7umPNjxeTx4vAe3Hf4FqXuoAfmPd/5w1kn+0Oj+ovQRYJDyf7LrK28/Pfij3+J3CcvyEF5EwzXf4iPrDk+CtHTxv+96a/Ql0UB86+sBxjQUeA6HfNGf0fKMZBUEQBEEQvgMf/S9QnUGdPVX+ckM4wIuEj/xnpn8Av+GYgiAIgiAIwu12+5WXqWF0/yEl+UDntBD8ZWp/Lcwp/19Z4QiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAge1vbCf0h/vJn4czCvujXDLPBFyf0nuUn5L3fjvSvjopf4whf7Dq9itlO+95+5dvVf29DO9XcxySF2cfSPeB79EPJD29G81vbjOCeTH32W7evKAHz68tHL8rB85oeVOfVyavx0dnT84ez8mj5zu2/3G3n7SpUfJqeaB6qoyojzBsiTwC6M/3f5OZOF83+9Heof+ecFnP7v+yHT05Wb5Tkxr1KcFlBbT4v2uhEeyusmXwer+lTnO+Muwfn22l72TD7f6vrlr0f5der88O/0k+p835H7fxMvvZf3oy/HNeGxNkvvsu2I67WdiS/1Pum1Hf3XljBffdxl22e/WLbd7mmOj610K+fRf37ztKv/tv+0rVeY6O9jlsPswvTP8Xz6IeOHtbN5MRegcoi9KPp5sfnO+5jRLgRJux+fUj909QnnVeTntCP1h8enWR6oPqTdC/1l2+H2vcQPkVPNA2UU9WT+4+dJYBfCsxNHWI6v6sQPyzPVdu4nnn9ehfV/LJLqyUWW8kluXtk4LaC4nlbtxXgor5t8HX9Fn4ynw4EvLcsVsAEIn8H6O/EZtE9y7sjnh2/zk+q6fGV9+X2YCqmtXZFgSidSce6bTSP3v3fyevk/k/B9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh0X7kh6MSwpg5HAYk/1f/ji8mqRgOWQuTD9qZ7EDxk/lDenSFjBGYQnBxbM5tB42Pl1lCP9hyKc2gXFixEwJl8m3/aBvHX9ybievSA/PJ8cGCyEeSA8M31YO7O7Pfu3U/btNec9Jmeg5LJfVf2N6Ynoc/UkdmE8O3ms5ueMn15PmGcy7Zk8P/sn5BO3E/9nfkj1v2R3Pz93H9TjFK6bDOX11CBjr0zey6ybyXU81Kc23+5YuuN02fa2gSPx4xvznXxrlaNpvtkN1y+wP2H5xF2nUvkhgS/yk/S6nNHn2zAePtprR1b9/BgDWKc1RcLJw7A+duvB857m/vfCme2wqIBYswPMK3eiSAiNVx13bXtraJHoVgM80ExNWCTMJ/HOIp27OMFyehVC/Yme1A85P6SdzKt7HM4eazA5zF7ramXCE4suBEg749OLl5E+d1w7tSHoUH82Lo0LzI9jRyNk2KtAvyI8M31YO7c7CXPCj5P3eLqo5AHqVzV/8/QcafL0ZHbheRLHUdnPaZ7sVJhbk+1xnp9OWCCfpN1bT6kfQv2Ldo/z81iC1OL07HmtSChshnL28vIe059s/oJ1MKXPhfmym4TDT+ZNOWyxPhCfTLP5Uj7d/Um6SKjmhwS+yU/y63Koz9dhmNtB7kTcs6NX+jxYpDs24JeeJ0F7+LySe77MtQ7fvLr6pMdd7CrRO7GpsNusJo6ddJEwTICFtN1jwPkGcrhdxr0f1JP54Y3zE/HWz+s2LMunVkQOtVfv0rjs2rurU9iO+fTjZSbKHdfSMJ8/df35uF7xOfPj2HFm3uGB83yhSMB2J+phfvwjn2hPkMoDzK9q/hYdTbk3CdPOEdjFWYxhHFX9nPJj5sz2Vpn2cDOXO+oj7dz/PT+E+hfzTJjnhxqhGqdVXCwSKvby815y3Rw+HX0orc8biwRqJlIkOP0D9PPFfPrrbL5IKOWHEN/mJ5V1OVyvvwzvLhJaa7PtsqNHAAdjSGKl3faIn2t/ZdzutMwcpHUcIELYI5NTkRDp702qcu01yfHtMt/dAz35osX4CXk7PniqMY4ARjb/IPbq9tuc0YofTmkeSpw1fHXcrp2Oy/wK85PYfOQeVqM8M31YO7M7U69HzE9RTtWvqv72Af8Zd82M/ygec37O8uSN55lSe5gn5zMNpOaFIoH6IdLzhTxDevP7oGKc5nBhPb1mr0evpP4V/y/pc2G+v18kJNw7iOt0kVDMDx6+zU+u9A/avwvjbzx+cFKzTtPdkfe/VX1sFKbfkxD6Kj5tdbOwj0Ut5rhmfKBk3Kn+zHcUR3lLLgZeHXezP90xmwYvyJm7o56B/r0opn+MadOAn5Ji+mM9qR8yfli7Y5dTT/OBwz+w13T0S4xTaZ/49M8w/JNgE4yO/4P+dFzuV8ifqR0PaXQlGPZNjGemD2lndh9HQeD89IziPVYlD7D51v3N0/MG/Cf2k8FkhOd4s5L2c5gnWZ6ptgd5EpyOYT5Ju7OeYj/Eer6SZ1InRNfitPC4UXU9vWgv1p5eN9l6UdanON+b3WKZwPt0kZBZ9wtxnS4SqvmB4dv8hPUPxnXbvw/2vuf+E5hj3SLXnfDhrOOW+JBn922TpPGxloCsXhmqT+c9zrN0ONmdcnKmuzruMEJngNCHR+K6Uzeq/2wUrr+HUY5rF6g/1ZP5IeWH8jbPa20/Pz/mUeeOBod/YC8z44dAs9qX1ER8Dvr/YPGdyah84odcHzgutRfhh9vxwR5LlwMPlGeuz9zu2J06bo4foIxDRCIPsPmW/Y3oyfOGo2fkn8zfuKOEfo75YXmm2u7p//iel68S6yD0f+qHXM8Ldmc843ldidNKkVBaTy/Yy+Xh4rpJnv1L6VPfPzhu3tbz0+PkddaI9/eA1/3MOkLc3IpK5ZlcfgD4Nj+5sC67+giCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAgQ6/DaxRKONy5/Dscb77phFvZCTPNJblJn/8yb8K6N67+j1ogZXi04vyJ45trVf21DO9d/xsMziHGxXQJpZNh38DPz4Olf4YHJd+Do2c2vf91pyW9PIVifUf4l/5lAeKPzjXlG+Yf5ytQexkvavD4Az8z/HT4TrzeN33rK/KQaXxf0+Twcf65Kyc2r6s9cfi2OOKB9E/lkxDxXP59fyj+unBwP1Th6z7rJ81JVn9ftfpfgfHttL++xMvuWDJ9czlvWdwff5idsPU2vXX8Z1vZCXtj2DxYJJjzWZul13iD/nMuS0Wxt9tXisfnq4y7bPvvFsu02dx8fW+lWzqP/2sYZufpv+0/beoWJ/gyMQmoXAjbfG+FnmNehBOOH8UBdoMhD1U+onoNQ+1r7kt8+RpnsS+Rf8x8Awhudb4rnLv9gfyDtQbw8pL+almeeqf87fC7bjtIsmy/sT/2ExEtVvtf/84j8OYfSvKr+zORX44iA2TeXT6g+Rg7N5xfyTzhuyEM5jorrAs17fB1/RZ+M3eHA1eWoDDYAXTeL+byYf6r5+dv8JLmeZvLzF8NUSG3tFmlTOpFKcd+s29z/3snr5c8nIvu2PD8vhMYyWDj66pTs7jMYJmW6ZCK1Oi5fsewRFHQaMNLklI7+D19cXi4S1uDOYBAJ/YTON8EPK2JtF8YD1Z/zAPX3eYZxAfV04s52wSc308kxtq8jP+8/5rB03CFF/jPOF/ZnehJ/oO1kXM/fOJ/QjpjnXjVDG+bTnj13Q5N50f79uHaRw/FSlc95dvgptL/gz11/4p/leRX9mcm/kB/CvJFcF8J8a/0hmc/DdZOhvJ7244ZxVF0XkLisbrm4rsy3O5buOF22vW3gSPz4xnxn3lrlaJoWRbH+mXxezT+Z/Q/DF/nJ1J+Nm9Hn22CLme66f9tp9dPf0MIi4eRhXJfn/H5vWVs6jfTMdvE2yHimzdESiSIhNF513LXtraGk3K1yeKCZmnCTN5wcz3I8/QG6x8ls+d+r0M0K+wmZL+PH+F7mCJDxQPUnPDD9mXwnLpCeNO46irqiGscRsa8vv+A/t3U1sYxWLeY/KX/jenJ/IHFExiX+xvhkdqRxZAfq/J/nE5henHnRdITzDI6XqnzSn8dFrb3oz3wdYf5ZnFfZn4n8an5w8wZbv4A+VA7xBzefF9ZNhvJ62n81llNcF5C488vuOpjS58J82U2CzTnz1nNuOdRObbLpehfpn9w/lPNPtP+h+CY/cW8SBh4Cfb4Ow9wOcifinh290ufBIt3BA3+afHo8oIHP/VFeyX1V5lqHOFmkT3rcxWbl3olNJd3gMRKQnd7kDRPAPJz60/kOy8io1WgX7idkvpQf60Q7eARh0MQpEjz9Zx7oMSrxkyjBDXt4HHeTOmaLiu2C7RvKzxcJ/dBse5x+fKPvz/Vk/uD4CR4X+hvl07G7G0dTXqoWCd68oj0HutYf46Uqn/Rn/BTbq/7srCOOf1Z5Nlqn/LlQJFR5G9WZ8/Z8NsK+Dv0hkw9feRziYpFQiaPSuoDEgU9Hu6f1eWORQJcJUiQ4/QOY+Yb6p/L5s2ch/0T7H4xv8xPUn43r6/N9eHeR0FqbbZcdPcJ00IIlVtptj/h3Dq+M251+mYOxjgNESOJ2MKe/H+XxIxyjp5jYnu2S22SbfzB+Qj3nc2PIg6M/ku9tFpH8DIG8PKaLSqxPJ8Qc0NeLBDIvY1nuQbFdcH+/SED+EPrJ5A/Q317YtE0swLzkxSPbvLJ5xXuO6/mnqM+bioQyz1yO459Vnl31gD/jIqGUH5LPG4T6lOUU82EZF9bTchy5eibXTUdOSZ8L8/39IsHMN9Q/lc95++X9z4xv85Mr/YP278L4G48fnEQ6Z+qsamqCx1fA70kIfRWftrpZ2MeiFnM8Yk9K1qFyuaFrU1PGZn5Ac2Hcbe/q9uO7XpAzd0c9A/17UUx/hntgnqLsyZ9/Cv7sdf+CM1/IzyAzPgcgPDD9GQ/Uz4l8Jy6Qnk7cYf+JT/UGZon8oynlP9OVguXT85/JLqS/pyfzB89P0A4e+hvjM7DjNATzfy8eyaafzgv1d/Pb2W6bS/JZf8bPa+0IE8+gP/fP6ryq/kzlF/MDaw/sC07xIr9N58PauslQXE+vxJH5brwuPLsOXuTk/5o+xfne7BarW14+WyRQP/f1z+bzDpn8U9X/2/wko3lBny+Fvd+9/wTmuEwg17jw4azjNumQd7DWXxUfuRFeHhNM181Yn857nGfdcLI75eRMd3XcYYTOAKEPj8T1myem/2wUrj/A2n5+fswjhd3DrAm7ZOeL+PHowTE+80D1d3lgDyEynp2HFoGeLO64/6A4GoWdEyPyq/5jLPwgsA95oCeeL+tPebiReHHa4bjUgRifjh1HnpP+3xfn9AtoXrw/8RMvXkryKc9xXNTbY38m/Zl/vjSvhD878qv5Ic6T1fiK8qebD8vrJkNhPa3HUXVdSOW9vuCs6VPfP8xueAzb1vPT4+R11oj39xCvd9V19oX8Exhy1qNql4/6idOfjVuZriAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgiAIgnDH2r/utIbjjcufw/HGu26Yhb3I0nySm9TZP/MmvGvj+u8U5m9mnV+JOXPt6r+2oZ3rD+How0Dfiwv1uSin0D+Sw/x/bK/x7PoDfP3irOdDA4fyypyv8OB9Z+yP45S31+KOg8kPdX/D2AgX7Fh+Hec0xhvjlPHJwOxYzVfXx83FV21deGueTMZ7pn0mp9pe1aeKalxX88N71k1u36o+r+exuwTn22t7wx6ruj+p5tVq/6q/fZuf+HH3+ny/DWu7WiTcbrdt/2CRYMJjbZZe+Mbwe6fnXJaMZms7+q8tYb76uMu2z36xbLvdQxwfW+lWzqP/2sYZufpv+0/beoWJ/gxMH4pen3m/OOpzSc4MxidrHwfD/m/aizx7frhs+zQc9wc6dehXtD3FZ8cDkw/7szil8VuNO6pBSn4Yj+/CBTtCfwiGmPR/V5xSPnNyxrhL56vquNX4Kq8Lb8yTlXhndmF8VtvL+lRRjOtyfiD8lNdNvo6/ok+GNzhw0d3KqO5PGA/MT6r9q/72bX6SzG/np8X9zHfAVEht7TYHpnQCldO9erJudv97J6+XPx837tvy/LwQGstg4eir02Jwn8EwKdMlE6nVcckOwIphixYYaXJKR/+HLy4vFQmOPtAfRhXM3LE+l+T4/Yfgxzwz/yftZZ4N7Lj2DM+GBtNz2/dtRWcobGeZ8LeOT4cHJp/kDTIWbvf5BHmG5BMm35sv+XptXNJetSPzB/tBuOMcQNgcKyUWX6QP5ifIn5V8hcfN2T2ML9bf45meKCA/MbDfrMZ7YBe46a+0V/UxX0otG+X11CCTHzJ+m1k3k+t4qE9tvt2xdMfpsu1tA0fixzembf1PaxePpjP7k6E/zKts35Lpn5GT0eeX/QQV53DcjD7fhuHw6EwW206rnx9jAOu0pkg4eRj3Y5aU5z3N/e9pB++Z7eJtWprwCpEoEkLjVcdd294auijrVic80ExNuOgap3xoNxcJVP9o4sPeA/qD8RnrEkyfqhzuh4xP2M78n8ZFmWc79GBf7GZY/+5xvv74BfsVb0d80vkSOby/nUh4LcP5ZPb18gkcl/gP46c6LtenaMcb8Yd1Nbm0OyEj+cRI67b2VA7mx3xzrMwRP0H+TOcrNq5r91J8kf6MH5InnXXw/OLAZSXeA7u8XCSU9Tk/ulQkFDZDqfwQ+21u3Uysg8l8VZ4vu0k4fGneNMMWu7bmNtml/YntEa6nF/qHcvgsvsdP3JuEYd2J8u23YZjbQe5E3LOjV/o8WKQ7+IlfkLnGAw743B/lldxXZa51iJNF+qTHXezuoHdiU0k3mMmB7PSiO0yALSF27+3PN18Kn8bfz0dwiD5VObw/4xO2M/9n7a/xPPoh3hRi/fvt5GkF5lfU3yCffL5YDu9vx2BrYH8sTeIuur7I5wfoh4Sf4riOPkU79mrQoe2pG7MvkMzlMH4InzTu3lwkDOOGdn/IrF22dP05P90XjjRWOXb19OH537PL54qEaD1K4mKRkM0PR1/CT3rdHD6FuSOjzxuLhO4gIFEkOP19pPYn5uPhVNXzk3z/a/72bX6C+rNxfX2+D+8uElprs+2yo0fABzaTxEq77dGvSTXbxeMOp1/4mh4Rwh6ZnK+PA/29SRWuvcZHD1JfRN2m9FaRk9usmH/g9itFwvt4RmsJ039+vPksEpBfsXasj18kzHLcIoHFKWwnfF4J5FJ+IPxUxy37IbPj4x/g7Lbbx3ZFgmvf6REhLMeZDuLTma8bF5V8VRrX0T+xVzP5pMZPrA+QUYl3Mm7U8+UiIadPjAvraSU/TKNNE0mum46ckj4X5vvrRUJmf3KjPFA/KfW/4G/f5idX+gft34XxNx7wGt1Opj+t6X47cmwgpt+TEPoqPm11s7CPRS3mmM6e2K1tqmrRGWQ7HTe+BLowbv+MyBmcXpAzd0c9A/17UUz/AKB6pP5g+4T6VOWw/oxP0s78n8bFBZ6pH4YHwN3mstPBGgz6ldN+wOjP54vl0P4sTlk745PZl+UTKh/Pl83ryriw/YId8QFe1znWHw0fyLFz6XyV2BHHqR8X6XyVG/dENb5wf84Py5NBvmLHB+l4t0jxeaG9rE/hcaPqelrND8N3c+fDgAfHvjV9ivO92S1WF0afLRKq+xPGAxu32r+q/7f5SUbzgj5fCvucxP0nMMdlArl+hQ9nHbdVh7xuXRy/MT7WEpA1XTdjfTrvKT9becrJme7quMMInQFCHx6J6zdzTP/ZKFx/DzCm+EO7V/XJySEPCbIvkHbm/6y9zDP2B+7QQM+1/fz8mEdG53NWrBFqT9HTzTch59mfxWkyflP2RfnEke/5IeatMK7XXrMjm4CZ2eOL0HeHmc1xSuVAfpL2Stixlq/S42Keq/EFn0mYeGZ5kvEA+S87aKLZqlRtvxQwlSKhtJ5eyA/u+nJx3ewLxZo+9f3DHL7HsG09Pz1OXmeNeP/UqPH+xOMBGaDa32tH+DY/cfq/Y7qCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCHWv/mtUajjcufw7HG++6YRb64svFfCEj/+yfeRPetXH9d9f270jvX+S3DDJmrl391za0c/0hHH3YuExPOmeoZx1Vu3sgesLXJmL/tN/gb2j2J/2ITBJcM/9+/6r/VO0by5/sAnmgPkLg9s/6VRh3XJ8uf0I51+zC/Oqqn4M87/Cfz+kOP/l4uZBnSvJL8O2Fx62vC7gdyfH0KebzW8av3pWHc3Kq+rxn3eS8VfV5nbe7BH8heHWPFfgJyJPMb3G8c/ksP1Tj9Nv8xF8Hk/r8RVjbC/uqbf9gkWDCY22W3vnN2ken51yWjGZrs6+yj81XH3fZ9tkvlm23ezuzwnVTPOQ8+q9tnJGr/7b/tK1XmOjPwPRh41I9CQ9UzyLKduegei7bjra52D9vtxubF5LDQKdC+Gf9y/5TtC9r9+yCeGBxQdHrOYye96ukn1NjnEUCllO1C/Or1/y8z/MO/yh+mUjKTyVeqnmmKr8Kj1oYv8V1gdqRyKH6FPM55ae6DroD5OWU9SH+Vl43ub1e0SfDGxy4aMY63AHmPMn8k+ZnJp/Yqxqn3+YnyfW0uI58G0yF1NZu8TClE6kI983mrPvfO3m9/J9J+L4tz88LobEMFo6+Oi2i9xkMkzJdMpFaHZfsCK0YttiDkcBmgur/8MXlpSKB6RPwBoON7IyxnszfzCEei7SBT+a3uJ3oac8OB5e249oZwHlROWRe275vKzhDYfyz/uhrnpyyff32p5huk4R4oHFB8snIrvmI+r9rxNu0kGD5PH9COVW7WLCQTfk50TPwZ7j4If9k/FyIlwOZPHMtHn1++nWKxR0Z98p6hNqdIgH6j+MckZ/br7rxDtZNhvJ6WtTHifcDmXUzaa9Qn9p8u2PpjtNl29sGjsSPb8x3a61Vjqa5bmz9tV+1m2man8lmOrRXdUvyRX6CDsvguBl9vg1DUdddl7Pq58cYwDqtKRL6RdGu75aU5z3N/e9p/5iOwUCs2QHm5JgoEkLjVcdd294a2hx0qy4eaKYm3OQZp3xo52ySSgdGs5s7vKFNBuSB6Un9bV2Nj5ELvX6nCOVw+cxeoXt0/kn5Z3LIvLrH+cwxDuOf9Tc6pvynbF+vncQj5hPHBc0nJkfZFMT49/LS0TqVepN8mj+ZnKpdhs/Glqyfu3o6/gzil8Qd4T+Qj+f1/FYyz1Tks3hn/uDZC457fT2KHzfy9KH9Iz8f+XHi/YUiobAZyunj+NvRmlg3E+tgSp8L82U3CYd952IStthcF59Mk/l66xT2W7JvYXyG9irvmL/JT9ybhG7cmIdvwzC3g9yJuGdHz5APFqnHAL8BJ2lO/1vEK7nny1zrECeL9EmPu9js3juxqcjbrCauYAubvE57tqXrnnp15zvtnWqbSMYD0ZP7W+9aJLuHcqh8ai84zf5r+Nhy5j/eZHRFArwHcooE797od4qE48MEDzAu3HxyfrifV+aYfz8vPaXNNd0on+VPLqdqFzs2y3uxn/t6looEHneQ/0A+n1chzxTk84KE+INnr3AvWFyP6Dpl5AT+M/QP/Rzkq8ub+0GFS3Iq+lB/e36aWTeHT0d7pfV5Y5FgC+9MkeD0D2Dmm/GkgvAAACAASURBVNknDI/N+PuWQf7Nt1d1v/xtfkIfqwbj+vp8H95dJLTWZttlR49gi2xHYqXd9kg8V/fCuMMpzhHsHQeIEPbI5LTJi/T3JlUo4ucbaG9cWCSQg2EkhBPb30KQmR1fv1QkMD3JGhD4J9x1gjNFPK/5Mf7jKBvyz/of4yT9p2rfoN183Ck38UniIumoqFvKr06Et8uTlngig5yqXW61vPfpIqEad558f16FPFOQX84Dnr3ivWB1uLg98J+xv+fnkP/X1sGX5LygD5hmct105JT0uTDf3y8S4jxJ+4f7Fip/ak/kt1HBr/KTK/2D9u/C+BuPH5yMrHP8dN7RHXbu2zK08dORW82nrW4W9rGoxRyz2BOX6Rcx6NrUlMOZHx5dGHezP/UxxYMX5MzdUc9A/14U0z8A2ul64+KyB/DAhsD+Zpfn/lCB8c/81vVnoife1Eb/yUymSHDn1T33bB4zgPzT/ueEcv5Tty9s9+IxPIA30nK3grH/B3LcZb4/gMP5k8mp2sXJexU/d/XMFwncPwedu/YL8VLKMxX5uTxwwrMXrglr64LTzuRAfVh/59aCPd3K473wuFF1Pb2kz422p9dNh7eaPsX53uwWyxj+00VCvO5P+wTon2zczL7C8hOvm9N3v8pPWP9gXLf9+2Dvme4/gTkuE8glJXw463g645Bn/WaSND7WEpA1XZtifTLPzJ0fjWOecnKmuzruMEJngNCHR+K6026q/2wUrr8HGFNwXEdPwkNCz/4HXofo+28HbHqH82IPFfKHDYGe2BFd/4TzYl+A81rb+c+JHsC/0/+C/5Tsy+Vju3DiaFygfOKHUexXw3eAnzP5LH8iOWW7eH5V9HOiJxuA2ZHGHeHnWrzk88wF+Zk8cP/EsZczQHFdSNixC65EHuD+AP43EYcHHAPZjU1hPb2gjxvvF9fNvgCu8lPdP8zLyzHs4XxPG4/qH2dlpL8HZ91HeTLhtyw/2/6we5QHRnybnzj9U+vF31EiCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIgCIIg/DrW4XWhJRxvXP4cjjfedcMs7AWa5pPcpM7+mTfhXRsXvWsYvn94eLWgnfK9/8y1q//ahnauf4SUnzj6Qz2d/s64ydcaPr45iWTtuflm4wX7Q8D/ZK+rfjKLn/kPeUjyyebly8d6En4uxAVDLd65ru64wI44j3F7sf5MfvfR0+Uu8FbSJxXv/E3bcbzH+WqMR6b/e9q5PvR98iRflfzhgh19+d46cnUzMMrJxRfTk8l5Dz/cjlV9ruSTHncJzrfX9vIeq55XHwNPX3H8fM5Ljnw3vwF8m59U86fL21+Atb2QF7b9g0WCCY+1WXrnN2sfnZ5zWTKare3ov7aE+erjLts++8Wy7XavZlaOboqHnEf/tYEw5vpv+0/beoWJ/jnEfsL0Z3rS/s64y7ZDNVA7E5lzWjbfmAfqDy7/wF49b4ecJM8J/jkPFT7ZvByTYj2pnGJcMFTj3Y9fMu5sR5rHCA+0P5E/TvIsEmq8sbzE9GHymf+X4z2Vr87+TH/Wzvin/Yk+tP8NxxHj813rQiAf9a+tmwzF+ErGRWivcn7g6/gr+mTyCRz4pWU5g2JevZG8R/P2Q9qYl5z84OS3Gd/mJ8n9WLwufzVMhdTWbtNjSidyUrJv1p3uf+/k9fLnk6R9W56fF0JjGSwcfXVKdvcZDJMyXTKRWh2X7GitGJaUwUgguVP9H744KFzNRsxPzOEY83g7UobnrpWMa8/krGux9m3ftxWcQbB2Ol8eLywu7FQyRQK012hS5Es8LCxvmH/GQ5VPr0hA/em8Ev6ZiQuWf6gfkv60ImXjsrjrVbPDhvYdJNG4Jv55IMNbJi+xmbEJp/KbE3dQKunP9M+0D8UG5oFvGmB/FkfOd9+yLjD5yf7huslQXk+JnkxONR8icVndMvrU5tsdS3ecLtveNnAkfnxjvqNrrXI0Xc2riX3LwH+Y95z8UN2SfIufuPmT5ZMXD+X/GIZDq+56mlU/P8YA435rGevOIa92yfF5T3P/e9o/puNkEGt2gDkjJ4qE0HjVcde2t4YuyrpdNh5opiYsEuYTNb6ZqBXwvZ/c1tXYGgmaw8LnuevPx2VfJ+3d43DmmI60s3GpPk5cYH8g/FN7mRjERzu5/ox/xk+VTzYv2p/NK/TPXFyw/MN5IPmKxi8el9qx/479V2Dfvj+R78bL8c0Eb3FeYisc3Ipm8xvXH/uDM1+mP2mn/JP+1D85b8GyMvD5nnWBy3f6F9ZNhvJ6SvSkcqr5EIg7v+yugyl9LsyX3SQc6WUuDmHLoTYrJucBCnmV5T3Cf5z3kvkwxjf5iXuT0I0b5/lvwzC3g9yJuGdHz5APFmkmnfj1VyZoD59Xcs+XudYhThbpkx53sbuk3onNKUKDx89AdrpIGCaAeTj1J/NlfnIbTTyrOuccP5kOewY6Lvk6ax8KmGMU3M7GpfqkEhz1Q+M/jr1OpnfwiMnsJ7C/UyRAfoZuB7z+aF5Of39eg5xwxNFAJH4xD7w/jV84bhh3KI95PEz9sXw/XvK83aK8xPOwc+IG/b8Q71bQs5an/Zn+fF6Y/yg/d/r4/Z2948znm9YFKt/v//jw+uMQF4sE4OdUTjUfTuLApzAmM/q8sUig4UCKBKd/gERedfIeWV/cvMfsUt0vf5ufoP5s3Hi9+y68u0horc22y44ewRbZjsRKu+0RP2/6yrjdqaQ5sOw4QISwbdO8KQn09yYV8eNsjumjI1j/UE9+7/BikTBKehYJqP0zRULNP4v2Cm9Dn3/D/DN+Hv9I88nGrfYP23NxQe1CeKD9WfyScftvgg1/IY8F/a38xCY7nU9oXvL1Sfoh7p8sEg45vD/TP8y3g57v7E+mA/l817rA5Dv9h4/5px4urKeEh4ycy/nBk1PS58J8f79ISPAW5D0uh5EAt+mV/2/i6/zkSv+g/bsw/sbjBy/q3Sajv3Oyv6nat2Voc08jKj5tdbOwj0Ut5rjSnlxOv4hB16bmuCbzw6ML43bPWJjiwQty5u6oZ6B/L4rpT0D8ZLpSyJxve3qO/al/jqMPQuDjMZ2qD7GknY1L9WFxwfwh5p+vNAneeH/CP+Pn8c8KnywuqHykZ8BPOi5o/iE8OPkKxi8bl6nK8tigc+czfv8hgzjxMinj6M/yUqDPJN/Nw4V4J/5A+zP9M5sqK5T1Z/5ZvRhhfL5rXQjsNfUvr5sMxfWU6pmQU8iHaL7QjmV9ivO92S2WIfrTRcKFvOrlvaf64X6DNWbyYar/L/kJ6x+M67Z/H+z90P0nMOjm3U4GPpx13Eod8mw+nSSNj7UEZPXKUH2yz3TiZHfKyZnu6rjDCJ0BQh8eieuqfKr/bBSuPwHxE2OZx7OLdnByt0D1BP3JuMwhYPv9aeXz0crz6Qzc7oxL20lcOP7g8A/s5blJyNu4bxvaHR5KfLJ5pXkmeiaemUnFRT8G80PWH8avM+5ttqOTxyAPbt4D8kc5nX8WeUMKRfqgeHfyYSHeb8wfaH/mWKSd+iH9IM7/0NnsJx6f71gXuPyU3XPrJkNhPU2u7wkzHvKu5AfyDG1Kn/r+YU4nx7BtPT89Tl5njXh/D6W8ihW9BfzD/QaQH+WTEd/mJ07/YvoRBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQBEEQHKzD6zlLON64/Dkcb7wb33VPXmG3mC9k5J/9M2/Cuzau/+5CI2Z4taCd8r3/zLWr/9qGdq6/i0lO3HscA843LWh+q/vYjv3kkv6wP5PvjDvw8Ii0SrA4/tANcjV+bxm/Im+MDuebHdt0h/PN8TbyUM0DKTVz/gz61+OO6e/OC/itH0MzrV4c8Tw2ZSr27UK7r2eN/6I/JOKOIRuPV/IhlT/aPVxH6uNmUVtPub8xOe9ZN3k8VvWpznfGXYLz7bW9YY+F7R7kpYJffZq3b/OTal76dNx9Gmt7YR3d9g8WCSY81mbpdd4k/pzLktFsbUf/tSXMVx932fbZL5ZtP9rsuFa6lfPov7ZxRq7+2/7Ttl5hor8PIMfBsu3oxe9wvr4YGE+wnfrJBf1RfybfGxfxUA0W5g89rscv8yvmz9X5BkNPdmTzzfF28lDOA76isOw3Y43nF3P/Ytw5/Dvzmv2WxZ1jd2ZfL49Nchjl1XYm38sniP+qP+TijiGOxwv50JE/253q7/jtW1BcT6m/MTlE//K6ydfxV/TJ2BEOfGlZLoD6mztw3q8+zdu3+UlyP/bn4u4jMBVSW7ukY0onUDndqyebdu5/7+T18n8m4fu2PD8vhMYyWDj66pTc7zMYJmW6ZCK1Oi7ZOVkxbBECI4FFner/8MXl1SKBykH2tWd49iM6XyKH7jgTO9FBU6y/URTvkzhR7BPbznjY9n1b0ZkI1+fAtPtD8cvk+PLnZNfPC36ame+N5Y2EHe18KW88jzH9qT5X/XksTiAP3J1YXmX6O/Oa/TbIM5HdhxWV9cRFArJXtZ3JZ/Ny/NB+t1u8r8bdmK+IH0L7UrswfVz5fr7iy2Yib5yfpZaN8npq0OcTLIfqb5BZN5PreKhPbb7dsXTH6bLtbQNH4sc3pmPHn9YKR9Pe+svzUt6v2Fjv4a0u/w/5iZuX7LgZfb4Ntpjpri+3nVY/P8YAY35fxnOmcR9oSXne09z/nnaPntku3gYZz4QH72T9IiE0XnXcte2tocWvy8p4oJmasEgwTvnQztnc55ILlOPZF9OI50vlMN4onx0lof63dTW+Op2wzv2JfK8d8dA9ntcd72B9OmnDOQmKXyrHl483iyyOzu+MGsITXJg3QjuO/gZ54zwQ/Zk+ZX82uXFIfbg/iTsnr0Z5bHq2Cvutm2e8IqG3r28vUCRgP6+2cz35vGgah/m5FHfMT6gfcvsS/bE+VH6cr9CRidEnk39eKBIKm6Gh8iFynLg7WhPrZmIdTOlzYb7sJuHwjbmYhy32ScbEyTTxN8JD2a/sQJ/hrST/D/mJe5PQx1ekz9dhmNtB7kTcs6NX+jxYpJ438etnfGgPn1dyX5W51iFOFumTHnexq1/vxGZj2uDxIZCdLhKGCWAeTv3pfLEc375kUwXmy+Uw3iif5mNyYdDz0A/NTsRhymA5fmwnRQK+1yH6gJ43Hr+OHF++e6IM4ig7X5o3AjtO/o95c3jA+hN9Lviz/dI+PeLjr3k27hKrI8tj084R+a2fZ5jdJ/sG9kJFAvTzajvV05lXxH/HZynumJ9QP+T2ZfpDfbifh3l+Xkew30Y8JHFx8wfyNpVD4+75aWbdHD4dJ5zW541FAk3vpEhw+gNE+417H7unrfrVo/VjvJXk3/6Mn6D+bFxfn+/Du4uE1tpsu+zoEaYDNiyx0m57ZJ/PuzbucCp5BHvHASKEPTI5FQmR/t6kKtdeU1ryeoITZThfdxOJeGPtt9BPBv37fUJ4gujIp+PiImGMvMcuI9Jnuq4kmxIsJ5IfhOR0OZOdb9m+pqXTEfOWKxKMGpcSRbyGTZec2f7JAKyovXBPQVttssihqtCxF3zcCPh5uZ3J9+ZV4r8Ud8wQ9SKB5kOoT2aCJM+HT0t449ZxYT2F/paTA+hNrpuOnJI+F+b7K0VCuN84PnA1OwG33R/l7dv85Er/oP27MP7Go3tmejHdzIlF52WmJnh8BfyehNBXKRKsbhb2sajFHEPZE6l1qFxu6NrUlNWZH9BcGLe7NTbFgxfkzN1Rz0D/XhTTP8a4+eD2DQ+GzSwcOZA31s78BGswHb2HvDH53rjkJqF7Tv34AY+vz7xpgvHL5ITy3ZNg68/V+bp5A9sXzZfw5uUxqj/Up+rPwxw7ZYndYdwxfZj+bn57SjeNQZ5Bhw7Mvp694ONGwF7ldibfmxfhH/BWjjvmJ9QPmX2x/lQfKt9RlTbOSgc8FB43qq6n1N8SckB7et104rGmT3G+N7vFMgH86SKB9Y/3Azm/+jRv3+YnrH8wrtv+fbD3SfefwBz7DHLtCB/OOm6hD3nW/yZJ42MtAVm9MlSfznucZ3Bxsjvl5Ex3ddxhhM4AoQ+PxPWbZqb/bBSuv4dAjhmbG4zOF8rhvKF210+g/uYbj2et4cDm/gLJ5+PCD+5PE5+PduOnHoA+8OzfCD/jl8lh7dyvkD8X5zt/FNiRzNfhjeYxHo9Mn5o/E3dO8ZDSh+nv5BkYp1hRbPcgjoC9oBxmr2o71ZPPK5V/4DMJybhj+Yr6YcK+/WEo1ofLJ3Yn+hO/dXmoFAml9TS5vuM1IfWceWrdTOQ3Z17V/YOzfLX1/PQ4eZ014v1To9L1N5VPHgqPxyIf5e3b/MTpz8Z19REEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAg1uH1jSUcb1z+HI433nXDLPRFrov5Qkb+2T/zJrxr46J3y8L3zQ6vFrRTvvefuXb1X9vQzvWvzvdGXiMI9XTmxfTH7+Ot6s9B3/f7spxHRJGgYHZk+tT8s/Ce6kBPX/7kmZeTAMg/b3s9JeECyq/GL9M/slc237r+OcV1VQ6JozD/ZPwqqxR/Q3xS2NC/GncX/PYCP868Svpj/wzyYdpP3pdXq/kKr+/FdaG8bvL5VvWpznfGXYLz7bW9YY9V5aG6/rJ1jfHJx7iV8tu3+U91fX/XPuTTWNvVIuF2u237B4sEEx5rG95fjzhd25lDl4xmazv6ry1hpvq4y7bP9l+23e7Vjo+tdCvn0X9t44xc/bf9p229wkR/Cqf/su2T2zA92byo/n372b+qPwHjvwyiJ3O9pB1jfgjovIp6BvKBfV9JAn3+QX51ATDumPxq/I7fPgSm7JXIt8wfbrcbjGsC6g8kjpJxGre7wPpX7Y76V+Ou6rdBfr7dbrPd2bwq+lP/dPNh3k/elVer+Yqu78V1obxu8nX8FX0yFMKB30U/Q5WHat5g8imfbl7N++23+c+L6/tnj9vLMJVQW7tFy5RIpLLcN5um7n/v5PXyfybh+7Y8Py+ExjJYMvrqVCTcZzBMynTJRGp1XLJCWDGsmAEjgSCk+j98bvlIkWDPwAYTQz0trESm/ziVQz7Tn/gba3f4Z34O25me275vKz9Dmfhhcqh9jQGGHS2c1wU9GQ9QfyonYRebfxy/wvoQHljcBX478ebt8LD+xF4s3ybsOBQVOK6L/pDJAzzdnvpQPZldiP5Vu7P+1bgr+22CH8sz0/Oi/pN8x440/8N5ef4A1k2G8npq0Ke34rpgkFk3k+t4qE9tvt3xc8fpsu1tA0ffxzfmu/fW6kfQlc3ulbzhrvvj/o2EFfbbBL7If9Lre0af34ItWrrr722nVc6PIdo6rSkSzvkOftXlo+d9zP3vJQc//9XF2yDjmYFHxhNFQmik6rhr21tDSb9bRfFAMzVhkWCc76Gds0i8+LiRQ5eTLOawgPwb3+tci+jD/I37Ieaf+Tn1f6Jn9xiePZZh/BA5nJ/VzP2nYxT6VVFPJ96h/kwO4Z/mn3GadlaYf8IDjTvHb/PxS/Un9uLz5fpDe9G4rvpDmAfmIgTGI23H+lD9b0W7k/7VuCv6ba8tOUMbOzt1Zk1/5J/EjoxnOq94XbtUJBQ2PUMlXFwXbqZzYt1MrIMpfS7Ml90kHHYfDq1Yi80J2RPoAg+X8oZXJPT7N5KfvfyQmBtW5M/7T3p9j/X5NQxzOEicCHp29EqcB1vUkyYegSeNByvwuT/vpIOcdofBwxd1V5/0uIvN+r2zmoWzzWriGEkXCcMEMA+n/pfme6lImPYeNMmeTrSTq8dDH+Zvjh9i/pmfe/4P9RyW93lHMfMD5VB++qnZzSXzq4qe0ZEG2mwBOYx/ln/gP5/jYU0IDzTusPxhsCh+uf7YXs58mR2pP5C4rvtDN+HEWsniEbcTfby8VLI77l+Ou5rf3qaeRjLluVgkePo/hgrzOeE5Ma9BThUXi4Rpfb+8LiTXzeHTcb5pfd5YJHQHHIkiwelPkefhWt5g8ic+WX5O7VsS8n/Zf9Lre6jP7+HdRUJrbbZRdvQI4CATSay02x7x83avjDucQpnrcjMjRAjbFs9394H+3qSq11vpZ8e5iefztwT/Ic9lByD8X3ekoduQ1TNFApRD+On2CVZ46Fc5PS8UCRU5bysSKA8s7oh8oJ8nxysSkL1of6o/06cTMtw7vMMfzEydVJiIx3BesAp5Q5FQi7uL/g/yGOe5ViS4+psBkFIJPynmsSourKdwfb+8LiTXTUdOSZ8L8/3eIqFbdy7lDbo5RgfxLD8fPXKbsW/znyv9g/bfwfhbDvh4xrBZ6bzG1ATHwjn9boTQVCkSrG4W9rGoxRy/2JOY8ZcyN3Rt2h0jxwcoF8bd7E+AzObDC3Lm1qhnoH8viunP4PW/UCSgnUHIv21n+jB/Y+2Mf+bnjv8zPfurfFx8QwwJDvAzHUXbzXG4eGT0DOYLH9uI5eApjjGO/Arrw3m4kbhj8ovxy/XH/kz6u/rbuQdxXfeHIA+4K3Qcp5l5pYoE1w+JHUtxV/RbqrwXd8WbBKgP8884n/eqsnm5cgqPG1XXU7a+V9eF41vJddNZR2r6FOd7s1ssY8jfKhIYDxfzBjrExHw6+Rlo4Ezou/yH9Q/Gddt/D/Ze5/4TmGPdIpeR8CGs49bokGf9aZI0PtYSkDJdj2J9Oi/hz4aSZHfKyZno6rjDCJ0BQl8dieuqcKr/bBSuf22+zDCOnnBeVH9OD9Mf+ZvTTgdgDxvidiTm/tT5+ciloZ/yk5mvaTeR8RgIWSBwN0dPNl+of14OUfPMP07AQ304DzcYd1x+MX5Z/kz589mf6u+lBxDXZX9w8wCIUyaGtLt2CReL2O6wfzXurvgt44cQweZV1d/xT8eOQf5nqubWTYbCeppc33Prwu3yukme/UvpU98/zOnkGLat56fHyeusEe+fGDPDw/iNOG9g+S6fZF/UfRAUP9/mPxfWd1cfQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRAEQRCEvxz/97//6Y/+6M+/8+e3U44gCIIgCH8Dfn3Loj/6oz9/8s9vpxxBEARBEP4G/PqWRX/0R3/+5J/fTjmCIAiCIPwN0NZBEP4FKNIFQRD+OczvZxU+gf8qz9o6CMK/gHykr+0HvKwxieONy5/D8Wa78V335FV1i/lCRv7ZP/PGu2vj+u9sJa8X/UGv3Jy5dvVf29DO9f8ojmFf95WavZj/cDnvsQvnuapPtT2PuwTn22v7cGynccF/WDwy/m+3G4gX572gBLF9QV6qxinQ03yEX32c9EMmn/Vn/LyLNxhfTjyysWv883xrPu++4PWv6OPyBvR8e974siIB7E9+NS+9tF/6AL5Nn18A8we9zthHKdLX9oKTffQ8xZh/bUOKRWZf2xkwqZOetR39x6wPUR932fbZRZdtP9rsuFa6lfPovzawHHL9t/2nbb3CRP8/gzecvRXtRf2HyenbD23LduF+8oo+cTsHVOhX3aGMvP+weKT83243FC8sTp1xoXw/L1XjFPQ33xsPU0p+SOSz/oyfd/HG4ovF443k2yr/NK6dQUn/kj4+b8DuH8gbX1Yk3F7cn3wA0udvwLLtosVFFOnmhKmtnZOZQxpwUnI/+bAZ8v73Tl4v/2cSvm/L8/PCFmUZVoboq9NicJ/BMCnTJbNjqo5LPNWKYVsfMNK0CDn6P9bd5XqR8Lgm2ozhl+FT4Cesnc8U2IWhbK9+dLNpw3JGCpHtMnZJ+kmoT7WdjvTTwfpe28DRIjy5f9w8tOwR8Z/ynwDsWwvYbo3xkolTZ1xI0CCnGqe4//nhdG5c8cNI/tif8fNm3vhm/TbEY2JnkOI/GJdssGEeqOjj8Ab1/ETeeKVIgOv7sNQ/NhI+LWR/4twonl/Yd8s53reYQ2Yriu5P+H6JkEDyHhnX0R+3V/ZvjP9qPh/kt/VxiWFvpMf7NLQPrPZ3wPzB3iGEonwenPXI5+ew1zV/wH5b5MeHH+n2kLS7rtp2o2t/6tDHCywSThKHfXI3n6df3P+e3ub1keldxz/pHUlMFAkh79Vx17a3hi7SO+/AAyUWLab/eSI1bz6o/gDdAynmmIv5CWs3Cr61SCjESec/VI7RGR4BZjcTMc8pfartDthNwuGWs3VgyzGdzObvj/gPA8sDz09NO42XME6d0XH+6eRU45Tr+Wh70Q99+WhejJ838uYWCVOJQ/Ltrca/Oy71Q3TTW9KH8cb0/ETeePUmYV7fJ27dou/m7E+M4jRT2Vsaum9ZV7NXwc+NnfpH+sxgeY+Ny/Qn7cX9G+f/Qj7vyiyjp72tMjfweB9Y7e8Da17ZHjAe/PXI18TaseoPzG+v8cPgRvoQoweb1Jm80vmxCtEd/LQ+gQwxFn7wOUq6i6THOvGTiyRpRvqkx10286ONfjNqEkSDHg4PCpNFwjABtqQ99afzHRLoQyvmJ9FicO0kODPfAJP/OHJO593BIx1Juwyfjn6S1ucPFAkwWRvBYwvq/9X+A/IA4J/ESxSnGG6+mlb4fJz6/aPnelJ+6Oozz4vx817e3Md+5toK5NtOWJp/PC7nGUVQSR/GG9Pzw0VCbh2MSLg9k899t+NvJ4CESU+UB6yqZq/FBuq3KPzgMKXPDJz3vHGh/qS9vH+j/Pv5f8Qw8f5ADcjh+8Bqfx9vKRKwPrXi9sbsWPQH4rdlfoL4/aNFQmtt3vM4mpdW+u4AgEustNse/TFd+l4jOe5Q7R3O3HHAsyoSPB5GBfp7k4r58WisJgAACwlJREFUwU84/2aRULYX9J+cHMBP0i6OnJI+1XYHnywSnEG/wX9i/icVkQK5xHUxX1XjdOqfIOc6D6w/4+fNvHEJg5Ys387iwPC5mwSPZ1gkVPSJeRv98/154/WbhFntte3bsrZ9W9u+rVG+ulIkdF8/T9zZAt0/4fKBIgHlPX9coD9prxcJjP+XigTzT3p4RIiq9vfxTUVC19vcJJT84fqGtoTwcSPLyQ+ZjE1e44NUx/cfX+na0OldJzXLtNXNwl4D2dsbdmVzNMx3FMdxTXQQd3Fce7eUreBZ/kA9A/17UUx/hufZw/HP44QY+wlrP0YkJ3PZM4OqvZj/ZOSA9rRdGM9lfartHP3zuO6JzjnI60XCH/AfPC6MR8q/HcOwWZ2vk694XgLjxnE6b22QF5T9kMhn/Rk/7+ItkIDiEebbKv98XDfa8HFPQZ+Yt+nq5O154xNFwm1trbV9W25r2/fwTIPuT+6Y80DX0N+8gX2L3T6iyzHf6HHssLzHx2X6k/by/o3x/8qhD34sqq/12T6w2t/Hp4qEaD2awe1Y8we2377GD0MU6d2tSDsfL+PXcPDh1uM29ZBn+J2vOcbrjyBZTNcrWJ/ss7B4M3rKyW1Tr447jMCuFx/zHppG4jqHo/rPRuH6s8maR2rZD4zwf+xn2/1rr0qRULJX0n9mXYj8i3bpE0FNn3o7xeSGhzptPT89Kv5ZU96f4o/4TzDbJP83HC+uQwzw5Dv5oRync3+nfKr6IZCfnFc+kNK8OfF1ozV7lG8TfLJxGc+uniV9PN4CP3lT3nihSODr+3KUq7l9Dt6f0DyANhvTfM0HxuMeCaqtrv5sv0TA8h4Zl+pP51XdvyH+L+TzTn5rDaSUxy903X1gvT9G0h8CMR4PfD8Mwe1Y84cb8dsiPwG+7/8xE/5CvPx8kPBPQ/7zR/DeAyaB4Z/gWVuH1/Fv5L03P/3yH8Z3+oMiXXgVpTsHQRgg/xGEvw7aOryIfyHvnWf4/905vgtf6w+KdEEQBEEQCtDWQRD+BSjSBUEQBEEoQFuHFKYf0rzlMXFhxLt4/jY5XwBFuiAIgiAIBWjrIAj/AhTpgiAIgiAUoK2DIPwLUKQLgiAIglCAtg6C8C9AkS4IgiAIQgHH1kF/9Ed//oU/v51yBEEQBEH4G/DrWxb90R/9+ZN/wpywhq8PdHC8cflzMK/mM8Ms7IV03bv8MvLP/pmfp18bF71zE76Hc/jlfP9O8ImEWP+1De1c/3jG3ftvPaPPbw115kW+7naB3EUSM/w7YPa9Zsd5XFefgh1xvLwPzA/ZuN680GfMnxn/vl1mPbE+xbiO/Hmy10W7z0KpVa/wj3Ah3p35Dl8r5YFem3GA6rzYd67Mt5JPruWqcXg3bw/51tM/kTd+fcuiP/qjP3/yTy4vvfCWh4++TM684nJtQ2pHOXdt51q+ZDRb29F/bYk0Xh932fY5rS/bbtck8y7ubord67vbelvbOCNX/23/aVuvMNGfgsj3qF22fXInNi8Gpwvk00GJfwZm36odBz7j9oe0rB1pvLwJzA/ZuA7P2I7E3xj/gV2ieDk/Lca178+zvSgPrt1nsLigdi/KvxLvznwn/qt5gOar6ryK/lbNP++yL/VbJ2/jfMuLhI/mjeJ6mvTbkM/yulnlIadPhjc4cHVZrqK6P6EKkfak/2fWBab/V/lJeX2p56svgDlRaGtXJJijBnJys2/Wze5/7+T18ucTuH1bnp/XdnthkTD0741xn8EwKdMlE6nVcVEGH8SwxRKMBDZnVP+HLy4vFQlM/rbv2wrOLu2Z1mB6Ni/oV0w+45PJucA/83P7XZtcSnYcTWQSN2y/vWDHMV6gUYzB6E0duhlz0tzACeaH2CUTj4xn0D7pSXm+kk8emP15thfjgepD7ELjgszR96vZz6vxzuZrBqR+kskDjg9QO0K/LfpbNf+U7VuNL6eIRfnW07+UN85BUstGeT0l4zI5jj8fyKybVR6YPrX5dtc4HafLtrcNXKEe35i29T+tXbhKe3w9zMMXigTo/xk7pg4LyFi/7Cfp9SWjz7fBFjOrfdxo22n182MMYJ3WFAknD4Mfgn3JvWVtaQfvme3ibZDxzJyjJRJFQmi86rhr21tDi2u3C8ADJRZdpv9Zqc+bS6o/AJPfPWZmjwVuEY1TJQP9isonfFL/LPLv+Dnxq6IdTUx1IUPaX7DjUPmTeFxXE7P4JHWM35u/+RsyIOGH2tGJRz+uc5stxv+VfPL8Yu/72F6cB+IPzC487sycQ/nMz6vxzud7m8T1yOUBahcyL+7nNX+r5p+qfcvx5fgnyrdU/2LeMFO7VCQUNkNDBUvksHi5mc4ZPyzywPS5MF92k3DYa940wxa7RuQ22ZX9CeOH8sbWU8ZPtG46s/geP8mvL6E+X4dhbge54Azh3tErfR4s0h08yO+TT48HIvB5ZMorua/KXOsQJ4v0SY+72GzdO7GppBs8rgay00XCMAHMw6k/na+ziNJzbTdXJktqJp/wSeUU+U+V+MM1YtWOZ3Ds20Db3F60YycLXyMM/tx/ZLZ+rv+zzd8cp4QfZpfMosviGrQjPRn/naB0PplrQGwv7idYH2gXP+5uiH8sn/p5Nd4D/+RFwpWjtd4uYF7cb6v+Vsw/Rfteji/EAysSPD+Bcm7Yf0q4WCRM4zpy/PhNrpvDpyEPf6BIOFvR+bR3Yu3PbkJqfzJ8gZRR9nlQ6P/OJt5fNzG+zU8q60u87nwX3l0ktNZm22VHj9AdLHGJlXbbo99L1WwXjzucEuFrfUQI275MRUKkvzep6/wMWStfJIx7KlokYPmET7dIKPCfvAc8u1204yQnbk/bEcULH8j0NCNEPNDk6O4SzT+4XTLxmOYtSDXVvAHaqVXmXbDrJ718Zhcv7ip50ikS6vFOmzz+k3mA6M/aL+SB2nwv5/OknpGEQc5TOeCWkb0mOQn/iXFhPSV542IeSK6bjpySPhfm++tFQmZ/wvqzdub/nLdMvHT4Nj+50j9o/y6Mv/HonmW0V6jmZLqzane4tW/L0EZP+55Ssz5tdbOwj0Ut5tjEnqCsbTrVnM9sumPk+BDlwrj9XfUZnF6QM3dHPQP9e1FMfwoi/x6w5xDJx41AFYr9ypEP+XT9s8C/ow+070U7PuWEBzRMFLMjixccj9ZS/SFNcAuHilU4rsMPtIvjb5B/N96DVDPIr+cT1779R5lF/dSH24XFBeOfzZfGy4V4R/P1Z1rKA2G+4vPqUPW3Uv6p2bceXx4P5CYB6l/NG8eH+f8bq7Se0nETckB7et0s88D0Kc73ZrdYXdh9tkio7k+cfAjbqT6En6r+3+YnGc1r6/s3wt4T338Cc1wmkEtQ+HDWcVt1yDtY669U798YH2sJyJquZbE+nfc4z7rhZHfKyZnu6rjDCJ0BQh8eieuqf6r/bBSuf2K+NpP+mEcZjXjPYCQ2Z79y5A9khP7J+1P+iRxm35odaXdPTNqOyXgxn5hvPAjvQ3vsj/3QG9eZWGRHUOG7/HfPqiI9XXXqce2sNcc3lqktRQ+0C40Lh//MfCP+o3gE83XyFeMt1scKyc2Lxdhr863kk6KeqTwPG40YX/9q3qgVCaX1NJmv0nny4rpZzZ/MjkmGJvc5hm3r+elx8jprxPunRs3tT+g+Ic6TbD1l+Twk7tv85ML6UpmuIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAiCIAjCfxb/DwLd9njqvnRlAAAAAElFTkSuQmCC" alt="" />
可以看到这里为discard_passdown,表示dm会将DISCARD传给底层设备(loopback device),queue_if_no_space表示如果thin pool没有空闲空间后,IO请求会被排队。另外,error_if_no_space表示如果thin pool没有空闲空间后,直接报错。
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.5G -rw-------. 1 root root 100G Nov 12 06:14 /var/lib/docker/devicemapper/devicemapper/data
[root@yy3 ~]# docker rmi dbyin/httpd
[root@yy3 ~]# ls -slh /var/lib/docker/devicemapper/devicemapper/data
2.0G -rw-------. 1 root root 100G Nov 12 06:15 /var/lib/docker/devicemapper/devicemapper/data
可以看到删除image前后,稀疏文件大小的变化。
dm.blkdiscard
docker还提供这个参数,默认值为true,即删除image后,会调用DISCARD,真正释放HOST上空间。
func (devices *DeviceSet) deleteDevice(info *DevInfo) error {
if devices.doBlkDiscard {
// This is a workaround for the kernel not discarding block so
// on the thin pool when we remove a thinp device, so we do it
// manually
if err := devices.activateDeviceIfNeeded(info); err == nil {
if err := BlockDeviceDiscard(info.DevName()); err != nil {
log.Debugf("Error discarding block on device: %s (ignoring)", err)
}
}
}
...
}
func BlockDeviceDiscard(path string) error {
...
if err := ioctlBlkDiscard(file.Fd(), 0, size); err != nil {
return err
}
...
}
func ioctlBlkDiscard(fd uintptr, offset, length uint64) error {
var r [2]uint64
r[0] = offset
r[1] = length
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd, BlkDiscard, uintptr(unsafe.Pointer(&r[0]))); err != 0 {
return err
}
return nil
}
主要参考
https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt https://github.com/docker/docker/tree/master/daemon/graphdriver/devmapper
Docker内部存储结构(devicemapper)解析(续)的更多相关文章
- [转]关于docker包存储结构说明
原文:http://blog.csdn.net/w412692660/article/details/49005631 前段时间与同事交流docker的安装包层次结构,并沟通相关每个文件的作用,但是一 ...
- Redis-基本数据类型与内部存储结构
1-概览 Redis是典型的Key-Value类型数据库,Key为字符类型,Value的类型常用的为五种类型:String.Hash .List . Set . Ordered Set 2- Redi ...
- docker的存储结构,和以前有了很大不同
在网上学习这一块知识点时,有一个URL讲得很详细, docker 镜像与容器存储目录结构精讲 http://blog.csdn.net/wanglei_storage/article/details/ ...
- 【SQL SERVER重新认识】数据内部存储结构简单探索
数据库经常需要打交道,但是从来没想过数据库内部是如何存储数据. 今天探索一下数据库内部如何存储数据,从下面几个方面探索 数据库内部如何存储数据 索引数据如何存储 操作数据对存储影响 总结 数据库内部如 ...
- docker 内部组件结构 -- docker daemon, container,runC
Docker, Containerd, RunC : 从 Docker 1.11 开始, docker 容器运行已经不是简单地通过 Docker Daemon 来启动, 而是集成了Container, ...
- SQLServer2012 表IAM存储结构探究
SQLServer2012 表IAM存储结构探究 Author:zfive5(zidong) Email: zfive5@163.com 引子 国庆节期间,一直在翻阅<程序猿的自我修养-链接.装 ...
- 深入PHP变量存储结构 标签: PHP存储
1.深入PHP变量存储结构 标签: PHP存储 分类: 编程语言(10) 首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PH ...
- PHP变量存储结构
首先声明,我并没有去读PHP的源码,只是对于PHP的有时候诡异的表现感兴趣,找了一下开发人员laruence的博客结合PHP提供的函数debug_zval_dump刺探得到了本博客所阐述的工作机理.如 ...
- docker存储结构解析
由于aufs并未并入内核,故而目前只有Ubuntu系统上能够使用aufs作为docker的存储引擎,而其他系统上使用lvm thin provisioning(overlayfs是一个和aufs类似的 ...
随机推荐
- Nancy总结(一)Nancy一个轻量的MVC框架
Nancy是一个基于.net 和Mono 构建的HTTP服务框架,是一个非常轻量级的web框架. 设计用于处理 DELETE, GET, HEAD, OPTIONS, POST, PUT 和 PATC ...
- css中的px、em、rem 详解
概念介绍: 1.px (pixel,像素):是一个虚拟长度单位,是计算机系统的数字化图像长度单位,如果px要换算成物理长度,需要指定精度DPI(Dots Per Inch,每英寸像素数),在扫描打印时 ...
- IPcamera
1.IPcamera 一般架构 IPcamera产品其实是一个小型的对外直播系统,一般通过rtsp协议将音视频数据传输到PC端播放器,通过rtmp协议将音视频数据传输到移动端播放器,当然有些还支持P2 ...
- 发现不错的cache系统Cache Manager Documentation
http://cachemanager.net/Documentation/Index/cachemanager_architecture https://www.nuget.org/packages ...
- Spring与Quartz的整合实现定时任务调度 以及crontab的用法
最近在研究Spring中的定时任务功能,最好的办法当然是使用Quartz来实现.对于一个新手来说,花了我不少时间,这里我写个笔记,给大家参考. 我使用的是Maven来管理项目,需要的Jar包我给大家贴 ...
- 使用 Elasticsearch ik分词实现同义词搜索(转)
1.首先需要安装好Elasticsearch 和elasticsearch-analysis-ik分词器 2.配置ik同义词 Elasticsearch 自带一个名为 synonym 的同义词 fil ...
- 密钥文件snk
1.(what)是什么? 由一个程序集的标识组成并通过公钥和数字签名(针对该程序集生成)加强的名称,其中的标识包括程序集的简单文 本名称.版本号和区域性信息(如果提供的话). 2.(why)为什么 ...
- 2015年11月25 Java基础系列(二)Thread Runnable线程初级讲解
序,线程是比进程小的进程,非常广泛的被使用. 一.继承Thread实现线程操作 1.注意setDaemon(boolean)方法,参数为true时为守护线程,参数为false时为用户线程. 守护线程的 ...
- linux下的库冲突问题
lib1.c #include <stdio.h>int fun(){ printf("lib1\n"); return 0;} lib2.c #include < ...
- nyoj 4 779 兰州烧饼
兰州烧饼 时间限制:1000 ms | 内存限制:65535 KB 难度:1 描述 烧饼有两面,要做好一个兰州烧饼,要两面都弄热.当然,一次只能弄一个的话,效率就太低了.有这么一个大平底锅,一 ...