nydusd 源码理解（一）

“

尝试通过 nydus[1] 源码理解工作流程。可能由于代码变动导致和本文记录的内容有出入。

1. 环境准备

git clone https://github.com/dragonflyoss/image-service.git
cd image-service
make

编译的目标文件位于 target 文件夹内，默认编译的 debug 版本。

可以看到，项目的二进制文件包含 nydusctl （命令行工具）、nydusd（nydus 主体程序，以守护进程的形式运行）、nydus-image（nydus 镜像文件处理工具）三种。

all: build
# Targets that are exposed to developers and users.
build: .format
    ${CARGO} build $(CARGO_COMMON)$(CARGO_BUILD_FLAGS)
    # Cargo will skip checking if it is already checked
    ${CARGO} clippy $(CARGO_COMMON) --workspace $(EXCLUDE_PACKAGES) --bins --tests -- -Dwarnings
.format:
    ${CARGO} fmt -- --check

执行 make编译项目时，会首先使用 cargo fmt -- --check 命令对代码格式进行检查。

本文使用的 nydus 版本：

./target/debug/nydusd --version

2. 代码流程理解

项目的入口函数位于 src/bin 目录下：

分别对应生成的二进制文件 nydusctl、nydusd 和 nydus-image，首先，理解最重要的部分nydusd。

“

Nydusd 是运行在用户态的守护进程，可以通过 nydus-snapshotter 进行管理，主要负责处理 fuse 下发的 I/O 请求，当数据不存在本地缓存时，从 backend（registry，OSS，localfs）获取数据内容。

nydusd启动命令：

mkdir /rafs_mnt
./target/debug/nydusd fuse --thread-num 4 --mountpoint /rafs_mnt --apisock api_sock

2.1 入口函数

src/bin/nydusd/main.rs

首先，从命令行提取参数值，开启日志。

接下来是解析子命令，nydusd 包括 3 个子命令，分别是 singleton、fuse 和 virtiofs：

对于每个子命令，都会再次获取对应的命令参数也就是 args 中 subcommand 的参数内容。fuse指定nydusd 作为专门针对 FUSE 的 server 运行，virtiofs指定nydusd专门作为 virtiofs 的 server 运行，singleton指定nydusd作为全局守护进程运行，可以同时为 blobcache/fscache/fuse/virtio-fs 提供服务。

2.2 FUSE subcommand 启动流程

process_default_fs_service(subargs, bti, apisock, true)?;
// 函数声明
fn process_default_fs_service(
    args: SubCmdArgs,    //提取的子命令参数
    bti: BuildTimeInfo,    // 编译时信息
    apisock: Option<&str>,    // api socket 路径
    is_fuse: bool,    // 是否为 fuse 文件系统
) -> Result<()> { 内容太长，省略 }

该函数初始化默认的文件系统服务。

首先根据三个参数生成挂载命令：

virtual_mnt 是挂载的目录位置。

（1）shared_dir 不为空时

let cmd = FsBackendMountCmd {
    fs_type: nydus::FsBackendType::PassthroughFs,
    source: shared_dir.to_string(),
    config: "".to_string(),
    mountpoint: virtual_mnt.to_string(),
    prefetch_files: None,
};

（2）bootstrap 不为空（只使用 rafs 文件系统）

检测是否传入localfs-dir参数，如果传入，则根据传入的参数生成配置信息，否则，必须传入config参数。此外，解析传入的 prefetch_files 列表：

let config = match args.value_of("localfs-dir") {
    Some(v) => {
        format!(
            r###"
{{
    "device": {{
        "backend": {{
            "type": "localfs",
            "config": {{
                "dir": {:?},
                "readahead": true
            }}
        }},
        "cache": {{
            "type": "blobcache",
            "config": {{
                "compressed": false,
                "work_dir": {:?}
            }}
        }}
    }},
    "mode": "direct",
    "digest_validate": false,
    "iostats_files": false
}}
"###,
            v, v
        )
    }
    None => match args.value_of("config") {
        Some(v) => std::fs::read_to_string(v)?,
        None => {
            let e = DaemonError::InvalidArguments(
                "both --config and --localfs-dir are missing".to_string(),
            );
            returnErr(e.into());
        }
    },
};
let prefetch_files: Option<Vec<String>> = args
    .values_of("prefetch-files")
    .map(|files| files.map(|s| s.to_string()).collect());

let cmd = FsBackendMountCmd {
    fs_type: nydus::FsBackendType::Rafs,
    source: b.to_string(),
    config: std::fs::read_to_string(config)?,
    mountpoint: virtual_mnt.to_string(),
    prefetch_files,
};

当生成挂载命令cmd后，接下来会根据 opts 参数新建 vfs 实例。

let vfs = fuse_backend_rs::api::Vfs::new(opts);
let vfs = Arc::new(vfs);

2.3 Vfs 结构体分析

/// A union fs that combines multiple backend file systems.
pubstruct Vfs {
    next_super: AtomicU8,
    root: PseudoFs,
    // mountpoints maps from pseudo fs inode to mounted fs mountpoint data
    mountpoints: ArcSwap<HashMap<u64, Arc<MountPointData>>>,
    // superblocks keeps track of all mounted file systems
    superblocks: ArcSuperBlock,
    opts: ArcSwap<VfsOptions>,
    initialized: AtomicBool,
    lock: Mutex<()>,
}

新建 Vfs 实例的时候：

impl Vfs {
    /// Create a new vfs instance
    pubfn new(opts: VfsOptions) -> Self {
        Vfs {
            // 下一个可用的 pseudo index
            next_super: AtomicU8::new((VFS_PSEUDO_FS_IDX + 1) asu8),
            // 挂载点，是一个 Hashmap
            mountpoints: ArcSwap::new(Arc::new(HashMap::new())),
            // 超级块，数组
            superblocks: ArcSwap::new(Arc::new(vec![None; MAX_VFS_INDEX])),
            // root，是一个 PseudoFs 实例
            root: PseudoFs::new(),
            // 传入的参数
            opts: ArcSwap::new(Arc::new(opts)),
            // 锁
            lock: Mutex::new(()),
            // 是否已经初始化
            initialized: AtomicBool::new(false),
        }
    }
    ...
}

next_super的值初始化为 1，长度为 64 位的 inode number 被拆分为两部分，前 8 位用于标记被挂载的文件系统类型，剩下的 56 位供后端文件系统使用，最大值为VFS_MAX_INO。

/// Maximum inode number supported by the VFS for backend file system
pubconst VFS_MAX_INO: u64 = 0xff_ffff_ffff_ffff;
// The 64bit inode number for VFS is divided into two parts:
// 1. an 8-bit file-system index, to identify mounted backend file systems.
// 2. the left bits are reserved for backend file systems, and it's limited to VFS_MAX_INO.
const VFS_INDEX_SHIFT: u8 = 56;
const VFS_PSEUDO_FS_IDX: VfsIndex = 0;

Vfs结构体中root的类型为PseudoFs：

pubstruct PseudoFs {
    // 下一个可用的 inode
    next_inode: AtomicU64,
    // 根 inode，指向 PseudoInode 类型的指针
    root_inode: Arc<PseudoInode>,
    // inodes，类行为 Hashmap
    inodes: ArcSwap<HashMap<u64, Arc<PseudoInode>>>,
    lock: Mutex<()>, // Write protect PseudoFs.inodes and PseudoInode.children
}

PseudoInode类型：

struct PseudoInode {
    // 当前 inode
    ino: u64,
    // parent 的 inode
    parent: u64,
    // children 的列表（PseudoInode 类型的指针）
    children: ArcSwap<Vec<Arc<PseudoInode>>>,
    name: String,
}

nydus 中 Vfs 结构体的组成图示：

回到新建 vfs 实例之后的流程。接下来会获取 daemon_id 和 supervisor 参数（在 live-upgrade/failover 的时候需要）。

然后，根据挂载命令创建 NydusDaemon。

2.4 针对 FUSE 的 NydusDaemon

is_fuse 为 true 时，开始创建 daemon：

（1）获取 fuse server 的线程数量值；

（2）获取 mountpoint 参数的值；

（3）创建 daemon

let daemon = {
    fusedev::create_fuse_daemon(
        mountpoint,     // 挂载点路径
        vfs,            // 创建的 vfs 实例
        supervisor,
        daemon_id,
        threads,        // 线程数量
        apisock,        // api socket 路径
        args.is_present("upgrade"),
        !args.is_present("writable"),
        p,              // failover-policy
        mount_cmd,      // 挂载命令
        bti,
    )
    .map(|d| {
        info!("Fuse daemon started!");
        d
    })
    .map_err(|e| {
        error!("Failed in starting daemon: {}", e);
        e
    })?
};
DAEMON_CONTROLLER.set_daemon(daemon);

在 fusedev::create_fuse_daemon 函数中，主要的逻辑如下：

（1）创建两个 channel

let (trigger, events_rx) = channel::<DaemonStateMachineInput>();
let (result_sender, result_receiver) = channel::<DaemonResult<()>>();

channel 是用于线程间通信，返回值分别为 sender 和 recver，例如：(trigger, events_rx) 中，trigger 为发送者，events_rx 为接收者。

（2）创建 Service 实例

let service = FusedevFsService::new(vfs, &mnt, supervisor.as_ref(), fp, readonly)?;
impl FusedevFsService {
    fn new(
        vfs: Arc<Vfs>,
        mnt: &Path,
        supervisor: Option<&String>,
        fp: FailoverPolicy,
        readonly: bool,
    ) -> Result<Self> {
        // 创建和 FUSE 的 session
        let session = FuseSession::new(mnt, "rafs", "", readonly).map_err(|e| eother!(e))?;
        let upgrade_mgr = supervisor
            .as_ref()
            .map(|s| Mutex::new(UpgradeManager::new(s.to_string().into())));
        Ok(FusedevFsService {
            vfs: vfs.clone(),
            conn: AtomicU64::new(0),
            failover_policy: fp,
            session: Mutex::new(session),
            server: Arc::new(Server::new(vfs)),
            upgrade_mgr,
            backend_collection: Default::default(),
            inflight_ops: Mutex::new(Vec::new()),
        })
    }
    ...
}

（3）创建 Daemon 实例：

let daemon = Arc::new(FusedevDaemon {
    bti,
    id,
    supervisor,
    threads_cnt,    // 线程数量
    state: AtomicI32::new(DaemonState::INIT asi32),
    result_receiver: Mutex::new(result_receiver),
    request_sender: Arc::new(Mutex::new(trigger)),
    service: Arc::new(service),
    state_machine_thread: Mutex::new(None),
    fuse_service_threads: Mutex::new(Vec::new()),
});

其中，FusedevFsService::new() 函数会调用FuseSession::new函数，创建和内核 FUSE 通信的 session，只是还没有挂载和连接请求。

FuseSession::new() 为外部 fuse-backend-rs[2] creat，对应代码如下：

创建好的 session 实例存储在 FusedevFsService 结构体的 session 属性，同时用 Mutex 包裹，只允许互斥访问。

创建好的service 作为 FusedevDaemon 结构体 service 属性的值，使用 Arc 包裹，允许并发访问。

2.5 nydusd 状态机

machine 是 DaemonStateMachineContext 结构体的实例，存储了 daemon 的 PID，指向 daemon 实例的指针，以及接收请求和返回结果的 channel，用于线程间通信。

let machine = DaemonStateMachineContext::new(daemon.clone(), events_rx, result_sender);

nydusd 的状态机用于维护 nydusd 的状态，具体的状态转移策略如下：

state_machine! {
    derive(Debug, Clone)
    pub DaemonStateMachine(Init)
    // Init意味着 nydusd 刚启动，可能已经配置好了，
    // 但还没有和内核协商双方的能力，也没有尝试通过
    // 挂载 /fuse/dev 来建立fuse会话（如果是fusedev后端）
    Init => {
        Mount => Ready,
        Takeover => Ready[Restore],
        Stop => Die[StopStateMachine],
    },
    // Ready表示 nydusd 已经准备就绪，
    // Fuse会话被创建。状态可以转换为 Running 或 Die
    Ready => {
        Start => Running[StartService],
        Stop => Die[Umount],
        Exit => Die[StopStateMachine],
    },
    // Running 意味着 nydusd 已经成功地准备好了
    // 作为用户空间 fuse 文件系统所需的内容，
    // 但是，必要的 capability 协商可能还没有完成，
    // 通过 fuse-rs 来判断
    Running => {
        Stop => Ready [TerminateService],
    },
}

machine.kick_state_machine() 方法用于启动状态机线程。

let machine_thread = machine.kick_state_machine()?;

该线程的名称为state_machine，通过 top -Hp NYDUSD_PID 可以看到：

该线程是一个死循环，用于接收来自 channel 消息。（消息从哪发送？）

self.request_receiver.recv()

其中，recv() 函数会阻塞，接收 DaemonStateMachineInput 类型的消息，保存在 event 变量中，self.sm.consume(&event) 方法处理每个 event，完成相应操作，并修改状态为新的值。

处理完成后，通过 result_sender channel 返回状态消息。（传递给谁？）

然后，会打印日志信息，包括上一次的状态，本次状态，输入和输出。

启动 nydusd 时打印的关于 State machine 的日志信息：

状态机线程接收的消息来自哪里呢？这就需要回到创建 channel的地方：

和request_receiver对应的 channel名为trigger，和result_sender对应的channel名为result_receiver，都存储在daemon中：

let daemon = Arc::new(FusedevDaemon {
    ...
    result_receiver: Mutex::new(result_receiver),
    request_sender: Arc::new(Mutex::new(trigger)),
    ...
});

这两个channel 在on_event函数中被使用：

impl DaemonStateMachineSubscriber for FusedevDaemon {
    fn on_event(&self, event: DaemonStateMachineInput) -> DaemonResult<()> {
        self.request_sender
            .lock()
            .unwrap()
            .send(event)
            .map_err(|e| DaemonError::Channel(format!("send {:?}", e)))?;
        self.result_receiver
            .lock()
            .expect("Not expect poisoned lock!")
            .recv()
            .map_err(|e| DaemonError::Channel(format!("recv {:?}", e)))?
    }
}

因此，state_machine 通过 channel接收来自nydusd 的消息，从而改变状态，例如，对于stop操作：

2.5.1 FUSE 启动 service

上面提到，state_machine线程会改变nydusd的状态，对于 StartService 事件，会运行 d.start() 方法，并且在运行成功之后通过 set_state(DaemonState::RUNNING) 将 Daemon 的状态设置为 RUNNING。

let r = match action {
    Some(a) => match a {
        StartService => d.start().map(|r| {
            d.set_state(DaemonState::RUNNING);
            r
        }),
        ...
    },
    _ => Ok(()),
};

不同类型 Daemon 的 d.start() 方法实现不一样，对于 FusedevDaemon，start() 内容如下：

fn start(&self) -> DaemonResult<()> {
    info!("start {} fuse servers", self.threads_cnt);
    for _ in0..self.threads_cnt {
        let waker = DAEMON_CONTROLLER.alloc_waker();
        self.kick_one_server(waker)
            .map_err(|e| DaemonError::StartService(format!("{:?}", e)))?;
    }
    Ok(())
}

这里会根据 threads_cnt，开启对应数量的线程。其中，DAEMON_CONTROLLER.alloc_waker() 只是复制了对 DAEMON_CONTROLLER.waker 的引用。

pubfn alloc_waker(&self) -> Arc<Waker> {
    self.waker.clone()
}

kick_one_server(waker) 是 FusedevDaemon 结构体的方法：

fn kick_one_server(&self, waker: Arc<Waker>) -> Result<()> {
    letmut s = self.service.create_fuse_server()?;
    let inflight_op = self.service.create_inflight_op();
    let thread = thread::Builder::new()
        .name("fuse_server".to_string())
        .spawn(move || {
            ifletErr(err) = s.svc_loop(&inflight_op) {
                warn!("fuse server exits with err: {:?}, exiting daemon", err);
                ifletErr(err) = waker.wake() {
                    error!("fail to exit daemon, error: {:?}", err);
                }
            }
            // Notify the daemon controller that one working thread has exited.
            Ok(())
        })
        .map_err(DaemonError::ThreadSpawn)?;
    self.fuse_service_threads.lock().unwrap().push(thread);
    Ok(())
}

kick_one_server方法启动了名为 fuse_server 的线程，成功启动的线程存储在 FusedevDaemon.fuse_service_threads 中。

2.5.2 FUSE server 线程（处理 FUSE 请求）

在启动线程前，创建了 fuse server 和 inflight operatoins。create_fuse_server() 是 FusedevFsService 结构实现的方法：

fn create_fuse_server(&self) -> Result<FuseServer> {
    FuseServer::new(self.server.clone(), self.session.lock().unwrap().deref())
}

create_fuse_server()方法通过 FuseServer::new()方法进行实例化，传入的参数中，self.server.clone() 是对 server 的引用，self.session.lock().unwrap().deref() 是 session 的去引用实例，方法的返回值是 FuseServer 结构的实例。

fn new(server: Arc<Server<Arc<Vfs>>>, se: &FuseSession) -> Result<FuseServer> {
    let ch = se.new_channel().map_err(|e| eother!(e))?;
    Ok(FuseServer { server, ch })
}

创建 FuseServer 结构的实例之前，首先通过 FuseSession 的 new_channel() 方法创建 fuse channel，并存储在 FuseServer 实例中。

FuseSession 是 fuse-backend-rs 中的结构，new_channel() 方法用于创建新的 channel:

FuseChannel::new()方法如下：

create_inflight_op() 方法也是 FusedevFsService 结构实现的方法，返回的 inflight_op 被添加到 FusedevFsService 结构的 inflight_ops中：

fn create_inflight_op(&self) -> FuseOpWrapper {
    let inflight_op = FuseOpWrapper::default();
    // "Not expected poisoned lock"
    self.inflight_ops.lock().unwrap().push(inflight_op.clone());
    inflight_op
}

FuseOpWrapper::default() 方法用于对 FuseOpWrapper 初始化，随后被追加到self.inflight_ops中。

创建好fuse server 和 inflight operatoins之后，启动fuse_server线程。其中，s.svc_loop(&inflight_op) 方法是线程的主要处理逻辑：

fn svc_loop(&mutself, metrics_hook: &dyn MetricsHook) -> Result<()> {
        // Given error EBADF, it means kernel has shut down this session.
        let _ebadf = Error::from_raw_os_error(libc::EBADF);
        loop {
            // 通过 channel（epoll）获取 FUSE 请求
            ifletSome((reader, writer)) = self.ch.get_request().map_err(|e| {
                warn!("get fuse request failed: {:?}", e);
                Error::from_raw_os_error(libc::EINVAL)
            })? {
                ifletErr(e) =
                    self.server
                        .handle_message(reader, writer.into(), None, Some(metrics_hook))
                {
                    match e {
                        fuse_backend_rs::Error::EncodeMessage(_ebadf) => {
                            returnErr(eio!("fuse session has been shut down"));
                        }
                        _ => {
                            error!("Handling fuse message, {}", DaemonError::ProcessQueue(e));
                            continue;
                        }
                    }
                }
            } else {
                info!("fuse server exits");
                break;
            }
        }
        Ok(())
    }

这是一个死循环，self.ch.get_request() 也是 fuse-backend-rs 中 FuseChannel 结构的方法，用于通过 channel 从 fuse 内核模块获取（通过 unix socket fd 进行通信） fuse 请求。

返回的值包括 reader 和 writer，作为方法handle_message() 的参数，同时还会传入metrics_hook用于收集数据。self.server.handle_message() 负责处理每个 fuse 请求，也是 fuse-backend-rs 中 Server 实现的方法：

fuse-backend-rs实现了针对不同Opcode的方法：

let res = match in_header.opcode {
    x if x == Opcode::Lookup asu32 => self.lookup(ctx),
    x if x == Opcode::Forget asu32 => self.forget(ctx), // No reply.
    x if x == Opcode::Getattr asu32 => self.getattr(ctx),
    x if x == Opcode::Setattr asu32 => self.setattr(ctx),
    x if x == Opcode::Readlink asu32 => self.readlink(ctx),
    x if x == Opcode::Symlink asu32 => self.symlink(ctx),
    x if x == Opcode::Mknod asu32 => self.mknod(ctx),
    x if x == Opcode::Mkdir asu32 => self.mkdir(ctx),
    x if x == Opcode::Unlink asu32 => self.unlink(ctx),
    x if x == Opcode::Rmdir asu32 => self.rmdir(ctx),
    x if x == Opcode::Rename asu32 => self.rename(ctx),
    x if x == Opcode::Link asu32 => self.link(ctx),
    x if x == Opcode::Open asu32 => self.open(ctx),
    x if x == Opcode::Read asu32 => self.read(ctx),
    x if x == Opcode::Write asu32 => self.write(ctx),
    x if x == Opcode::Statfs asu32 => self.statfs(ctx),
    x if x == Opcode::Release asu32 => self.release(ctx),
    x if x == Opcode::Fsync asu32 => self.fsync(ctx),
    x if x == Opcode::Setxattr asu32 => self.setxattr(ctx),
    x if x == Opcode::Getxattr asu32 => self.getxattr(ctx),
    x if x == Opcode::Listxattr asu32 => self.listxattr(ctx),
    x if x == Opcode::Removexattr asu32 => self.removexattr(ctx),
    x if x == Opcode::Flush asu32 => self.flush(ctx),
    x if x == Opcode::Init asu32 => self.init(ctx),
    x if x == Opcode::Opendir asu32 => self.opendir(ctx),
    x if x == Opcode::Readdir asu32 => self.readdir(ctx),
    x if x == Opcode::Releasedir asu32 => self.releasedir(ctx),
    x if x == Opcode::Fsyncdir asu32 => self.fsyncdir(ctx),
    x if x == Opcode::Getlk asu32 => self.getlk(ctx),
    x if x == Opcode::Setlk asu32 => self.setlk(ctx),
    x if x == Opcode::Setlkw asu32 => self.setlkw(ctx),
    x if x == Opcode::Access asu32 => self.access(ctx),
    x if x == Opcode::Create asu32 => self.create(ctx),
    x if x == Opcode::Bmap asu32 => self.bmap(ctx),
    x if x == Opcode::Ioctl asu32 => self.ioctl(ctx),
    x if x == Opcode::Poll asu32 => self.poll(ctx),
    x if x == Opcode::NotifyReply asu32 => self.notify_reply(ctx),
    x if x == Opcode::BatchForget asu32 => self.batch_forget(ctx),
    x if x == Opcode::Fallocate asu32 => self.fallocate(ctx),
    x if x == Opcode::Readdirplus asu32 => self.readdirplus(ctx),
    x if x == Opcode::Rename2 asu32 => self.rename2(ctx),
    x if x == Opcode::Lseek asu32 => self.lseek(ctx),
    #[cfg(feature = "virtiofs")]
    x if x == Opcode::SetupMapping asu32 => self.setupmapping(ctx, vu_req),
    #[cfg(feature = "virtiofs")]
    x if x == Opcode::RemoveMapping asu32 => self.removemapping(ctx, vu_req),
    // Group reqeusts don't need reply together
    x => match x {
        x if x == Opcode::Interrupt asu32 => {
            self.interrupt(ctx);
            Ok(0)
        }
        x if x == Opcode::Destroy asu32 => {
            self.destroy(ctx);
            Ok(0)
        }
        _ =>ctx.reply_error(io::Error::from_raw_os_error(libc::ENOSYS)),
    },
};

在每个方法中，调用了self.fs.xxx()方法完成操作，以mkdir为例：

这个fs指的是什么呢？在Server结构体定义中看到，fs是实现了FileSystem + Sync的 trait：

/// Fuse Server to handle requests from the Fuse client and vhost user master.
pubstruct Server<F: FileSystem + Sync> {
    fs: F,
    vers: ArcSwap<ServerVersion>,
}

还记得创建FuseServer的时候吗？

struct FuseServer {
    server: Arc<Server<Arc<Vfs>>>,
    ch: FuseChannel,
}
impl FuseServer {
    fn new(server: Arc<Server<Arc<Vfs>>>, se: &FuseSession) -> Result<FuseServer> {
        let ch = se.new_channel().map_err(|e| eother!(e))?;
        Ok(FuseServer { server, ch })
    }
    ...
}

这里FuseServer结构体中server类型Arc<Server<Arc<Vfs>>>中的Server就是Server结构体，因此，fs的类型是Arc<Vfs>。

在 fuse-backend-rs中对 Vfs 实现了 FileSystem trait:

fuse_server 线程可以通过top -Hp NYDUSD_PID 看到：

日志信息：

2.5.3 FUSE 终止 service

状态机收到TerminateService事件时，先执行d.interrupt()，然后等待线程结束，最后设置状态。

TerminateService => {
    d.interrupt();
    let res = d.wait_service();
    if res.is_ok() {
        d.set_state(DaemonState::READY);
    }
    res
}

interrupt() 方法：

fn interrupt(&self) {
    let session = self
        .service
        .session
        .lock()
        .expect("Not expect poisoned lock.");
    ifletErr(e) = session.wake().map_err(DaemonError::SessionShutdown) {
        error!("stop fuse service thread failed: {:?}", e);
    }
}

wait_service() 方法：

fn wait_service(&self) -> DaemonResult<()> {
    loop {
        let handle = self.fuse_service_threads.lock().unwrap().pop();
        ifletSome(handle) = handle {
            handle
                .join()
                .map_err(|e| {
                    DaemonError::WaitDaemon(
                        *e.downcast::<Error>()
                            .unwrap_or_else(|e| Box::new(eother!(e))),
                    )
                })?
                .map_err(DaemonError::WaitDaemon)?;
        } else {
            // No more handles to wait
            break;
        }
    }
    Ok(())
}

2.5.4 FUSE Umount 操作

Umount 事件和 TerminateService 事件的操作几乎一样，只是会在执行d.interrupt()之前先断开和 fuse 内核模块的连接：

Umount => d.disconnect().map(|r| {
    // Always interrupt fuse service loop after shutdown connection to kernel.
    // In case that kernel does not really shutdown the session due to some reasons
    // causing service loop keep waiting of `/dev/fuse`.
    d.interrupt();
    d.wait_service()
        .unwrap_or_else(|e| error!("failed to wait service {}", e));
    // at least all fuse thread stopped, no matter what error each thread got
    d.set_state(DaemonState::STOPPED);
    r
}),

断开连接的d.disconnect() 方法：

fn disconnect(&self) -> DaemonResult<()> {
    self.service.disconnect()
}

最终调用了session.umount() 方法：

fn disconnect(&self) -> DaemonResult<()> {
    let mutsession = self.session.lock().expect("Not expect poisoned lock.");
session.umount().map_err(DaemonError::SessionShutdown)?;
session.wake().map_err(DaemonError::SessionShutdown)?;
    Ok(())
}

fuse-backend-rs 中umount方法的实现：

/// Destroy a fuse session.
pub fnumount(&mutself) -> Result<()> {
    ifletSome(file) =self.file.take() {
        ifletSome(mountpoint) =self.mountpoint.to_str() {
            fuse_kern_umount(mountpoint, file)
        } else {
            Err(SessionFailure("invalid mountpoint".to_string()))
        }
    } else {
        Ok(())
    }
}

此外，还有 Restore 和 StopStateMachine 事件：

Restore => {
    let res = d.restore();
    if res.is_ok() {
        d.set_state(DaemonState::READY);
    }
    res
}
StopStateMachine => {
    d.set_state(DaemonState::STOPPED);
    Ok(())
}

Daemon 的状态为 STOPPED 时会结束此进程：

if d.get_state() == DaemonState::STOPPED {
    break;
}

状态机的功能到此结束。

回到create_fuse_daemon函数，到目前为止，已经创建了daemon对象并启动了状态机线程，状态机线程存储在daemon中：

2.6 Mount FUSE 文件系统

如果不是热升级和 failover 操作，会向 FUSE 内核模块发起 mount 操作请求：

// 1. api_sock 已经存在，但不是热升级操作，也不是 failover
// 2. api_sock 不存在
if (api_sock.as_ref().is_some() && !upgrade && !is_crashed(&mnt, api_sock.as_ref().unwrap())?)
    || api_sock.is_none()
{
    ifletSome(cmd) = mount_cmd {
        daemon.service.mount(cmd)?;
    }
    daemon.service.session.lock().unwrap()
        .mount()
        .map_err(|e| eother!(e))?;
    daemon.on_event(DaemonStateMachineInput::Mount)
        .map_err(|e| eother!(e))?;
    daemon.on_event(DaemonStateMachineInput::Start)
        .map_err(|e| eother!(e))?;
    daemon.service.conn
        .store(calc_fuse_conn(mnt)?, Ordering::Relaxed);
}

如果mount_cmd不为 None，则通过daemon.service.mount(cmd)挂载后端文件系统：

// NOTE: This method is not thread-safe, however, it is acceptable as
// mount/umount/remount/restore_mount is invoked from single thread in FSM
fn mount(&self, cmd: FsBackendMountCmd) -> DaemonResult<()> {
    ifself.backend_from_mountpoint(&cmd.mountpoint)?.is_some() {
        returnErr(DaemonError::AlreadyExists);
    }
    let backend = fs_backend_factory(&cmd)?;
    let index = self.get_vfs().mount(backend, &cmd.mountpoint)?;
    info!("{} filesystem mounted at {}", &cmd.fs_type, &cmd.mountpoint);
    self.backend_collection().add(&cmd.mountpoint, &cmd)?;
    // Add mounts opaque to UpgradeManager
    ifletSome(mutmgr_guard) = self.upgrade_mgr() {
        upgrade::add_mounts_state(&mutmgr_guard, cmd, index)?;
    }
    Ok(())
}

首先通过self.backend_from_mountpoint(&cmd.mountpoint)方法检查传入的路径是否已经被挂载。如果已经存在，则返回错误。

backend_from_mountpoint方法调用了Vfs的get_rootfs方法，首先得到传入path的inode，然后查看对应inode是否存在mountpoints Hashmap 中：

/// Get the mounted backend file system alongside the path if there's one.
pubfn get_rootfs(&self, path: &str) -> VfsResult<Option<Arc<BackFileSystem>>> {
    // Serialize mount operations. Do not expect poisoned lock here.
    let _guard = self.lock.lock().unwrap();
    let inode = matchself.root.path_walk(path).map_err(VfsError::PathWalk)? {
        Some(i) => i,
        None => returnOk(None),
    };
    ifletSome(mnt) = self.mountpoints.load().get(&inode) {
        Ok(Some(self.get_fs_by_idx(mnt.fs_idx).map_err(|e| {
            VfsError::NotFound(format!("fs index {}, {:?}", mnt.fs_idx, e))
        })?))
    } else {
        // Pseudo fs dir inode exists, but that no backend is ever mounted
        // is a normal case.
        Ok(None)
    }
}

然后，通过fs_backend_factory(&cmd)方法获取文件系统后端，该方法的返回值是实现了BackendFileSystem+Sync+Sendtrait 的结构体。

在fs_backend_factory方法中，首先验证预取文件列表：

然后根据传入的fs_type分别进行实例化，目前支持两种类型：

pubenum FsBackendType {
    Rafs,
    PassthroughFs,
}

2.6.1 初始化 RAFS backend

首先，解析从cmd传入的config内容，并根据传入的bootstrap文件路径，打开用于（从 bootstrap 中）读取文件系统的元数据信息的reader，绑定到bootstrap变量。接下来创建 rafs 实例，传入参数包括配置信息、挂载路径、bootstrap文件对应的reader：

FsBackendType::Rafs => {
    let rafs_config = RafsConfig::from_str(cmd.config.as_str())?;
    let mutbootstrap = <dyn RafsIoRead>::from_file(&cmd.source)?;
    let mutrafs = Rafs::new(rafs_config, &cmd.mountpoint, &mutbootstrap)?;
rafs.import(bootstrap, prefetch_files)?;
    info!("RAFS filesystem imported");
    Ok(Box::new(rafs))
}

通过Rafs::new(rafs_config, &cmd.mountpoint, &mut bootstrap)方法创建 rafs 实例。

首先，准备配置信息storage_conf，并通过传入的conf参数创建RafsSuper实例。创建RafsSuper只是初始化配置信息，包括 RafsMode（有 Direct 和 Cached 两种可选）。接下来，通过sb.load(r)方法从bootstarp加载 RAFS 超级块的信息。RAFS V5 和 V6 两个版本的加载方式不同，try_load_v6方法：

pub(crate) fntry_load_v6(&mutself,r: &mut RafsIoReader) -> Result<bool> {
    let end =r.seek_to_end(0)?;
r.seek_to_offset(0)?;
    // 创建 RAFSV6SuperBlock 实例
    let mutsb = RafsV6SuperBlock::new();
    // 读取 RAFS V6 的超级块信息
    // offset 1024，length 128
    ifsb.load(r).is_err() {
        returnOk(false);
    }
    if !sb.is_rafs_v6() {
        returnOk(false);
    }
sb.validate(end)?;
    // 设置 RAFS 超级块的 meta 信息
self.meta.version = RAFS_SUPER_VERSION_V6;
self.meta.magic =sb.magic();
self.meta.meta_blkaddr =sb.s_meta_blkaddr;
self.meta.root_nid =sb.s_root_nid;
    // 创建 RafsV6SuperBlockExt 实例
    let mutext_sb = RafsV6SuperBlockExt::new();
    // 读取 RAFS V6 的扩展超级块信息
    // offset 1024 + 128，length 256
ext_sb.load(r)?;
ext_sb.validate(end)?;
    // 设置 RAFS 超级块的 meta 信息
self.meta.chunk_size =ext_sb.chunk_size();
self.meta.blob_table_offset =ext_sb.blob_table_offset();
self.meta.blob_table_size =ext_sb.blob_table_size();
self.meta.chunk_table_offset =ext_sb.chunk_table_offset();
self.meta.chunk_table_size =ext_sb.chunk_table_size();
self.meta.inodes_count =sb.inodes_count();
self.meta.flags = RafsSuperFlags::from_bits(ext_sb.flags())
        .ok_or_else(|| einval!(format!("invalid super flags {:x}",ext_sb.flags())))?;
    info!("rafs superblock features: {}",self.meta.flags);
    // 设置 RAFS 超级块 meta 中的预取列表信息
self.meta.prefetch_table_entries =ext_sb.prefetch_table_size() / size_of::<u32>() asu32;
self.meta.prefetch_table_offset =ext_sb.prefetch_table_offset();
    trace!(
        "prefetch table offset {} entries {} ",
self.meta.prefetch_table_offset,
self.meta.prefetch_table_entries
    );
    matchself.mode {
        // 如果 RAFS 模式是 Direct，还需要创建
        // DirectSuperBlockV6 实例并读取相关信息
        RafsMode::Direct => {
            let mutsb_v6 = DirectSuperBlockV6::new(&self.meta);
sb_v6.load(r)?;
self.superblock = Arc::new(sb_v6);
            Ok(true)
        }
        RafsMode::Cached => Err(enosys!("Rafs v6 does not support cached mode")),
    }
}

RAFS 超级块信息加载后，获取blob信息，然后创建rafs实例：

pubfn new(conf: RafsConfig, id: &str,r: &mut RafsIoReader) -> RafsResult<Self> {
    let storage_conf = Self::prepare_storage_conf(&conf)?;
    let mutsb = RafsSuper::new(&conf).map_err(RafsError::FillSuperblock)?;
sb.load(r).map_err(RafsError::FillSuperblock)?;
    // 获取 super block 之后，从中获取 blob 信息（BlobInfo）
    let blob_infos =sb.superblock.get_blob_infos();
    // 根据配置信息和 blobs 信息，遍历每条 blob_info，
    // 创建 BlobDevice 的实例
    let device =
        BlobDevice::new(&storage_conf, &blob_infos).map_err(RafsError::CreateDevice)?;
    // 创建 rafs 实例
    let rafs = Rafs {
        id: id.to_string(),
        device,    // BlobDevice
        ios: metrics::FsIoStats::new(id),
        sb: Arc::new(sb),
        initialized: false,    // 还未初始化
        digest_validate: conf.digest_validate,
        fs_prefetch: conf.fs_prefetch.enable,    // 支持预取
        amplify_io: conf.amplify_io,
        prefetch_all: conf.fs_prefetch.prefetch_all,
        xattr_enabled: conf.enable_xattr,    // 开启 xattr
        i_uid: geteuid().into(),    // uid
        i_gid: getegid().into(),    // gid
        i_time: SystemTime::now()
            .duration_since(SystemTime::UNIX_EPOCH)
            .unwrap()
            .as_secs(),
    };
    // Rafs v6 does must store chunk info into local file cache. So blob cache is required
    if rafs.metadata().is_v6() {
        if conf.device.cache.cache_type != "blobcache" {
            returnErr(RafsError::Configure(
                "Rafs v6 must have local blobcache configured".to_string(),
            ));
        }
        if conf.digest_validate {
            returnErr(RafsError::Configure(
                "Rafs v6 doesn't support integrity validation yet".to_string(),
            ));
        }
    }
    rafs.ios.toggle_files_recording(conf.iostats_files);
    rafs.ios.toggle_access_pattern(conf.access_pattern);
    rafs.ios
        .toggle_latest_read_files_recording(conf.latest_read_files);
    Ok(rafs)
}

关于 rafs 文件系统（以 v6 为例）元数据在 bootstrap 文件中的分布，在 rafs/src/metadata/layout/v6.rs 中有详细定义：

/// EROFS metadata slot size.
pubconst EROFS_INODE_SLOT_SIZE: usize = 1 << EROFS_INODE_SLOT_BITS;
/// EROFS logical block size.
pubconst EROFS_BLOCK_SIZE: u64 = 1u64 << EROFS_BLOCK_BITS;
/// EROFS plain inode.
pubconst EROFS_INODE_FLAT_PLAIN: u16 = 0;
/// EROFS inline inode.
pubconst EROFS_INODE_FLAT_INLINE: u16 = 2;
/// EROFS chunked inode.
pubconst EROFS_INODE_CHUNK_BASED: u16 = 4;
/// EROFS device table offset.
pub constEROFS_DEVTABLE_OFFSET: u16 =
    EROFS_SUPER_OFFSET + EROFS_SUPER_BLOCK_SIZE + EROFS_EXT_SUPER_BLOCK_SIZE;
pubconst EROFS_I_VERSION_BIT: u16 = 0;
pubconst EROFS_I_VERSION_BITS: u16 = 1;
pubconst EROFS_I_DATALAYOUT_BITS: u16 = 3;
// Offset of EROFS super block.
pub constEROFS_SUPER_OFFSET: u16 = 1024;
// Size of EROFS super block.
pubconst EROFS_SUPER_BLOCK_SIZE: u16 = 128;
// Size of extended super block, used for rafs v6 specific fields
const EROFS_EXT_SUPER_BLOCK_SIZE: u16 = 256;
// Magic number for EROFS super block.
const EROFS_SUPER_MAGIC_V1: u32 = 0xE0F5_E1E2;
// Bits of EROFS logical block size.
const EROFS_BLOCK_BITS: u8 = 12;
// Bits of EROFS metadata slot size.
const EROFS_INODE_SLOT_BITS: u8 = 5;

创建rafs实例后，通过rafs.import(bootstrap, prefetch_files)方法初始化（导入bootstrap和prefetch信息）：

/// Import an rafs bootstrap to initialize the filesystem instance.
pub fnimport(
    &mutself,
    r: RafsIoReader,
    prefetch_files: Option<Vec<PathBuf>>,
) -> RafsResult<()> {
    ifself.initialized {
        returnErr(RafsError::AlreadyMounted);
    }
    ifself.fs_prefetch {
        // Device should be ready before any prefetch.
self.device.start_prefetch();
self.prefetch(r, prefetch_files);
    }
self.initialized = true;
    Ok(())
}

主要是开启prefetch线程，self.prefetch(r, prefetch_files)方法传入两个参数，r是 bootstrap 文件的 reader，prefetch_files是已经从 bootstrap 读取的预取文件列表：

fn prefetch(&self, reader: RafsIoReader, prefetch_files: Option<Vec<PathBuf>>) {
    let sb = self.sb.clone();
    let device = self.device.clone();
    let prefetch_all = self.prefetch_all;
    let root_ino = self.root_ino();
    let _ = std::thread::spawn(move || {
        Self::do_prefetch(root_ino, reader, prefetch_files, prefetch_all, sb, device);
    });
}

在do_prefetch方法中，首先设置每个blob对应device的状态为允许prefetch，然后，根据prefetch_files进行预取：

pub fnimport(
    &mutself,
    r: RafsIoReader,
    prefetch_files: Option<Vec<PathBuf>>,
) -> RafsResult<()> {
    ifself.initialized {
        returnErr(RafsError::AlreadyMounted);
    }
    ifself.fs_prefetch {
        // Device should be ready before any prefetch.
self.device.start_prefetch();
self.prefetch(r, prefetch_files);
    }
self.initialized = true;
    Ok(())
}

在self.prefetch(r, prefetch_files)方法中，开启了预取线程：

fn prefetch(&self, reader: RafsIoReader, prefetch_files: Option<Vec<PathBuf>>) {
    let sb = self.sb.clone();
    let device = self.device.clone();
    let prefetch_all = self.prefetch_all;
    let root_ino = self.root_ino();
    let _ = std::thread::spawn(move || {
        Self::do_prefetch(root_ino, reader, prefetch_files, prefetch_all, sb, device);
    });
}

线程中运行do_prefetch方法，按 chunk 粒度进行预取：

fn do_prefetch(
    root_ino: u64,
    mutreader: RafsIoReader,    // bootstrap 对应的 reader
    prefetch_files: Option<Vec<PathBuf>>,
    prefetch_all: bool,
    sb: Arc<RafsSuper>,
    device: BlobDevice,
) {
    // First do range based prefetch for rafs v6.
    if sb.meta.is_v6() {
        // 生成 BlobPrefetchRequest，按 chunk 为粒度的请求
        let mutprefetches = Vec::new();
        for blob in sb.superblock.get_blob_infos() {
            let sz = blob.prefetch_size();
            if sz > 0 {
                let mutoffset = 0;
                whileoffset < sz {
                    // 按 chunk 为粒度生成请求
                    let len = cmp::min(sz -offset, RAFS_DEFAULT_CHUNK_SIZE);
prefetches.push(BlobPrefetchRequest {
                        blob_id: blob.blob_id().to_owned(),
                        offset,
                        len,
                    });
offset+= len;
                }
            }
        }
        if !prefetches.is_empty() {
            // 通过 device 的 prefetch 进行预取
            device.prefetch(&[], &prefetches).unwrap_or_else(|e| {
                warn!("Prefetch error, {:?}", e);
            });
        }
    }
    let fetcher = |desc: &mut BlobIoVec, last: bool| {
        ifdesc.size() asu64 > RAFS_MAX_CHUNK_SIZE
            ||desc.len() > 1024
            || (last &&desc.size() > 0)
        {
            trace!(
                "fs prefetch: 0x{:x} bytes for {} descriptors",
desc.size(),
desc.len()
            );
            device.prefetch(&[desc], &[]).unwrap_or_else(|e| {
                warn!("Prefetch error, {:?}", e);
            });
desc.reset();
        }
    };
    let mutignore_prefetch_all = prefetch_files
        .as_ref()
        .map(|f| f.len() == 1 && f[0].as_os_str() == "/")
        .unwrap_or(false);
    // Then do file based prefetch based on:
    // - prefetch listed passed in by user
    // - or file prefetch list in metadata
    let inodes = prefetch_files.map(|files| Self::convert_file_list(&files, &sb));
    let res = sb.prefetch_files(&device, &mutreader, root_ino, inodes, &fetcher);
    match res {
        Ok(true) =>ignore_prefetch_all = true,
        Ok(false) => {}
        Err(e) => info!("No file to be prefetched {:?}", e),
    }
    // Last optionally prefetch all data
    if prefetch_all && !ignore_prefetch_all {
        let root = vec![root_ino];
        let res = sb.prefetch_files(&device, &mutreader, root_ino, Some(root), &fetcher);
        ifletErr(e) = res {
            info!("No file to be prefetched {:?}", e);
        }
    }
}

生成预取请求列表后，通过device的prefetch方法进行预取：

/// Try to prefetch specified blob data.
pubfn prefetch(
    &self,
    io_vecs: &[&BlobIoVec],
    prefetches: &[BlobPrefetchRequest],
) -> io::Result<()> {
    for idx in0..prefetches.len() {
        // 根据 blob_id 获取 blob 信息
        ifletSome(blob) = self.get_blob_by_id(&prefetches[idx].blob_id) {
            // 通过 blob 的 prefetch 方法进行预取
            let _ = blob.prefetch(blob.clone(), &prefetches[idx..idx + 1], &[]);
        }
    }
    for io_vec in io_vecs.iter() {
        ifletSome(blob) = self.get_blob_by_iovec(io_vec) {
            // Prefetch errors are ignored.
            let _ = blob
                .prefetch(blob.clone(), &[], &io_vec.bi_vec)
                .map_err(|e| {
                    error!("failed to prefetch blob data, {}", e);
                });
        }
    }
    Ok(())
}

根据 blob_id获取 blob 后，调用prefetch方法：

fn prefetch(
    &self,
    blob_cache: Arc<dyn BlobCache>,
    prefetches: &[BlobPrefetchRequest],
    bios: &[BlobIoDesc],
) -> StorageResult<usize> {
    // Handle blob prefetch request first, it may help performance.
    for req in prefetches {
        // 生成异步预取请求消息
        let msg = AsyncPrefetchMessage::new_blob_prefetch(
            blob_cache.clone(),
            req.offset asu64,
            req.len asu64,
        );
        // 将请求消息通过 channel 传递给 worker
        let _ = self.workers.send_prefetch_message(msg);
    }
    // Then handle fs prefetch
    let max_comp_size = self.prefetch_batch_size();
    let mutbios = bios.to_vec();
bios.sort_by_key(|entry| entry.chunkinfo.compressed_offset());
    self.metrics.prefetch_unmerged_chunks.add(bios.len() asu64);
    BlobIoMergeState::merge_and_issue(
        &bios,
        max_comp_size,
        max_comp_size asu64 >> RAFS_MERGING_SIZE_TO_GAP_SHIFT,
        |req: BlobIoRange| {
            // 生成异步预取请求消息
            let msg = AsyncPrefetchMessage::new_fs_prefetch(blob_cache.clone(), req);
            let _ = self.workers.send_prefetch_message(msg);
        },
    );
    Ok(0)
}

接收预取消息并进行处理的函数：

asyncfn handle_prefetch_requests(mgr: Arc<AsyncWorkerMgr>, rt: &Runtime) {
    // Max 1 active requests per thread.
    mgr.prefetch_sema.add_permits(1);
    whileletOk(msg) = mgr.prefetch_channel.recv().await {
        mgr.handle_prefetch_rate_limit(&msg).await;
        let mgr2 = mgr.clone();
        match msg {
            AsyncPrefetchMessage::BlobPrefetch(blob_cache, offset, size) => {
                let token = Semaphore::acquire_owned(mgr2.prefetch_sema.clone())
                    .await
                    .unwrap();
                if blob_cache.is_prefetch_active() {
                    rt.spawn_blocking(move || {
                        let _ = Self::handle_blob_prefetch_request(
                            mgr2.clone(),
                            blob_cache,
                            offset,
                            size,
                        );
                        drop(token);
                    });
                }
            }
            AsyncPrefetchMessage::FsPrefetch(blob_cache, req) => {
                let token = Semaphore::acquire_owned(mgr2.prefetch_sema.clone())
                    .await
                    .unwrap();
                if blob_cache.is_prefetch_active() {
                    rt.spawn_blocking(move || {
                        let _ = Self::handle_fs_prefetch_request(mgr2.clone(), blob_cache, req);
                        drop(token)
                    });
                }
            }
            AsyncPrefetchMessage::Ping => {
                let _ = mgr.ping_requests.fetch_add(1, Ordering::Relaxed);
            }
            AsyncPrefetchMessage::RateLimiter(_size) => {}
        }
        mgr.prefetch_inflight.fetch_sub(1, Ordering::Relaxed);
    }
}

目前，有两种预取的方法：Blob 模式和 Fs 模式。

(1) Blob 模式预取

对应的处理函数为handle_blob_prefetch_request：

fn handle_blob_prefetch_request(
    mgr: Arc<AsyncWorkerMgr>,
    cache: Arc<dyn BlobCache>,
    offset: u64,
    size: u64,
) -> Result<()> {
    trace!(
        "storage: prefetch blob {} offset {} size {}",
        cache.blob_id(),
        offset,
        size
    );
    if size == 0 {
        returnOk(());
    }
    // 获取 blob object
    ifletSome(obj) = cache.get_blob_object() {
        // 获取 (offset, offset + size) 范围内的内容
        ifletErr(e) = obj.fetch_range_compressed(offset, size) {
            warn!(
                "storage: failed to prefetch data from blob {}, offset {}, size {}, {}, will try resend",
                cache.blob_id(),
                offset,
                size,
                e
            );
            ASYNC_RUNTIME.spawn(asyncmove {
                let mutinterval = interval(Duration::from_secs(1));
interval.tick().await;
                // 如果失败，重新发起预取消息
                let msg = AsyncPrefetchMessage::new_blob_prefetch(cache.clone(), offset, size);
                let _ = mgr.send_prefetch_message(msg);
            });
        }
    } else {
        warn!("prefetch blob range is not supported");
    }
    Ok(())
}

其中，主要的处理函数为obj.fetch_range_compressed(offset, size)：

fn fetch_range_compressed(&self, offset: u64, size: u64) -> Result<()> {
    let meta = self.meta.as_ref().ok_or_else(|| einval!())?;
    let meta = meta.get_blob_meta().ok_or_else(|| einval!())?;
    let mutchunks = meta.get_chunks_compressed(offset, size, self.prefetch_batch_size())?;
    ifletSome(meta) = self.get_blob_meta_info()? {
chunks = self.strip_ready_chunks(meta, None,chunks);
    }
    ifchunks.is_empty() {
        Ok(())
    } else {
        self.do_fetch_chunks(&chunks, true)
    }
}

meta.get_chunks_compressed方法用于获取包含（offset, offset + size）范围的chunk列表：

pubfn get_chunks_compressed(
    &self,
    start: u64,
    size: u64,
    batch_size: u64,
) -> Result<Vec<Arc<dyn BlobChunkInfo>>> {
    let end = start.checked_add(size).ok_or_else(|| {
        einval!(einval!(format!(
            "get_chunks_compressed: invalid start {}/size {}",
            start, size
        )))
    })?;
    if end > self.state.compressed_size {
        returnErr(einval!(format!(
            "get_chunks_compressed: invalid end {}/compressed_size {}",
            end, self.state.compressed_size
        )));
    }
    let batch_end = if batch_size <= size {
        end
    } else {
        std::cmp::min(
            start.checked_add(batch_size).unwrap_or(end),
            self.state.compressed_size,
        )
    };
    self.state
        .get_chunks_compressed(start, end, batch_end, batch_size)
}

BlobMetaChunkArray::V2版本的self.state.get_chunks_compressed方法实际的处理函数内容如下：

fn _get_chunks_compressed<T: BlobMetaChunkInfo>(
    state: &Arc<BlobMetaState>,
    chunk_info_array: &[T],
    start: u64,
    end: u64,
    batch_end: u64,
    batch_size: u64,
) -> Result<Vec<Arc<dyn BlobChunkInfo>>> {
    let mutvec = Vec::with_capacity(512);
    let mutindex = Self::_get_chunk_index_nocheck(chunk_info_array, start, true)?;
    let entry = Self::get_chunk_entry(state, chunk_info_array,index)?;
    // Special handling of ZRan chunks
    if entry.is_zran() {
        let zran_index = entry.get_zran_index();
        let pos = state.zran_info_array[zran_index asusize].in_offset();
        let mutzran_last = zran_index;
        whileindex > 0 {
            let entry = Self::get_chunk_entry(state, chunk_info_array,index - 1)?;
            if !entry.is_zran() {
                returnErr(einval!(
                    "inconsistent ZRan and non-ZRan chunk information entries"
                ));
            } elseif entry.get_zran_index() != zran_index {
                // reach the header chunk associated with the same ZRan context.
                break;
            } else {
index-= 1;
            }
        }
        let mutvec = Vec::with_capacity(128);
        for entry in &chunk_info_array[index..] {
            entry.validate(state)?;
            if !entry.is_zran() {
                returnErr(einval!(
                    "inconsistent ZRan and non-ZRan chunk information entries"
                ));
            }
            if entry.get_zran_index() !=zran_last {
                let ctx = &state.zran_info_array[entry.get_zran_index() asusize];
                if ctx.in_offset() + ctx.in_size() asu64 - pos > batch_size
                    && entry.compressed_offset() > end
                {
                    returnOk(vec);
                }
zran_last = entry.get_zran_index();
            }
vec.push(BlobMetaChunk::new(index, state));
        }
        returnOk(vec);
    }
vec.push(BlobMetaChunk::new(index, state));
    let mutlast_end = entry.compressed_end();
    iflast_end >= batch_end {
        Ok(vec)
    } else {
        whileindex + 1 < chunk_info_array.len() {
index+= 1;
            let entry = Self::get_chunk_entry(state, chunk_info_array,index)?;
            // Avoid read amplify if next chunk is too big.
            iflast_end >= end && entry.compressed_end() > batch_end {
                returnOk(vec);
            }
vec.push(BlobMetaChunk::new(index, state));
last_end = entry.compressed_end();
            iflast_end >= batch_end {
                returnOk(vec);
            }
        }
        Err(einval!(format!(
            "entry not found index {} chunk_info_array.len {}",
index,
            chunk_info_array.len(),
        )))
    }
}

获取包含的chunks之后，通过self.strip_ready_chunks方法分离这些chunks（具体含义未深究）：

fn strip_ready_chunks(
    &self,
    meta: Arc<BlobMetaInfo>,
    old_chunks: Option<&[Arc<dyn BlobChunkInfo>]>,
    mutextended_chunks: Vec<Arc<dyn BlobChunkInfo>>,
) -> Vec<Arc<dyn BlobChunkInfo>> {
    ifself.is_zran {
        let mutset = HashSet::new();
        for c inextended_chunks.iter() {
            if !matches!(self.chunk_map.is_ready(c.as_ref()), Ok(true)) {
set.insert(meta.get_zran_index(c.id()));
            }
        }
        let first = old_chunks.as_ref().map(|v| v[0].id()).unwrap_or(u32::MAX);
        let mutstart = 0;
        whilestart <extended_chunks.len() {
            let id =extended_chunks[start].id();
            if id == first ||set.contains(&meta.get_zran_index(id)) {
                break;
            }
start+= 1;
        }
        let last = old_chunks
            .as_ref()
            .map(|v| v[v.len() - 1].id())
            .unwrap_or(u32::MAX);
        let mutend =extended_chunks.len() - 1;
        whileend >start {
            let id =extended_chunks[end].id();
            if id == last ||set.contains(&meta.get_zran_index(id)) {
                break;
            }
end-= 1;
        }
        assert!(end >=start);
        ifstart == 0 &&end ==extended_chunks.len() - 1 {
extended_chunks
        } else {
extended_chunks[start..=end].to_vec()
        }
    } else {
        while !extended_chunks.is_empty() {
            let chunk = &extended_chunks[extended_chunks.len() - 1];
            if matches!(self.chunk_map.is_ready(chunk.as_ref()), Ok(true)) {
extended_chunks.pop();
            } else {
                break;
            }
        }
extended_chunks
    }
}

然后，通过self.do_fetch_chunks(&chunks, true)方法获取chunks的数据：

fn do_fetch_chunks(&self, chunks: &[Arc<dyn BlobChunkInfo>], prefetch: bool) -> Result<()> {
    // Validate input parameters.
    assert!(!chunks.is_empty());
    if chunks.len() > 1 {
        for idx in0..chunks.len() - 1 {
            assert_eq!(chunks[idx].id() + 1, chunks[idx + 1].id());
        }
    }
    // Get chunks not ready yet, also marking them as in-flight.
    let bitmap = self
        .chunk_map
        .as_range_map()
        .ok_or_else(|| einval!("invalid chunk_map for do_fetch_chunks()"))?;
    let chunk_index = chunks[0].id();
    let count = chunks.len() asu32;
    let pending = match bitmap.check_range_ready_and_mark_pending(chunk_index, count)? {
        None => returnOk(()),
        Some(v) => v,
    };
    let mutstatus = vec![false; count asusize];
    let (start_idx, end_idx) = ifself.is_zran {
        for chunk_id in pending.iter() {
status[(*chunk_id - chunk_index) asusize] = true;
        }
        (0, pending.len())
    } else {
        let mutstart = u32::MAX;
        let mutend = 0;
        for chunk_id in pending.iter() {
status[(*chunk_id - chunk_index) asusize] = true;
start = std::cmp::min(*chunk_id - chunk_index,start);
end = std::cmp::max(*chunk_id - chunk_index,end);
        }
        ifend <start {
            returnOk(());
        }
        (start asusize,end asusize)
    };
    let start_chunk = &chunks[start_idx];
    let end_chunk = &chunks[end_idx];
    let (blob_offset, blob_end, blob_size) =
        self.get_blob_range(&chunks[start_idx..=end_idx])?;
    trace!(
        "fetch data range {:x}-{:x} for chunk {}-{} from blob {:x}",
        blob_offset,
        blob_end,
        start_chunk.id(),
        end_chunk.id(),
        chunks[0].blob_index()
    );
    // 从 backend 读取数据
    matchself.read_chunks_from_backend(
        blob_offset,
        blob_size,
        &chunks[start_idx..=end_idx],
        prefetch,
    ) {
        Ok(mutbufs) => {
            ifself.is_compressed {
                let res =
                    Self::persist_cached_data(&self.file, blob_offset,bufs.compressed_buf());
                for idx in start_idx..=end_idx {
                    ifstatus[idx] {
                        self.update_chunk_pending_status(chunks[idx].as_ref(), res.is_ok());
                    }
                }
            } else {
                for idx in start_idx..=end_idx {
                    let mutbuf = matchbufs.next() {
                        None => returnErr(einval!("invalid chunk decompressed status")),
                        Some(Err(e)) => {
                            for idx in idx..=end_idx {
                                ifstatus[idx] {
                                    bitmap.clear_range_pending(chunks[idx].id(), 1)
                                }
                            }
                            returnErr(e);
                        }
                        Some(Ok(v)) => v,
                    };
                    ifstatus[idx] {
                        ifself.dio_enabled {
                            self.adjust_buffer_for_dio(&mutbuf)
                        }
                        self.persist_chunk_data(chunks[idx].as_ref(),buf.as_ref());
                    }
                }
            }
        }
        Err(e) => {
            for idx in0..chunks.len() {
                ifstatus[idx] {
                    bitmap.clear_range_pending(chunks[idx].id(), 1)
                }
            }
            returnErr(e);
        }
    }
    if !bitmap.wait_for_range_ready(chunk_index, count)? {
        if prefetch {
            returnErr(eio!("failed to read data from storage backend"));
        }
        // if we are in on-demand path, retry for the timeout chunks
        for chunk in chunks {
            matchself.chunk_map.check_ready_and_mark_pending(chunk.as_ref()) {
                Err(e) => returnErr(eio!(format!("do_fetch_chunks failed, {:?}", e))),
                Ok(true) => {}
                Ok(false) => {
                    info!("retry for timeout chunk, {}", chunk.id());
                    let mutbuf = alloc_buf(chunk.uncompressed_size() asusize);
                    self.read_chunk_from_backend(chunk.as_ref(), &mutbuf)
                        .map_err(|e| {
                            self.update_chunk_pending_status(chunk.as_ref(), false);
                            eio!(format!("read_raw_chunk failed, {:?}", e))
                        })?;
                    ifself.dio_enabled {
                        self.adjust_buffer_for_dio(&mutbuf)
                    }
                    self.persist_chunk_data(chunk.as_ref(), &buf);
                }
            }
        }
    }
    Ok(())
}

其中self.read_chunks_from_backend方法实现从 backend 读取数据：

fn read_chunks_from_backend<'a, 'b>(
    &'aself,
    blob_offset: u64,
    blob_size: usize,
    chunks: &'b [Arc<dyn BlobChunkInfo>],
    prefetch: bool,
) -> Result<ChunkDecompressState<'a, 'b>>
where
    Self: Sized,
{
    // Read requested data from the backend by altogether.
    let mutc_buf = alloc_buf(blob_size);
    let start = Instant::now();
    let nr_read = self
        .reader()
        .read(c_buf.as_mut_slice(), blob_offset)
        .map_err(|e| eio!(e))?;
    if nr_read != blob_size {
        returnErr(eio!(format!(
            "request for {} bytes but got {} bytes",
            blob_size, nr_read
        )));
    }
    let duration = Instant::now().duration_since(start).as_millis();
    debug!(
        "read_chunks_from_backend: {} {} {} bytes at {}, duration {}ms",
        std::thread::current().name().unwrap_or_default(),
        if prefetch { "prefetch" } else { "fetch" },
        blob_size,
        blob_offset,
        duration
    );
    let chunks = chunks.iter().map(|v| v.as_ref()).collect();
    Ok(ChunkDecompressState::new(blob_offset, self, chunks,c_buf))
}

self.reader().read方法是对 backend 的抽象，每个请求失败后会重试retry_count次：

fn read(&self,buf: &mut [u8], offset: u64) -> BackendResult<usize> {
    let mutretry_count = self.retry_limit();
    let begin_time = self.metrics().begin();
    loop {
        matchself.try_read(buf, offset) {
            Ok(size) => {
                self.metrics().end(&begin_time,buf.len(), false);
                returnOk(size);
            }
            Err(err) => {
                ifretry_count > 0 {
                    warn!(
                        "Read from backend failed: {:?}, retry count {}",
                        err,retry_count
                    );
retry_count-= 1;
                } else {
                    self.metrics().end(&begin_time,buf.len(), true);
                    ERROR_HOLDER
                        .lock()
                        .unwrap()
                        .push(&format!("{:?}", err))
                        .unwrap_or_else(|_| error!("Failed when try to hold error"));
                    returnErr(err);
                }
            }
        }
    }
}

不同 backend 的try_read方法实现不同，目前，nydus分别实现了localfs、registry、OSS三种 backend。

(2) Fs 模式预取

对应的处理函数为handle_fs_prefetch_request：

fn handle_fs_prefetch_request(
    mgr: Arc<AsyncWorkerMgr>,
    cache: Arc<dyn BlobCache>,
    req: BlobIoRange,
) -> Result<()> {
    let blob_offset = req.blob_offset;
    let blob_size = req.blob_size;
    trace!(
        "storage: prefetch fs data from blob {} offset {} size {}",
        cache.blob_id(),
        blob_offset,
        blob_size
    );
    if blob_size == 0 {
        returnOk(());
    }
    // Record how much prefetch data is requested from storage backend.
    // So the average backend merged request size will be prefetch_data_amount/prefetch_mr_count.
    // We can measure merging possibility by this.
    mgr.metrics.prefetch_mr_count.inc();
    mgr.metrics.prefetch_data_amount.add(blob_size);
    ifletSome(obj) = cache.get_blob_object() {
        obj.prefetch_chunks(&req)?;
    } else {
        cache.prefetch_range(&req)?;
    }
    Ok(())
}

Fs 模式的预取有两种情况，（1）如果有缓存的blob时：

fn prefetch_chunks(&self, range: &BlobIoRange) -> Result<()> {
    let chunks_extended;
    let mutchunks = &range.chunks;
    ifletSome(v) = self.extend_pending_chunks(chunks, self.prefetch_batch_size())? {
        chunks_extended = v;
chunks = &chunks_extended;
    }
    let mutstart = 0;
    whilestart <chunks.len() {
        // Figure out the range with continuous chunk ids, be careful that `end` is inclusive.
        let mutend =start;
        whileend <chunks.len() - 1 &&chunks[end + 1].id() ==chunks[end].id() + 1 {
end+= 1;
        }
        self.do_fetch_chunks(&chunks[start..=end], true)?;
start =end + 1;
    }
    Ok(())
}

准备好chunks后，也是调用了do_fetch_chunks方法，和 Blob 模式相同。

（2）如果没有缓存blob，则使用cache.prefetch_range(&req)方法：

fn prefetch_range(&self, range: &BlobIoRange) -> Result<usize> {
    let mutpending = Vec::with_capacity(range.chunks.len());
    if !self.chunk_map.is_persist() {
        let mutd_size = 0;
        for c in range.chunks.iter() {
d_size = std::cmp::max(d_size, c.uncompressed_size() asusize);
        }
        let mutbuf = alloc_buf(d_size);
        for c in range.chunks.iter() {
            ifletOk(true) = self.chunk_map.check_ready_and_mark_pending(c.as_ref()) {
                // The chunk is ready, so skip it.
                continue;
            }
            // For digested chunk map, we must check whether the cached data is valid because
            // the digested chunk map cannot persist readiness state.
            let d_size = c.uncompressed_size() asusize;
            matchself.read_file_cache(c.as_ref(), &mutbuf[0..d_size]) {
                // The cached data is valid, set the chunk as ready.
                Ok(_v) => self.update_chunk_pending_status(c.as_ref(), true),
                // The cached data is invalid, queue the chunk for reading from backend.
                Err(_e) =>pending.push(c.clone()),
            }
        }
    } else {
        for c in range.chunks.iter() {
            ifletOk(true) = self.chunk_map.check_ready_and_mark_pending(c.as_ref()) {
                // The chunk is ready, so skip it.
                continue;
            } else {
pending.push(c.clone());
            }
        }
    }
    let muttotal_size = 0;
    let mutstart = 0;
    whilestart <pending.len() {
        // Figure out the range with continuous chunk ids, be careful that `end` is inclusive.
        let mutend =start;
        whileend <pending.len() - 1 &&pending[end + 1].id() ==pending[end].id() + 1 {
end+= 1;
        }
        let (blob_offset, _blob_end, blob_size) = self.get_blob_range(&pending[start..=end])?;
        matchself.read_chunks_from_backend(blob_offset, blob_size, &pending[start..=end], true)
        {
            Ok(mutbufs) => {
total_size+= blob_size;
                ifself.is_compressed {
                    let res = Self::persist_cached_data(
                        &self.file,
                        blob_offset,
bufs.compressed_buf(),
                    );
                    for c inpending.iter().take(end + 1).skip(start) {
                        self.update_chunk_pending_status(c.as_ref(), res.is_ok());
                    }
                } else {
                    for idx instart..=end {
                        let buf = matchbufs.next() {
                            None => returnErr(einval!("invalid chunk decompressed status")),
                            Some(Err(e)) => {
                                forchunk in &mutpending[idx..=end] {
                                    self.update_chunk_pending_status(chunk.as_ref(), false);
                                }
                                returnErr(e);
                            }
                            Some(Ok(v)) => v,
                        };
                        self.persist_chunk_data(pending[idx].as_ref(), &buf);
                    }
                }
            }
            Err(_e) => {
                // Clear the pending flag for all chunks in processing.
                forchunk in &mutpending[start..=end] {
                    self.update_chunk_pending_status(chunk.as_ref(), false);
                }
            }
        }
start =end + 1;
    }
    Ok(total_size)
}

明确需要获取的数据 range 后，直接调用read_chunks_from_backend从 backend 读取内容。

2.6.2 初始化 PassthroughFs backend

创建 fs 配置信息实例，根据配置信息创建 PassthroughFs 实例：

let fs_cfg = Config {
    root_dir: cmd.source.to_string(),
    do_import: false,
    writeback: true,
    no_open: true,
    xattr: true,
    ..Default::default()
};
// TODO: Passthrough Fs needs to enlarge rlimit against host. We can exploit `MountCmd`
// `config` field to pass such a configuration into here.
let passthrough_fs =
    PassthroughFs::<()>::new(fs_cfg).map_err(DaemonError::PassthroughFs)?;
passthrough_fs
    .import()
    .map_err(DaemonError::PassthroughFs)?;
info!("PassthroughFs imported");
Ok(Box::new(passthrough_fs))

创建 PassthroughFs 实例：

/// Create a Passthrough file system instance.
pubfn new(cfg: Config) -> io::Result<PassthroughFs<S>> {
    // Safe because this is a constant value and a valid C string.
    let proc_self_fd_cstr = unsafe { CStr::from_bytes_with_nul_unchecked(PROC_SELF_FD_CSTR) };
    // 打开 /proc/self/fd 文件
    let proc_self_fd = Self::open_file(
        libc::AT_FDCWD,
        proc_self_fd_cstr,
        libc::O_PATH | libc::O_NOFOLLOW | libc::O_CLOEXEC,
        0,
    )?;
    Ok(PassthroughFs {
        inode_map: InodeMap::new(),
        next_inode: AtomicU64::new(fuse::ROOT_ID + 1),
        handle_map: HandleMap::new(),
        next_handle: AtomicU64::new(1),
        mount_fds: MountFds::new(),
        proc_self_fd,
        writeback: AtomicBool::new(false),
        no_open: AtomicBool::new(false),
        no_opendir: AtomicBool::new(false),
        killpriv_v2: AtomicBool::new(false),
        no_readdir: AtomicBool::new(cfg.no_readdir),
        perfile_dax: AtomicBool::new(false),
        cfg,
        phantom: PhantomData,
    })
}

passthrough_fs.import() 初始化文件系统。

/// Initialize the Passthrough file system.
pubfn import(&self) -> io::Result<()> {
    let root = CString::new(self.cfg.root_dir.as_str()).expect("CString::new failed");
    let (file_or_handle, st, ids_altkey, handle_altkey) = Self::open_file_or_handle(
        self.cfg.inode_file_handles,
        libc::AT_FDCWD,
        &root,
        &self.mount_fds,
        |fd, flags, _mode| {
            let pathname = CString::new(format!("{}", fd))
                .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
            Self::open_file(self.proc_self_fd.as_raw_fd(), &pathname, flags, 0)
        },
    )
    .map_err(|e| {
        error!("fuse: import: failed to get file or handle: {:?}", e);
        e
    })?;
    // Safe because this doesn't modify any memory and there is no need to check the return
    // value because this system call always succeeds. We need to clear the umask here because
    // we want the client to be able to set all the bits in the mode.
    unsafe { libc::umask(0o000) };
    // Not sure why the root inode gets a refcount of 2 but that's what libfuse does.
    self.inode_map.insert(
        fuse::ROOT_ID,
        InodeData::new(
            fuse::ROOT_ID,
            file_or_handle,
            2,
            ids_altkey,
            st.get_stat().st_mode,
        ),
        ids_altkey,
        handle_altkey,
    );
    Ok(())
}

初始化 backend 文件系统完成。

回到daemon.service.mount(cmd)方法。接下来，通过self.get_vfs().mount(backend, &cmd.mountpoint)方法挂载 backend 文件系统：

/// Mount a backend file system to path
pubfn mount(&self, fs: BackFileSystem, path: &str) -> VfsResult<VfsIndex> {
    let (entry, ino) = fs.mount().map_err(VfsError::Mount)?;
    if ino > VFS_MAX_INO {
        fs.destroy();
        returnErr(VfsError::InodeIndex(format!(
            "Unsupported max inode number, requested {} supported {}",
            ino, VFS_MAX_INO
        )));
    }
    // Serialize mount operations. Do not expect poisoned lock here.
    let _guard = self.lock.lock().unwrap();
    ifself.initialized() {
        let opts = self.opts.load().deref().out_opts;
        fs.init(opts).map_err(|e| {
            VfsError::Initialize(format!("Can't initialize with opts {:?}, {:?}", opts, e))
        })?;
    }
    let index = self.allocate_fs_idx().map_err(VfsError::FsIndex)?;
    self.insert_mount_locked(fs, entry, index, path)
        .map_err(VfsError::Mount)?;
    Ok(index)
}

首先，通过fs.mount()方法获取 backend 文件系统root inode的entry和最大的inode，对于 RAFS:

impl BackendFileSystem for Rafs {
    fn mount(&self) -> Result<(Entry, u64)> {
        let root_inode = self.sb.get_inode(self.root_ino(), self.digest_validate)?;
        self.ios.new_file_counter(root_inode.ino());
        let e = self.get_inode_entry(root_inode);
        // e 为 root inode 的 entry，第二个参数是支持的最大 inode 值
        Ok((e, self.sb.get_max_ino()))
    }
    ...
}

然后，通过self.allocate_fs_idx()方法分配可用的index:

由于nydus通过index区分不同的pseudofs文件系统（具体来说，长度为 64 位的 inode 中前 8 位），因此，最多可以有 256 个pseudofs文件系统。

接下来，通过self.insert_mount_locked(fs, entry, index, path)方法挂载path，并且将index和新建pseudofs的entry关联起来：

fn insert_mount_locked(
    &self,
    fs: BackFileSystem,
    mutentry: Entry,
    fs_idx: VfsIndex,
    path: &str,
) -> Result<()> {
    // The visibility of mountpoints and superblocks:
    // superblock should be committed first because it won't be accessed until
    // a lookup returns a cross mountpoint inode.
    let mutsuperblocks = self.superblocks.load().deref().deref().clone();
    let mutmountpoints = self.mountpoints.load().deref().deref().clone();
    // 挂载 path，得到 inode
    let inode = self.root.mount(path)?;
    let real_root_ino =entry.inode;
    // 根据 index 对 inodes 进行 hash
entry.inode = self.convert_inode(fs_idx,entry.inode)?;
    // 如果已经存在 mountpoint，先设置为 None
    // Over mount would invalidate previous superblock inodes.
    ifletSome(mnt) =mountpoints.get(&inode) {
superblocks[mnt.fs_idx asusize] = None;
    }
superblocks[fs_idx asusize] = Some(Arc::new(fs));
    self.superblocks.store(Arc::new(superblocks));
    trace!("fs_idx {} inode {}", fs_idx, inode);
    let mountpoint = Arc::new(MountPointData {
        fs_idx,
        ino: real_root_ino,
        root_entry:entry,
        _path: path.to_string(),
    });
    // 将新的 mount 添加到 self.mountpoints
mountpoints.insert(inode, mountpoint);
    self.mountpoints.store(Arc::new(mountpoints));
    Ok(())
}

其中，self.root.mount(path)方法创建新的pseudofs，如果path对应的pseudofs已经存在，则直接返回，否则，创建新的pseudofs：

// mount creates path walk nodes all the way from root
// to @path, and returns pseudo fs inode number for the path
pubfn mount(&self, mountpoint: &str) -> Result<u64> {
    let path = Path::new(mountpoint);
    if !path.has_root() {
        error!("pseudo fs mount failure: invalid mount path {}", mountpoint);
        returnErr(Error::from_raw_os_error(libc::EINVAL));
    }
    letmut inodes = self.inodes.load();
    letmut inode = &self.root_inode;
    'outer: for component in path.components() {
        trace!("pseudo fs mount iterate {:?}", component.as_os_str());
        match component {
            Component::RootDir => continue,
            Component::CurDir => continue,
            Component::ParentDir => inode = inodes.get(&inode.parent).unwrap(),
            Component::Prefix(_) => {
                error!("unsupported path: {}", mountpoint);
                returnErr(Error::from_raw_os_error(libc::EINVAL));
            }
            Component::Normal(path) => {
                let name = path.to_str().unwrap();
                // Optimistic check without lock.
                for child in inode.children.load().iter() {
                    if child.name == name {
                        inode = inodes.get(&child.ino).unwrap();
                        continue'outer;
                    }
                }
                ...
                // 没找到对应 name 的 node，新建
                let new_node = self.create_inode(name, inode);
                inodes = self.inodes.load();
                inode = inodes.get(&new_node.ino).unwrap();
            }
        }
    }
    // Now we have all path components exist, return the last one
    Ok(inode.ino)
}

self.convert_inode(fs_idx, entry.inode)方法将pseudofs的 inode 根据 index 进行偏移，避免多个pseudofs的 inode 相同：

// 1. Pseudo fs 的根 inode 不进行 hash
// 2. 由于 Index 总是大于 0，因此 pseudo fs 的 inodes 不受影响（也会进行 hash）
// 3. 其它 inodes通过 (index << 56 | inode) 进行 hash
fn convert_inode(&self, fs_idx: VfsIndex, inode: u64) -> Result<u64> {
    // Do not hash negative dentry
    if inode == 0 {
        returnOk(inode);
    }
    if inode > VFS_MAX_INO {
        returnErr(Error::new(
            ErrorKind::Other,
            format!(
                "Inode number {} too large, max supported {}",
                inode, VFS_MAX_INO
            ),
        ));
    }
    let ino: u64 = ((fs_idx asu64) << VFS_INDEX_SHIFT) | inode;
    trace!(
        "fuse: vfs fs_idx {} inode {} fuse ino {:#x}",
        fs_idx,
        inode,
        ino
    );
    Ok(ino)
}

挂载 backend 文件系统结束。

根据mount_cmd准备好文件系统后端（例如，RAFS backend），接下来通过 FUSE 进行挂载。daemon.service.session.lock().unwrap().mount()函数是fuse-backend-rs中FuseSession结构体的方法：

在fuse_kern_mount方法中，准备好需要的参数后，会调用nix crate 中的mount方法，这个方法最终调用了libc中的mount函数：

接下来，会向状态机线程发送Mount和Start两个事件，状态机的变化如下：

当状态转换为StartService时，会执行上面分析的d.start()方法，最终将状态修改为RUNNING：

StartService => d.start().map(|r| {
    d.set_state(DaemonState::RUNNING);
    r
}),

“

nydusd 在运行期间有 8 个线程，到目前为止，我们已经启动了其中的 6 个线程（fuse_server 的数量可以配置），接下来，还要启动两个线程 nydus-http-server 和 api-server。

最后，获取挂载点的 major 和 minor 信息，存储在元数据中。

create_fuse_daemon() 方法执行完成后，如果成功会打印如下日志信息：

参考资料

[1] nydus: https://github.com/dragonflyoss/image-service.git

[2] fuse-backend-rs: https://github.com/cloud-hypervisor/fuse-backend-rs