Implement FuseFS — wrap around FileSystem that implements fuser::Filesystem trait to mount it. by Ycyken · Pull Request #7 · Piletskii-Oleg/chunkfs

Ycyken · 2025-04-05T00:42:03Z

Also added the ability to read the exact size from a file and at a specific offset.
Remove Hash copying in FileLayer::read_complete and FileLayer::read methods.

Piletskii-Oleg

неплох

Piletskii-Oleg · 2025-04-05T16:03:14Z


    /// Retrieves a multitude of values, corresponding to the keys, in the correct order.
-    fn get_multi(&self, keys: &[K]) -> io::Result<Vec<V>> {
+    fn get_multi(&self, keys: Vec<&K>) -> io::Result<Vec<V>> {


Suggested change

fn get_multi(&self, keys: Vec<&K>) -> io::Result<Vec<V>> {

fn get_multi(&self, keys: &[&K]) -> io::Result<Vec<V>> {

так работает?

Piletskii-Oleg · 2025-04-05T16:03:40Z

+    }
+
+    pub fn offset(&self) -> &usize {
+        &self.offset


usize предпочтительнее чем &usize (они занимают одинаково места в памяти)

Piletskii-Oleg · 2025-04-05T16:10:34Z

+            .storage
+            .retrieve(spans.iter().map(|span| span.hash()).collect())?;
+
+        let first_span = spans.first().unwrap();


в чем смысл этих двух действий внизу?

Т.к. мы читаем по оффсету, который может быть не ровно по оффсету FileSpan, а где-то посередине, то надо убрать лишнее в начале первого спана и лишнее в конце последнего спана. Добавил побольше комментариев

Piletskii-Oleg · 2025-04-05T16:11:11Z

    /// Retrieves the data from the storage based on hashes of the data [`segments`][Segment],
    /// or Error(NotFound) if some of the hashes were not present in the base.
-    pub fn retrieve(&self, request: &[Hash]) -> io::Result<Vec<Vec<u8>>> {
+    pub fn retrieve(&self, request: Vec<&Hash>) -> io::Result<Vec<Vec<u8>>> {


Suggested change

pub fn retrieve(&self, request: Vec<&Hash>) -> io::Result<Vec<Vec<u8>>> {

pub fn retrieve(&self, request: &[&Hash]) -> io::Result<Vec<Vec<u8>>> {

и снизу можно будет убрать .iter().collect() который не к месту

Piletskii-Oleg · 2025-04-05T19:25:25Z

+    }
+
+    pub fn offset(&self) -> usize {
+        self.offset.clone()


clone не нужен, usize маленький тип и имплементит Copy, потому он автоматически скопируется

Piletskii-Oleg · 2025-04-05T19:26:12Z

+    FileAttr, FileType, Filesystem, ReplyAttr, ReplyCreate, ReplyData, ReplyDirectory, ReplyEmpty,
+    ReplyEntry, ReplyOpen, ReplyWrite, Request, TimeOrNow,
+};
+use libc::{


убери эту строку импорта и вставь везде префикс libc где используются константы

Piletskii-Oleg · 2025-04-16T11:08:27Z

+
+    let session = fuser::spawn_mount2(fuse_fs, MOUNT_POINT, &vec![]).unwrap();
+
+    let file_path = format!("{}/{}", MOUNT_POINT, "file");


можно MOUNT_POINT сделать Path и file_path инициализировать через MOUNT_POINT.join(...)

Piletskii-Oleg · 2025-04-16T11:11:40Z

+        }
+        let last_span = spans.last().unwrap();


Suggested change

}

let last_span = spans.last().unwrap();

}

let last_span = spans.last().unwrap();

Piletskii-Oleg · 2025-04-16T11:14:40Z

+        // Since we read by offset, which may be somewhere in the middle of the FileSpan offset,
+        // we need to remove the extra at the beginning of the first span
+        let first_span = spans.first().unwrap();
+        let last_span = spans.last().unwrap();


перекинь last_span чуть ниже, перед read_size_possible

Piletskii-Oleg · 2025-04-16T11:15:45Z

+    /// Starting point is based on the `FileHandle`'s offset.
+    ///
+    /// If `size` + file handle offset is greater than file size, then returns FileSpans up to the end of the file.
+    pub fn read(&self, handle: &mut FileHandle, size: usize) -> Vec<&FileSpan<Hash>> {


почему он возвращает вектор FileSpan а не Hash?

Потому что если возвращать просто хэши, то будет непонятно, как обрезать данные в FileSystem::read(), т.к. информация об оффсетах потерялась. Можно попробовать возвращать вместе с хэшами какую-то информацию о начале и конце первого/последнего спана, но звучит не очень.

Piletskii-Oleg · 2025-04-16T11:16:36Z

-    pub fn read_from_file(&self, handle: &mut FileHandle) -> io::Result<Vec<u8>> {
-        let hashes = self.file_layer.read(handle);
-        Ok(self.storage.retrieve(&hashes)?.concat())
+    pub fn read_1mb_from_file(&self, handle: &mut FileHandle) -> io::Result<Vec<u8>> {


лучше вернуть на read_from_file

Piletskii-Oleg · 2025-04-16T11:20:07Z

по-хорошему бы накидать миллион тестов, проверяющих инварианты фьюза (не можем открыть файл к которому нет прав доступа, не можем создать папку, лукап работает верно, связка гетаттр/сетаттр и т.д.),

а чтобы в каждом из них не инициализировать это все в несколько строк, можно написать структурку FuseFixture, у которой будет какой-нибудь new или default, в нем будет собственно инициализация, монтирование и создание файла. к нему можно прикрутить методы типа "получить этот файл", а затем создавать через let fs = FuseFixture::new(); let file = fs.file()

тестов пока очень мало

Добавил тестов

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…f fh counter Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…orrect read() Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…with_chunker

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Piletskii-Oleg

норм, замечания по мелочи

Piletskii-Oleg · 2025-04-30T12:12:31Z

+        Ok(())
+    }
+
+    fn drop_cache(&mut self, file: Inode, handle: Fh) -> io::Result<()> {


дока...

добавил больше документации

Piletskii-Oleg · 2025-04-30T12:19:39Z

+}
+#[test]


пробельчики

Suggested change

}

#[test]

}

#[test]

Piletskii-Oleg · 2025-04-30T12:21:16Z

+    };
+    let read_denied = || {
+        let res = OpenOptions::new().read(true).open(&file_path);
+        assert!(res.is_err());


какой именно err?

добавил проверку на PermissionDenied

Piletskii-Oleg · 2025-04-30T12:21:30Z

+
+    let dir_path = mount_point.join("directory");
+    let res = fs::create_dir(&dir_path);
+    assert!(res.is_err());


какой err?

Piletskii-Oleg · 2025-04-30T12:24:48Z

+}
+
+#[test]
+fn offset_change_not_affects_cache_drop() {


Suggested change

fn offset_change_not_affects_cache_drop() {

fn offset_change_does_not_affect_cache_drop() {

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

… behaviour Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…ry file's content change Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…possible inconsistency Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

…bench Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken changed the title ~~Implement FuseFS - wrap around FileSystem that implements fuser::Filesystem trait.~~ Implement FuseFS — wrap around FileSystem that implements fuser::Filesystem trait to mount it. Apr 5, 2025

Piletskii-Oleg reviewed Apr 5, 2025

View reviewed changes

Ycyken requested a review from Piletskii-Oleg April 9, 2025 20:41

Piletskii-Oleg reviewed Apr 16, 2025

View reviewed changes

Ycyken force-pushed the fuse branch from 1b158e4 to 19fe108 Compare April 20, 2025 16:53

Ycyken added 20 commits April 20, 2025 21:00

create FuseFS, implement basic methods

d15338f

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add Send trait bound for Chunker, Hasher, Scrub

92ab57f

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

implement read with exact size, FileHandle::set_offset()

7071341

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

finalized write and read in fuse_filesystem

014c678

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add some docs to fuse_filesystem

9bf46ff

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add fuse fs test

5581bdf

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

change FileHandle::offset return type from &usize to usize

0397aff

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add ability to change offset of any file handle

9b1d094

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add example of FS mounting

bac565a

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

change FileSpan::{offset, len} return type from &usize to usize

5b854ac

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add more comments in FileSystem::read method

5b73523

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

change keys type from Vec<&K> to &[&K] in Database::get_multi

b1e54ca

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

refactor: libc flags import, remove usize.clone()

e5e068d

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

some refactor

117bf92

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix FuseFS::write() - was checking permission for read instead of write

5f22426

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add permissions tests for FuseFS

bf74f04

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

remove metadata change possibility with write access, add ctime change

7e77eda

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add atime and ctime update in read()

cb5f6af

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add fuse tests: readdir, metadata time

db0cfe0

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix release(): was removing by file handle by inode counter instead o…

b60c015

…f fh counter Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken force-pushed the fuse branch from 2e88256 to b60c015 Compare April 20, 2025 18:03

Ycyken requested a review from Piletskii-Oleg April 20, 2025 18:07

Ycyken added 2 commits April 21, 2025 22:28

fix clippy warnings

7719512

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add installation of FUSE dependencies in CI

dc903c9

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken added 5 commits April 26, 2025 05:25

add O_DIRECT to all write-read fuse tests: cache can return wrongly c…

bcdbfa9

…orrect read() Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fuse tests: replace SimpleHasher with Sha256Hasher, add FuseFixture::…

542644e

…with_chunker

add fuse tests with boundary chunk reads

c6de056

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add cache drop test

a88e74a

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

some refactor

a246e1e

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken closed this Apr 26, 2025

Ycyken reopened this Apr 26, 2025

Piletskii-Oleg reviewed Apr 30, 2025

View reviewed changes

Ycyken added 5 commits May 1, 2025 03:31

ref: typos, impl order, spaces

933a8e8

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

test: change error assertion to concrete error type

96f46fe

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix readdir test

f646ab9

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix lookup permission check: FUSE was sendind EIO on incorrect daemon…

f055e5c

… behaviour Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add more docs to FuseFS

fe221ae

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken force-pushed the fuse branch from 2e50bbb to fe221ae Compare May 1, 2025 04:59

Ycyken requested a review from Piletskii-Oleg May 1, 2025 16:30

Ycyken added 3 commits May 2, 2025 08:41

fix EIO FUSE error causing by incorrect file generation update on eve…

e843aa1

…ry file's content change Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

move reply in FuseFS::create() after all filesystem updates to avoid …

72afbaf

…possible inconsistency Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add O_DIRECT flag to fuse benches

9d579f4

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken force-pushed the fuse branch 3 times, most recently from f1c61a4 to 77dbc11 Compare May 8, 2025 10:53

Ycyken added 4 commits May 8, 2025 15:46

fix FuseFS: total cache was not increasing in write()

f1f0ec5

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

add dedup ratio and avg chunk size request for FuseFS via ioctl

8cb0939

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

update Readme with native dependencies for FUSE

de92078

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix global cache drop with concurrent file handles in FuseFS

6656e49

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Ycyken force-pushed the fuse branch from 77dbc11 to 6656e49 Compare May 8, 2025 12:48

Ycyken and others added 4 commits May 8, 2025 16:11

change custom Dataset structure to bench::Dataset in fuse_write_read …

6310a90

…bench Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

fix benches: super and ultra were mix up

b70e410

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

increase FuseFS cache sizes

213e1a5

Signed-off-by: Gleb Nasretdinov <gleb.nasretdinov@proton.me>

Merge branch 'main' into fuse

c33c321

	fn get_multi(&self, keys: Vec<&K>) -> io::Result<Vec<V>> {
	fn get_multi(&self, keys: &[&K]) -> io::Result<Vec<V>> {

	pub fn retrieve(&self, request: Vec<&Hash>) -> io::Result<Vec<Vec<u8>>> {
	pub fn retrieve(&self, request: &[&Hash]) -> io::Result<Vec<Vec<u8>>> {


		let session = fuser::spawn_mount2(fuse_fs, MOUNT_POINT, &vec![]).unwrap();

		let file_path = format!("{}/{}", MOUNT_POINT, "file");

-}
-#[test]
+}
+#[test]

	fn offset_change_not_affects_cache_drop() {
	fn offset_change_does_not_affect_cache_drop() {

Conversation

Ycyken commented Apr 5, 2025

Uh oh!

Piletskii-Oleg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Piletskii-Oleg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants