request: squashfs in the linux kernel

Kernel, Main, Utilities & Apps, Misc Devices
Johnny Love
Posts: 9
Joined: Sun Aug 16, 2020 3:51 pm

request: squashfs in the linux kernel

Unread post by Johnny Love »

Just an example, the 2019-03-04 Neo Geo Roll-UP BETA 3.1 7zip file is 2.5GB

MiSTer can't read it, so you have to extract it, then it ends up being 7.3GB

After a few minutes of compression with "gensquashfs -D . -c xz -b 1048576 ../squash.neogeo.img" it goes from being 1447 uncompressed files, down to a single file that is only 1.8GB, smaller than the original 7zip file.

Unlike zipped roms, this would require no changes to cores or MiSTer itself. There are no downsides when it comes to rom files other than slightly slower rom loading since the ARM core would have to do the work, transparently behind the scenes. It might be an issue with ISO images, and of course would not work with hard drive disk images that need the ability to write back to them unless copy-on-write was used to write the changes to another file. For any kind of ROM based system like Neo Geo or below, this is the only way I would do it if I had the option.
Johnny Love
Posts: 9
Joined: Sun Aug 16, 2020 3:51 pm

Re: request: squashfs in the linux kernel

Unread post by Johnny Love »

If anyone is interested, I have compiled a kernel with squashfs support. See https://github.com/MiSTerJohnnyLove/MiSTerSquashfs and https://redd.it/ibl8j3
ZigZag
Posts: 72
Joined: Sun Jun 28, 2020 7:05 am

Re: request: squashfs in the linux kernel

Unread post by ZigZag »

Johnny Love wrote: Mon Aug 17, 2020 7:53 pm If anyone is interested, I have compiled a kernel with squashfs support. See https://github.com/MiSTerJohnnyLove/MiSTerSquashfs and https://redd.it/ibl8j3
Thanks, I'll have to test it sometime.
retrorepair
Posts: 166
Joined: Sun May 24, 2020 9:06 pm

Re: request: squashfs in the linux kernel

Unread post by retrorepair »

Very interesting, especially as it is optional and only runs on the linux side.

How easy is it to compress/decompress these files and how does the menu handle them? Like folders? I imagine it would have to if you have the entire Neo Geo set in one file.
Johnny Love
Posts: 9
Joined: Sun Aug 16, 2020 3:51 pm

Re: request: squashfs in the linux kernel

Unread post by Johnny Love »

It takes quite a lot of CPU resources to compress them, but I only plan on doing it once a year or so. Probably RAM too, but I didn't check. So far I haven't really noticed much of a change at all, and I have most cores working now, excluding the cdrom based ones. Decompression is always easier than the initial compression.

It is literally a filesystem, so it looks the same from within the menu. The squashfs images get mounted out of the way, and then whatever is in them gets symlinked into the Games/CORE/ folder. It is best for the only thing to appear in the root of the squashfs image to be the folder/directory for a rom pack so only one symlink has to be made. it Doesn't matter, there can be more than one thing and they will all get symlinked in. You can also have more than one squashfs per core.
User avatar
Sorgelig
Site Admin
Posts: 416
Joined: Thu May 21, 2020 9:49 pm

Re: request: squashfs in the linux kernel

Unread post by Sorgelig »

Looks like linux style: do something a whole day digging in config files.
While MiSTer becomes more and more complicated i still try to keep it in plug-and-play form.
Mounting additional images is not user friendly. Pressing on zip to open it as a folder is more user friendly even though it can be a little bigger.
Your original problem with oversized zip can be solved by splitting to smaller zips like A.zip B.zip .. Z.zip.
Johnny Love
Posts: 9
Joined: Sun Aug 16, 2020 3:51 pm

Re: request: squashfs in the linux kernel

Unread post by Johnny Love »

I think you may be misinterpreting what my goal is here. Squashfs is part of the linux kernel. All I had to do was grab the MiSTer kernel source code from github, go into the filesystem settings of the kernel, and turn on squashfs support before compiling it. I enabled all of the compression options so people can experiment, but xz seems to be the best so far.

There's no digging through config files all day.You simply take a squashfs image file (which is about as easy as making a zip file at a command prompt, really incredibly easy) and upload it to /media/fat/squashfs/CORE/ and the next time you reboot, you now have access to all of those files that were compressed with XZ compression, not zip files which are hardly compressed at all.

It might not seem user friendly to "mount additional images" and I get that, that's why I made some bash scripts that do it for you. It doesn't even have to be run manually once it is sitting in /etc/init.d, it just runs at boot.

In any case I don't mind compiling a kernel for this purpose, it just seems easy enough to add to the default kernel for those that want it. It saves a lot of sd card space and if you don't know its there, it can't do anything to make your experience worse. It also isn't proprietary or hard to access, the windows executable to make these images works the same as the linux one.
KremlingKuthroat19
Posts: 28
Joined: Sat Aug 22, 2020 3:08 am

Re: request: squashfs in the linux kernel

Unread post by KremlingKuthroat19 »

I'm no expert whatsoever on the topic of compression and decompression. I have experience using CHD and love using it in Retroarch because it not only shrinks the file sizes drastically, but it conveniently makes each bin/cue into a single file with a CHD extension. It's very clean and snappy. I'm wondering if you created a CHD and zipped it if it would make the file even smaller? (obviously at the cost of booting time). I understand that CHD might be too obscure of a compression method to implement, but I've used it and can vouch for it.

I digress. The point I'm making is that I've never heard of squashfs until reading this thread, but if it's a good compression method then I'm on board. However, there comes a certain point in time that I believe there should be a compression method that's simple and compatible with all bin/cues and ideally works with all other rom extensions for the MiSTer project. I say this because MiSTer currently supports Sega CD and PC Engine CD libraries which each have a few hundred games. However, once the PS1 core comes out (who knows when exactly honestly), then we may have a problem that needs to be addressed. The PS1 has 2500+ games. That library is absolutely massive and each game is around 700 MB.

TLDR: If there was a universal compression method that MiSTer adopted, it could futureproof this issue by the time the PS1 core becomes available. Obviously I don't want to undermine the work requied to make this feature a reality, but just wanted to give my two cents.
User avatar
tankjr
Posts: 2
Joined: Sun May 31, 2020 4:14 am

Re: request: squashfs in the linux kernel

Unread post by tankjr »

Great work! Your update and mount scripts are finicky and didn't want to play nice with my system.

I compressed all the data on my main Linux machine and transferred it onto my SD card. I managed to get everything working by reading your scripts and doing all the heavy lifting myself over an ssh terminal. I've easily freed up 20gb of space on my 128gb card and my load times have only increased slightly. It's not like I'm swapping games every minute... so the change in load time doesn't bother me. Big thumbs up overall.

I think there are ways to make it more user friendly. As is, it works great, but it's definitely tailored for someone who knows there way around a Linux machine.
User avatar
jrronimo
Posts: 34
Joined: Sat Jul 04, 2020 6:03 pm
Has thanked: 1 time

Re: request: squashfs in the linux kernel

Unread post by jrronimo »

tankjr wrote: Tue Oct 13, 2020 4:53 am Great work! Your update and mount scripts are finicky and didn't want to play nice with my system.

I compressed all the data on my main Linux machine and transferred it onto my SD card. I managed to get everything working by reading your scripts and doing all the heavy lifting myself over an ssh terminal. I've easily freed up 20gb of space on my 128gb card and my load times have only increased slightly. It's not like I'm swapping games every minute... so the change in load time doesn't bother me. Big thumbs up overall.

I think there are ways to make it more user friendly. As is, it works great, but it's definitely tailored for someone who knows there way around a Linux machine.
Is this with CHD or with SquashFS? Either way, it sounds like a win to me!
User avatar
tankjr
Posts: 2
Joined: Sun May 31, 2020 4:14 am

Re: request: squashfs in the linux kernel

Unread post by tankjr »

jrronimo wrote: Wed Oct 14, 2020 9:52 pm Is this with CHD or with SquashFS? Either way, it sounds like a win to me!
Sorry for the late reply. This was with SquashFS.
Post Reply