sym-links on exFAT

Kernel, Main, Utilities & Applications, Miscellaneous Devices.
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

sym-links on exFAT

Unread post by Sorgelig »

There is a good and bad "news" about symlinks on exFAT used on MiSTer.
Good: I guess, many already noticed you can create a symlink on exFAT.
Bad: it's not native feature of exFAT (and any other FAT) file system and Linux uses undocumented reserved flag for that. It works till you try to copy files on Windows and other OS supporting exFAT. Once you copy symlink, it becomes a generic text file. It won't be recognized as symlink anymore.

What i plan to do is to change the symlink flag to system flag (S). It's documented file attribute and pretty much useless in Linux. When you will copy such file, it should be preserved - it of course depends on app where you copy, but at least you can find the way to copy and keep the attributes.
12characters
Posts: 112
Joined: Wed Jun 03, 2020 5:39 pm
Has thanked: 8 times
Been thanked: 30 times

Re: sym-links on exFAT

Unread post by 12characters »

As a linux user accustomed to symlinks, I think it is good news that one has found a way to implement them on exFAT. With the planned change, will symlinks created by the vanilla linux driver still be recognised as such by MiSTer and vice versa?
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

Re: sym-links on exFAT

Unread post by Sorgelig »

I don't know about other systems. FAT file systems don't support linux links. So it's workaround just for MiSTer.
User avatar
keith.f.kelly
Posts: 88
Joined: Sun May 24, 2020 11:10 pm
Location: Sammamish, WA, USA
Has thanked: 3 times
Been thanked: 8 times
Contact:

Re: sym-links on exFAT

Unread post by keith.f.kelly »

Is there some good reason to keep using exFAT instead of moving to an ext* filesystem? Is it just for convenience for Windows users? There are ext* filesystem drivers available for Windows these days.
tRAcE
Posts: 13
Joined: Tue May 26, 2020 11:14 am

Re: sym-links on exFAT

Unread post by tRAcE »

Keith, really, can you point me to a usable ext* driver for Windows?

I think exFat is a nice solution as it is read-/writeable on all mayor systems natively these days.
Only caveat are the symlinks, but honestly I for my part don't use a single one on my cards. But as always YMMV :)
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

Re: sym-links on exFAT

Unread post by Sorgelig »

exFAT is pretty much right file system for MiSTer.
It doesn't have complexities such as access rights, case sensitivity and supported in all OS.
Links are basically not supposed to be used in ROM folders. They give some convenience but not necessary.
User avatar
zomgugoff
Posts: 33
Joined: Sun May 24, 2020 8:54 pm

Re: sym-links on exFAT

Unread post by zomgugoff »

The Nintendo Switch modding community largely dissuades the use of exFAT on the SD card because it's prone to corruption. My understanding of the reason for it is down to the driver implementation. I assume the driver being used for MiSTer is more robust.
User avatar
keith.f.kelly
Posts: 88
Joined: Sun May 24, 2020 11:10 pm
Location: Sammamish, WA, USA
Has thanked: 3 times
Been thanked: 8 times
Contact:

Re: sym-links on exFAT

Unread post by keith.f.kelly »

tRAcE wrote: Sun Jun 07, 2020 8:02 am Keith, really, can you point me to a usable ext* driver for Windows?
https://www.paragon-software.com/us/hom ... s-windows/
User avatar
keith.f.kelly
Posts: 88
Joined: Sun May 24, 2020 11:10 pm
Location: Sammamish, WA, USA
Has thanked: 3 times
Been thanked: 8 times
Contact:

Re: sym-links on exFAT

Unread post by keith.f.kelly »

So, will something bad happen if I use chkdsk /F against the exFAT partition (given that it is somehow non-standard) with the microSD card inserted into my Windows 10 machine? Because I just ran it without the /F flag, and it claims to have found problems:

Code: Select all

C:\Windows\System32>chkdsk G:
The type of the file system is exFAT.
Volume Serial Number is 864C-AC08
Windows is verifying files and folders...
Volume label is MiSTer_Data.
Windows found errors on the disk, but will not fix them
because disk checking was run without the /F (fix) parameter.
Corruption was found while examining the volume bitmap.
File and folder verification is complete.

Windows has checked the file system and found problems.
Run CHKDSK with the /F (fix) option to correct these.

 249855616 KB total disk space.
 180957184 KB in 23032 files.
    152448 KB in 1173 indexes.
         0 KB in bad sectors.
       384 KB in use by the system.
  68745600 KB available on disk.

    131072 bytes in each allocation unit.
   1951997 total allocation units on disk.
    537075 allocation units available on disk.

C:\Windows\System32>
User avatar
keith.f.kelly
Posts: 88
Joined: Sun May 24, 2020 11:10 pm
Location: Sammamish, WA, USA
Has thanked: 3 times
Been thanked: 8 times
Contact:

Re: sym-links on exFAT

Unread post by keith.f.kelly »

I had a recent backup, so I went ahead and ran chkdsk /F against the exFAT partition, and it seems to have fixed whatever problem it found without messing up any of my symlinks. So I guess that was a safe enough thing to do :)
User avatar
keith.f.kelly
Posts: 88
Joined: Sun May 24, 2020 11:10 pm
Location: Sammamish, WA, USA
Has thanked: 3 times
Been thanked: 8 times
Contact:

Re: sym-links on exFAT

Unread post by keith.f.kelly »

It appears that if a chkdsk-validated microSD exFAT partition is used in MiSTer, and then MiSTer is simply powered off from the OSD, it somehow corrupts the "volume bitmap" of that partition. At least, that's what chkdsk again reports if I remove the microSD from MiSTer and check it again on my Windows 10 PC. I repeated this cycle a couple times to confirm the issue.

So then I wrote a quick /media/fat/Scripts/shutdown.sh that I can easily run from the OSD:

Code: Select all

#!/bin/bash
echo Wait 5 seconds and then power off.
halt -f
As long as I use that shutdown script before powering down each time, then the exFAT partition still reads as good when I check the microSD again on my Windows PC.

So it seems the standard guidance about "just go to the OSD and power off" is still not great. It would be better to add a proper shutdown command to the main screen of the OSD, so the user doesn't have to go hunting through the Scripts folder just to shut down. (For now, I created a symlink /media/fat/Scripts/_shutdown.sh just to force it to appear first in the sort order, but that's still a bit of a kludge.)
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

Re: sym-links on exFAT

Unread post by Sorgelig »

There is some flag in exFAT telling that not all writeable files are closed before unmounting. It doesn't mean corruption.
So general guide remains the same. There WON'T be shutdown procedure for MiSTer. But if your config went too far and something constantly is writing to files, then you can invent some specific options like shutdown script. Up to you, but unnecessary for normal use.

Also chkdsk doesn't check extended unused attributes for files on exFAT. So it won't find anything. Just create some link file on MiSTer then plug SD card to Windows and copy this link file to other place on the same SD card. Then plug it back in MiSTer and you will see it's no longer a link but normal text file.
dukrous
Posts: 3
Joined: Sun May 24, 2020 7:07 pm

Re: sym-links on exFAT

Unread post by dukrous »

The only issue with symlinks is that each file will occupy the size of one block on the SD Card (128K for SD cards larger than 32GB). Sadly, this cannot be avoided as there is a strict 1 file per block limitation and symlink files are usually less than 100 bytes. Therefore symlinks on MiSTer right now only save space if the file is greater than 128K.
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

Re: sym-links on exFAT

Unread post by Sorgelig »

Abusing feature is usually a bad idea. Links on exFAT give some convenience, but making a tons of links isn't good.
amoore2600
Posts: 2
Joined: Sun May 24, 2020 6:46 pm

Re: sym-links on exFAT

Unread post by amoore2600 »

The block size seems to effect more than symlinks. Small files in general are effected. As they are forced to be at 128K minimum regardless of their size.

On larger SD cards this is not as felt but on smaller SD cards a lot of space can be eaten up by the block size.

One example that was recently pointed out to me was a 40MB NES set taking 70MB because of the block size.

I'd like to understand more of the reasons to set block size smaller or larger and what's effected and what is a good fit for MiSTer and it's users?

mkfs.exfat sets defaults at:
Number of physical sectors per cluster (cluster is an allocation unit in exFAT). Must be a power of 2, i.e. 1, 2, 4, 8, etc. Cluster size can not exceed 32 MB. Default cluster sizes are: 4 KB if volume size is less than 256 MB, 32 KB if volume size is from 256 MB to 32 GB, 128 KB if volume size is 32 GB or larger.
Does someone know the defaults for MiSTer in the SD installer and why are they choosen?
dukrous
Posts: 3
Joined: Sun May 24, 2020 7:07 pm

Re: sym-links on exFAT

Unread post by dukrous »

I checked out the SD Installer and it looks like BlockSize is not specified. So it'll just use the default you listed there.

The main issue you have with BlockSize is it's a compromise between used space per file and read/write speeds. If Blocks are large, then large files read/write faster but small files take up more space. If blocks are small, then small files take up less space but large files need more read/write time.

I'm wondering if there's a case to be made for allowing different FileSystems in MiSTer. exFAT could remain the default, but giving the option for ext4 would be a good power-user option for people to use more advanced techniques like file links to manage their ever-growing collection. The more cores are added on to MiSTer, the more cramped for space these SD cards will be thanks to the large block size and inability to use file links.
User avatar
Sorgelig
Site Admin
Posts: 877
Joined: Thu May 21, 2020 9:49 pm
Has thanked: 2 times
Been thanked: 211 times

Re: sym-links on exFAT

Unread post by Sorgelig »

32KB should be ideal for MiSTer needs. Probably sources of installer can be tweaked to set 32KB permanently. I just can't handle everything myself. So if someone will adjust and make PR, then it will be perfect.

MiSTer supports USB drives where you can keep large collection of ROMs and CDs, so you don't have to stick to SD card. It can be like 8GB only to hold just cores and essential files. Then you can plug some tiny USB stick with something like 512GB where you can use different kinds of file systems: FAT32/exFAT/NTFS/ext4. But be careful with ext4 as it's native Linux system and automatically applies restrictions such as case sensitivity, access rights and other redundant features not used by ROM/CD storage.

SD card file system is tightly bound to booting process and requires to be exFAT.
jca
Top Contributor
Posts: 1911
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 145 times
Been thanked: 454 times

Re: sym-links on exFAT

Unread post by jca »

You also have to be careful with NTFS. I recently discovered that NTFS is also case sensitive and I think a lot of people do not know about it as Windows ignores the case sensitivity in order not to disorient Windows users. The way NTFS files systems are mounted under MISTer preserve the case sensitivity. It is possible to tell Linux to be case insensitive with NTFS using the mount option -o ignore_case but it also requires mounting the drive using the type -t lowntfs-3g. I did not try it yet.
Post Reply