I think "/media/fat" should be depricated

Kernel, Main, Utilities & Applications, Miscellaneous Devices.
held
Posts: 185
Joined: Sun Sep 26, 2021 2:18 pm
Has thanked: 24 times
Been thanked: 29 times

I think "/media/fat" should be depricated

Unread post by held »

I find "/media/fat" to be a problem when data resides elsewhere. Should we start using a symlink "/media/MiSTer" to point to the actual data folder?

Benefits:
* path independent scripts that work on sdcard and usb
* easily switch from sdcard to usb storage by updating the symlink

Code: Select all

drwxr-xr-x  2 root root     40 Jan  1  1970 cifs
drwxr-xr-x 32 root root 131072 Jan  1  1970 fat
drwxr-xr-x  2 root root     40 Jan  1  1970 rootfs
drwxr-xr-x  2 root root     40 Jan  1  1970 usb0
drwxr-xr-x  2 root root     40 Jan  1  1970 usb1
drwxr-xr-x  2 root root     40 Jan  1  1970 usb2
lrwxrwxrwx  1 root root     40 Jan  1  1970 MiSTer -> fat

Code: Select all

drwxr-xr-x  2 root root     40 Jan  1  1970 cifs
drwxr-xr-x 32 root root     40 Jan  1  1970 fat
drwxr-xr-x  2 root root     40 Jan  1  1970 rootfs
drwxr-xr-x  2 root root 131072 Jan  1  1970 usb0
drwxr-xr-x  2 root root     40 Jan  1  1970 usb1
drwxr-xr-x  2 root root     40 Jan  1  1970 usb2
lrwxrwxrwx  1 root root     40 Jan  1  1970 MiSTer -> usb0
:idea:
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

A long time ago Sorge mentioned that "Switch to USB" was kind of obsolete and one should use symlinks. This is what I did and made many posts about it.
You can place about any folder on USB and use a symlink in media/fat.
This is what I did: I have everything on USB except config and Scripts.
A little annoyance if you place _Arcade, _Computer, .. on USB and the USB drive is not yet mounted when MISTer displays the menu you will get No Files!
Wait a little, press twice F12 to flip the OSD back and forth et voila.
FPGA64
Posts: 327
Joined: Mon Mar 01, 2021 3:10 pm
Has thanked: 31 times
Been thanked: 104 times

Re: I think "/media/fat" should be depricated

Unread post by FPGA64 »

You dont need to symlink anything. The USB external drive overlays the SD card. If a directory is on the external drive it replaces whatever is on the SD. if the directory is not present on the external drive then the SD directory is used. Its all automatic

The switch to USB option should not be touched
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

I did what Sorge said to do at the time and it works for me.
Good to know that you don't need the symlinks, I was under the impression that this automatic switch was only for the games directory.
It is hard to know what is going on on the Linux side of MISTer due to the paucity of information.
I have a few questions:
Are the directories located on USB mounted in /media/fat?
What about the update script, do you leave the default setting to the SD card? On my system with symlinks I left it to the SD card.
FPGA64
Posts: 327
Joined: Mon Mar 01, 2021 3:10 pm
Has thanked: 31 times
Been thanked: 104 times

Re: I think "/media/fat" should be depricated

Unread post by FPGA64 »

The external I only store Games om. There was no point moving the cores from the SD as they are small

The /media/usb0 is the external drive and on there I have a games directory exactly as it was on the SD

The Update script happily copies files to the external drive
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

FPGA64 wrote: Thu Nov 24, 2022 12:46 pm The external I only store Games om. There was no point moving the cores from the SD as they are small
...
They are small but are updated frequently -> SD card wear.
Do you still have an emptys games on the SD?
When I am close to my MISTer I will make some test.
FPGA64
Posts: 327
Joined: Mon Mar 01, 2021 3:10 pm
Has thanked: 31 times
Been thanked: 104 times

Re: I think "/media/fat" should be depricated

Unread post by FPGA64 »

No I have a full games directory on the SD card so it can be moved around without the external HD, well full bar the PSX. The external has it all and as such no SD card directory is observable in the cores
held
Posts: 185
Joined: Sun Sep 26, 2021 2:18 pm
Has thanked: 24 times
Been thanked: 29 times

Re: I think "/media/fat" should be depricated

Unread post by held »

It still leaves the scripting argument.
And how do we measure sd-card writes?
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

held wrote: Thu Nov 24, 2022 1:13 pm It still leaves the scripting argument.
And how do we measure sd-card writes?
If scripting refers to the location of the Scripts folder the problem is due to the fact that the USB drive is mounted with the NOEXEC option. If you want to run a script via SSH and the script is on USB you have to use "bash <scriptname>". I am not sure how a script is run if you run it via the Menu.
riblad_om
Posts: 40
Joined: Sun May 24, 2020 10:06 pm
Has thanked: 14 times
Been thanked: 2 times

Re: I think "/media/fat" should be depricated

Unread post by riblad_om »

I think you guys may be a bit paranoid about sd card wear. I have a Micro Center card I've been running update all and mame getter on weekly for years with no issue. I'm sure it'll wear out eventually but then I'll just buy a new one.
held
Posts: 185
Joined: Sun Sep 26, 2021 2:18 pm
Has thanked: 24 times
Been thanked: 29 times

Re: I think "/media/fat" should be depricated

Unread post by held »

Maybe its just me but I'd rather have my roms on a SSD.

@jca
Yeah and no, so I'll gather my thoughts and do some more investigation before I say something stupid ;)
User avatar
Newsdee
Posts: 661
Joined: Mon May 25, 2020 1:07 am
Has thanked: 68 times
Been thanked: 141 times

Re: I think "/media/fat" should be depricated

Unread post by Newsdee »

Just backup your card once in a while ;)

Wish the google drive scripts worked again, they made it very easy to save amd restore save data.
theypsilon
Scripting Wizard
Posts: 80
Joined: Sun May 24, 2020 8:20 pm
Been thanked: 37 times

Re: I think "/media/fat" should be depricated

Unread post by theypsilon »

held wrote: Thu Nov 24, 2022 1:13 pm It still leaves the scripting argument.
USB external drive also overlays the SD card with the updaters, the same as in the firmware.
riblad_om
Posts: 40
Joined: Sun May 24, 2020 10:06 pm
Has thanked: 14 times
Been thanked: 2 times

Re: I think "/media/fat" should be depricated

Unread post by riblad_om »

held wrote: Thu Nov 24, 2022 4:26 pm Maybe its just me but I'd rather have my roms on a SSD.
I don't keep my non-arcade roms on the sd card, if that's what you're implying. Those go on the usb drive.

It's really the path of least resistance. Let update_all manage the system files and arcade roms on the sd card. Plug in a usb drive with a games directory for console roms and those will automatically be mounted at boot.
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

riblad_om wrote: Thu Nov 24, 2022 4:18 pm I think you guys may be a bit paranoid about sd card wear. I have a Micro Center card I've been running update all and mame getter on weekly for years with no issue. I'm sure it'll wear out eventually but then I'll just buy a new one.
It is not a question of being paranoid on my side: I was using "Switch to USB" until Sorge said it was better to use symlinks which is what I did. Why not do it for almost every directory. It also has the advantage of making things faster if I have to redo my SD card. But it is just me.
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

So I made a test to see if this automatic switch worked also for directories other than the games directory: it does not.
I removed the symlink SD -> USB for _Arcade and rebooted: no more Arcades.
I created an empty _Arcade on the SD card and rebooted: I have an empty _Arcade directory.
riblad_om
Posts: 40
Joined: Sun May 24, 2020 10:06 pm
Has thanked: 14 times
Been thanked: 2 times

Re: I think "/media/fat" should be depricated

Unread post by riblad_om »

The Arcades collection is so small that there is no reason not to keep it on the sd card.
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

@theypsilon Thanks for your post. It made me think and leads me to a question:
When I placed everything on USB I created symlinks SD -> USB, includings games. I do not have a games directory on the SD card, just a symlink. After that In had some problems with Mame Getter and HB mame Getter as they did not find any mra. To fix the problem I created ini files for the getter to point the Arcade directory and the ROM directories to USB.
Would getting rid of the games symlink and creating an empty games directory on the SD card allow me to also get rid of these ini files?
jca
Posts: 1275
Joined: Wed May 27, 2020 1:59 pm
Has thanked: 93 times
Been thanked: 255 times

Re: I think "/media/fat" should be depricated

Unread post by jca »

riblad_om wrote: Thu Nov 24, 2022 5:22 pm The Arcades collection is so small that there is no reason not to keep it on the sd card.
1. We live in a free world or may be not but we can pretend we do. :D
2. Arcades was just for a test after FPGA64 sowed doubt in my mind. Shame on you FPGA64. :D
As I said earlier I have almost everything on the USB drive. This was after using "Switch to USB" and switching to symlinks. I also like experimenting.
Here is the contents of my SD card.
Attachments
Fat.jpg
Fat.jpg (36.36 KiB) Viewed 682 times
theypsilon
Scripting Wizard
Posts: 80
Joined: Sun May 24, 2020 8:20 pm
Been thanked: 37 times

Re: I think "/media/fat" should be depricated

Unread post by theypsilon »

@jca, if I were you, I'd get rid of that symlink setup. All folders should be on the SD card, except games and docs that can stay on your external drive if you want. Once you have that, the default downloader.ini options will install "games" files on your external drive for each system that already has its folders there (same as what the firmware does to load the games).
Malor
Posts: 571
Joined: Wed Feb 09, 2022 11:50 pm
Has thanked: 32 times
Been thanked: 124 times

Re: I think "/media/fat" should be depricated

Unread post by Malor »

FPGA64 wrote: Thu Nov 24, 2022 11:51 am You dont need to symlink anything. The USB external drive overlays the SD card. If a directory is on the external drive it replaces whatever is on the SD. if the directory is not present on the external drive then the SD directory is used. Its all automatic

The switch to USB option should not be touched
In my experience, that's what the docs say should happen, but it doesn't actually work, at least not with /media/fat/cifs. Update_all, for instance, was particularly annoying about insisting that the AO486 BIOS files always ended up in /media/fat/games/AO486. There seemed to be numerous papercuts of that nature.

Symlinks, on the other hand, instantly fixed the problem. Smart programs go into /media/fat/cifs/whatever, and stupid programs are forced into /media/fat/cifs via symlinks.
theypsilon wrote: Thu Nov 24, 2022 11:32 pm @jca, if I were you, I'd get rid of that symlink setup. All folders should be on the SD card, except games and docs that can stay on your external drive if you want. Once you have that, the default downloader.ini options will install "games" files on your external drive for each system that already has its folders there (same as what the firmware does to load the games).
My ZFS NAS is enormously more robust than a simple microSD card, and offers timed snapshots, so I can easily restore older versions of save files or savestates. (This has saved my butt at least once.) It's also a lot faster; the gigabit Ethernet port seems to actually have a dedicated gigabit, where the USB devices all have to share 480Mbps between them. Putting as many things as possible on the NAS makes things snappier now, and also makes it very easy to rebuild the card if it ever fails. I just have to connect the /media/fat/cifs share and remake the symlinks.

I should probably write a script to do that, but haven't bothered.

edit: well, no, I'd have to do some mucking around in /etc/, too. A card rebuild would still be pretty painful. But it wouldn't take as much downloading, at least.
FPGA64
Posts: 327
Joined: Mon Mar 01, 2021 3:10 pm
Has thanked: 31 times
Been thanked: 104 times

Re: I think "/media/fat" should be depricated

Unread post by FPGA64 »

I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive
held
Posts: 185
Joined: Sun Sep 26, 2021 2:18 pm
Has thanked: 24 times
Been thanked: 29 times

Re: I think "/media/fat" should be depricated

Unread post by held »

theypsilon wrote: Thu Nov 24, 2022 4:39 pm
held wrote: Thu Nov 24, 2022 1:13 pm It still leaves the scripting argument.
USB external drive also overlays the SD card with the updaters, the same as in the firmware.
Does it use bind mount ? because that is a far better option than I initially suggested, and would negate my deprecation request/suggestion entirely.
Malor
Posts: 571
Joined: Wed Feb 09, 2022 11:50 pm
Has thanked: 32 times
Been thanked: 124 times

Re: I think "/media/fat" should be depricated

Unread post by Malor »

FPGA64 wrote: Fri Nov 25, 2022 7:50 am I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive
IMO, this is bad advice. Hard drives are typically much more reliable than SD cards. They also will retain data for very long periods while unplugged, where both SDs and SSDs will often start losing data after about a year without power.
cursedverses
Posts: 103
Joined: Sun May 24, 2020 9:13 pm
Has thanked: 94 times
Been thanked: 14 times

Re: I think "/media/fat" should be depricated

Unread post by cursedverses »

Malor wrote: Fri Nov 25, 2022 10:53 am
FPGA64 wrote: Fri Nov 25, 2022 7:50 am I would not use switch to USB option. Its simply not needed for an External drive. Its much simpler to keep all the cores/ saves/ scripts and executables on the SD card and use the external HD just for the Games. I find it best to format the external drive as exFat32 to avoid having to deal with case sensitivity of filenames.

Some of the automatic updates such as A0486 may not work, but its simple to run it and then copy the new file to the external drive
IMO, this is bad advice. Hard drives are typically much more reliable than SD cards. They also will retain data for very long periods while unplugged, where both SDs and SSDs will often start losing data after about a year without power.
That's the first I've ever heard of this. Do you have anything to back up that claim?!
Malor
Posts: 571
Joined: Wed Feb 09, 2022 11:50 pm
Has thanked: 32 times
Been thanked: 124 times

Re: I think "/media/fat" should be depricated

Unread post by Malor »

Yeah, they absolutely do; they need to stay connected to power so that the card controller can, over time, move the data around. The storage cells work by being charged up, and the charge leaks out again. If the card isn't powered, this can result in data loss.

It has a lot to do with how hot the card was when the data was written. Weirdly, the hotter the better. You want the chipset cool so it runs well, but you want the memory banks hot while being written, but then cold while being stored.

This link has some data:

https://www.partitionwizard.com/clone-d ... orage.html

Search for "Is SSD Good for Long-Term Storage?" The table shows that if the SSD is stored, without power, in 55C heat, then data can be lost in just a week if it was written at 25C, and can be lost in 8 weeks if it was written at 55C.

On the other hand, the best case in that chart is writing at 55C and storing at 25C, where data can last over 400 weeks.

This link has more graphs, from a different vendor, but showing roughly comparable figures:

https://www.anandtech.com/show/9248/the ... -retention

Regardless, the underlying lesson is this: cards and SSDs that are not connected to power degrade. They can degrade very quickly in hot temperatures. Their retention will be fine as long as they have power, as the controller can detect the weakening charge and cycle the data to different cells, starting the leak time over again.

You should expect SD cards to be markedly worse than this; SSDs are built to a much higher quality standard.

Hard drives, on the other hand, deteriorate extremely slowly. The failure point there is that the drive might not spin up after being stored a long time, as opposed to the magnetic media losing its charge. You can stick one on the shelf and usually leave it there a decade. This, IMO, is a *much* better place to put your save games.
cursedverses
Posts: 103
Joined: Sun May 24, 2020 9:13 pm
Has thanked: 94 times
Been thanked: 14 times

Re: I think "/media/fat" should be depricated

Unread post by cursedverses »

Wow! That's some really useful and interesting content!

I guess if we're turning our MiSTers at least every couple of weeks or so we wouldn't see data loss, right? I guess from a data view, you'd want to keep games and saves on your external drive along with a copy of your mister.ini/downloader.ini. Cores can be redownloaded and the same for Scripts.

Using the USB drive for everything doesn't seem to need to be the case from my perspective.
Malor
Posts: 571
Joined: Wed Feb 09, 2022 11:50 pm
Has thanked: 32 times
Been thanked: 124 times

Re: I think "/media/fat" should be depricated

Unread post by Malor »

Yeah, if you're turning on your Mister every few weeks, that should be fine, as long as it's on for a few hours to give the controller time to refresh anything that needs it. And there's no critical need to store cores on the USB drive, unless you think the project might get nailed by someone claiming copyright infringement. It might save you a bunch of downloading if your card dies... but you might be trading away time fiddling with creating symlinks before running update_all.

Storing ROMs and saves on spinning media seems like an extremely good idea to me.

FWIW, so far I've left the actual cores and the config directory on the SD, but put everything else under /media/fat/cifs. I thought I'd moved the cores, but I looked last night, and haven't done that yet. As you say, that can be re-downloaded, and I don't have a bandwidth cap, so maybe I won't bother.

You probably want the config dir on the same filesystem with the Mister, because it needs to be available when the menu launches. On a cold boot, network shares don't come up until a few seconds after the menu starts, which is too late, so it drops back to default settings. (but then remembers its settings after a warm boot, which is confusing if you don't understand what's going on.) I'm not sure if USB filesystems would come up fast enough or not.... I'd err in the direction of just keeping the config dir on the same filesystem as the main MiSTer binary.

Taking occasional backups of config probably wouldn't hurt. Each file usually doesn't take that long to recreate, but there can be a lot of them.
theypsilon
Scripting Wizard
Posts: 80
Joined: Sun May 24, 2020 8:20 pm
Been thanked: 37 times

Re: I think "/media/fat" should be depricated

Unread post by theypsilon »

Malor wrote: Fri Nov 25, 2022 6:03 am
My ZFS NAS is enormously more robust than a simple microSD card, and offers timed snapshots, so I can easily restore older versions of save files or savestates. (This has saved my butt at least once.) It's also a lot faster; the gigabit Ethernet port seems to actually have a dedicated gigabit, where the USB devices all have to share 480Mbps between them. Putting as many things as possible on the NAS makes things snappier now, and also makes it very easy to rebuild the card if it ever fails. I just have to connect the /media/fat/cifs share and remake the symlinks.

I should probably write a script to do that, but haven't bothered.

edit: well, no, I'd have to do some mucking around in /etc/, too. A card rebuild would still be pretty painful. But it wouldn't take as much downloading, at least.
Until your network is down for any random reason. What you mention is a good option for power users that know what they are doing, know well the drawbacks, and can troubleshoot everything in case they face some problems. But it's not a setup that I would recommend to 99% of the users, who only want to install everything and play. If you opt into that setup, you are on your own.

Btw, update_all and downloader work as described before (games folder overlays) since May/June as long as you have the default options.
Malor
Posts: 571
Joined: Wed Feb 09, 2022 11:50 pm
Has thanked: 32 times
Been thanked: 124 times

Re: I think "/media/fat" should be depricated

Unread post by Malor »

This isn't your problem, but it seems to me that making the use of a NAS easy would be a lot more user friendly than leaving people out to hang with possibly bad SD cards. There are a lot of counterfeits out there, and even good SDs aren't that good.

If update_all is now properly using multiple targets correctly, that's a great step in the right direction, but when first transitioning to the NAS, I was having lots of issues with various different parts of Mister. Symlinks made all that pain go away.
Post Reply